Code

History

Commits on main to apps/api/src/scraper/WebScraper35+
  1. feat(crawler): more debug logging (#2091)tom committed 1 year ago4c44e58
  2. Format api and add pre-commit hooks (ENG-3408) (#2088)tom committed 1 year agoc527796
  3. Fix sitemap parsing (ENG-3361) (#2085)tom committed 1 year ago688bf93
  4. fix(api/crawler/sitemap): bump sitemap limit (#2079)Gergő Móricz committed 1 year ago68d3b84
  5. fix(sitemap): switch orderGergő Móricz committed 1 year ago91faf34
  6. fix(map): use tlsclient to fetch sitemap for better reliability (#2034)Gergő Móricz committed 1 year ago879908a
  7. feat(v2): error handling (#1957)Gergő Móricz committed 1 year agob32fa6f
  8. feat(v2/timeout): initial new waterfalling system (ENG-2922) (#1950)Gergő Móricz committed 1 year ago9c001a9
  9. fix(map): full functionalityGergő Móricz committed 1 year ago33fb6bb
  10. feat(v2): parsers + merge w/ main (#1907)Gergő Móricz committed 1 year ago7400e10
  11. v2 scrape changes + rebase scrapeurl on v2Gergő Móricz committed 1 year ago281ce25
  12. fix(crawler/sitemap): improvements (#1842)Gergő Móricz committed 1 year agoebdd2ea
  13. ENG-2829: Fix isSubdomain bug (#1845)devin-ai-integration[bot] committed 1 year ago104b632
  14. feat: rewrite sitemap XML parsing from JavaScript to Rust (ENG-2904) (#1840)devin-ai-integration[bot] committed 1 year agoe0b26f8
  15. Restore Rust link filtering logic (#1822)Gergő Móricz committed 1 year ago17241e0
  16. (feat/rtxt) Improved robots control on scrape via flags (#1820)Nicolas committed 1 year agob23535c
  17. Add blocked domain exceptionMicah Stairs committed 1 year ago8e125b7
  18. Add blocked domain exceptionMicah Stairs committed 1 year ago5d378e6
  19. Add another blocked URL exceptionMicah Stairs committed 1 year agoe8531b9
  20. Add a couple exceptions to our blocked list (#1816)Micah Stairs committed 1 year agoe8d4e51
  21. Nick: no sections edge case (#1814)Nicolas committed 1 year ago51a7ad4
  22. Fix bug which sometimes caused crawl to only return 1 result (#1810)Micah Stairs committed 1 year agoa30af24
  23. fix(crawler): handle infinite limitGergő Móricz committed 1 year ago7695a8c
  24. feat(crawler): port filterLinks to Rust (#1801)Gergő Móricz committed 1 year agoa4f4145
  25. Make error message for backwards crawling more clear (#1758)Micah Stairs committed 1 year ago4c1af8b
  26. feat(api): zero data retention (ENG-2376) (#1687)Gergő Móricz committed 1 year ago4506b21
  27. Fix unreachable allowSubdomains code in crawler filterURL method (#1719)devin-ai-integration[bot] committed 1 year ago070d1c1
  28. Remove old cache mechanisms (redis cache, PDF cache, crawl maps, etc.) (FIR-2266) (#1667)Gergő Móricz committed 1 year ago3f0b8b8
  29. Improve URL filtering error messages with specific denial reasons (FIR-2352) (#1676)devin-ai-integration[bot] committed 1 year ago9710bdf
  30. feat(apps/api): add support for a separate, non-eviction Redis (#1600)Gergő Móricz committed 1 year agoa5efff0
  31. fix(crawler): always set expiry on sitemap links in redisGergő Móricz committed 1 year ago474e5a0
  32. feat(acuc): propagate team flags (FIR-1879) (#1522)Gergő Móricz committed 1 year agofa58199
  33. Add crawl delay functionality with per-crawl concurrency limiting (FIR-249) (#1413)devin-ai-integration[bot] committed 1 year ago411ecdf
  34. new cost trackingGergő Móricz committed 1 year ago8546bca
  35. compare format (FIR-1560) (#1405)Gergő Móricz committed 1 year ago24f5199