Code

History

Commits on main to apps/api/src/scraper35+
  1. build is failing. Getting back tsl config for engines. Keeping the fallbacklist without itrafaelmmiller committed 1 year ago69f0cc7
  2. Temporarily comment out TLS client engine options in the scraperrafaelmmiller committed 1 year ago1da56fe
  3. Remove TLS client engine options from the scraper temporarilyrafaelmmiller committed 1 year ago0a60985
  4. removing tlsclient from fallback list for nowrafaelmmiller committed 1 year ago2005a31
  5. feat(crawler): more debug logging (#2091)tom committed 1 year ago4c44e58
  6. Format api and add pre-commit hooks (ENG-3408) (#2088)tom committed 1 year agoc527796
  7. Fix sitemap parsing (ENG-3361) (#2085)tom committed 1 year ago688bf93
  8. Replace custom address validation functions with ipaddr.js (ENG-3404) (#2084)tom committed 1 year ago52a8fad
  9. fix(api/scrapeURL/index): re-sign expired screenshot URLs (#2080)Gergő Móricz committed 1 year ago4291af2
  10. fix(api/crawler/sitemap): bump sitemap limit (#2079)Gergő Móricz committed 1 year ago68d3b84
  11. fix(scrapeURL): title in metadataGergő Móricz committed 1 year ago309b1a8
  12. Update index.tsNicolas committed 1 year ago50d2335
  13. Update index.tsNicolas committed 1 year agoc1d6137
  14. Update index.tsNicolas committed 1 year ago4eacdb0
  15. Update index.tsNicolas committed 1 year ago0e8fd92
  16. Update index.tsNicolas committed 1 year agofd2a170
  17. feat: add maxPages parameter to PDF parser in v2 scrape API (#2047)devin-ai-integration[bot] committed 1 year ago7bea613
  18. feat(pdf-parser): get PDF title and show in metadata (#2062)Gergő Móricz committed 1 year agob3607eb
  19. fix(scrapeURL/fire-engine): turn off instantReturnGergő Móricz committed 1 year agod1418c8
  20. fix(scrapeURL/fire-engine): lower checkStatus frequencyGergő Móricz committed 1 year agob153a66
  21. feat(scrapeURL/rewrite): scrape Google Drive TXT/PDF files and sheets (#2053)Gergő Móricz committed 1 year ago822b95f
  22. feat(api): support extraction of data-* attributes in scrape endpoints (#2006)Vishnu Krishnan committed 1 year ago0589994
  23. feat(api): add image extraction support to v2 scrape endpoint (#2008)Vishnu Krishnan committed 1 year ago30c6bdd
  24. fix(sitemap): switch orderGergő Móricz committed 1 year ago91faf34
  25. fix(map): use tlsclient to fetch sitemap for better reliability (#2034)Gergő Móricz committed 1 year ago879908a
  26. fix(scrapeURL): incorrect log messageGergő Móricz committed 1 year agod95d09c
  27. fix(scrapeURL): invalid Add/RemoveFeatureError handlingGergő Móricz committed 1 year ago34214ff
  28. fix(scrapeURL): fix HTML transforming logic (#1999)Gergő Móricz committed 1 year agoa4a2fcf
  29. fix(scrapeURL): odd timeout on large HTML (#1994)Gergő Móricz committed 1 year ago6af184d
  30. feat(api/abtest): send off AB requests at a higher levelGergő Móricz committed 1 year ago71218e7
  31. feat(scrapeURL): AI maxAgeGergő Móricz committed 1 year agof0665fa
  32. feat(scrapeURL): improve waterfalling MRT logicGergő Móricz committed 1 year agofacba27
  33. Merge branch 'main' into nsc/v2Gergő Móricz committed 1 year ago17008ac
  34. feat(v2/crawl): change sitemap paramGergő Móricz committed 1 year ago548f19a
  35. Update mu (#1959)Thomas Kosmas committed 1 year agocfb09fd