Code

History

Commits on main to apps/api/sharedLibs35+
  1. fix(api/native/pdf-parser): trim null chars out of pdf titles (#2086)Gergő Móricz committed 1 year ago03c9339
  2. Fix go-html-to-md on Windows (ENG-3398) (#2082)tom committed 1 year agof466400
  3. feat(pdf-parser): get PDF title and show in metadata (#2062)Gergő Móricz committed 1 year agob3607eb
  4. fix(api/html-transformer): bad mergeGergő Móricz committed 1 year agoddd330a
  5. feat(api): support extraction of data-* attributes in scrape endpoints (#2006)Vishnu Krishnan committed 1 year ago0589994
  6. feat(api): add image extraction support to v2 scrape endpoint (#2008)Vishnu Krishnan committed 1 year ago30c6bdd
  7. feat(html-transformer): excludeGergő Móricz committed 1 year ago1f21ec3
  8. mendableai -> firecrawlGergő Móricz committed 1 year ago2f3bc4e
  9. feat(crawler): replace robotstxt library with texting_robots for ENG-3016 (#1895)devin-ai-integration[bot] committed 1 year agoe7e2c02
  10. Revert "fix(go): add mutex to prevent concurrent access issues in html-to-markdown (#1883)"Gergő Móricz committed 1 year ago96a9210
  11. fix(go): add mutex to prevent concurrent access issues in html-to-markdown (#1883)Gergő Móricz committed 1 year ago52e7d0d
  12. Revert go version in Dockerfile (#1873)Gergő Móricz committed 1 year ago1cf5e73
  13. fix(crawler/sitemap): improvements (#1842)Gergő Móricz committed 1 year agoebdd2ea
  14. Fix robots.txt parser panic with content type validation (#1843)devin-ai-integration[bot] committed 1 year ago0f477be
  15. feat: rewrite sitemap XML parsing from JavaScript to Rust (ENG-2904) (#1840)devin-ai-integration[bot] committed 1 year agoe0b26f8
  16. Restore Rust link filtering logic (#1822)Gergő Móricz committed 1 year ago17241e0
  17. fix(crawler): handle negative limitGergő Móricz committed 1 year ago7336e84
  18. fix(crawler): handle infinite limitGergő Móricz committed 1 year ago7695a8c
  19. feat(crawler): show serde errorGergő Móricz committed 1 year ago8d41c99
  20. feat(crawler): port filterLinks to Rust (#1801)Gergő Móricz committed 1 year agoa4f4145
  21. html-transformer: never panic (#1778)Gergő Móricz committed 1 year ago8f119d8
  22. Revert "fix(api/html-transformer): relative base tag URL handling (#1776)"Gergő Móricz committed 1 year ago9cad755
  23. fix(api/html-transformer): relative base tag URL handling (#1776)Gergő Móricz committed 1 year ago4e81246
  24. fix(api/html-transformer): stop panicing on arabic sites (#1773)Gergő Móricz committed 1 year ago71c8224
  25. feat(html-transformer, scrapeURL): omce support (#1764)Gergő Móricz committed 1 year ago9cc32c2
  26. html-transformer improvements (#1762)Gergő Móricz committed 1 year ago88549c2
  27. fix(html-transformer): check base tag when resolving relative URLs (#1761)Gergő Móricz committed 1 year ago8d81209
  28. Implement JavaScript-only base href handling (ENG-2302) (#1756)devin-ai-integration[bot] committed 1 year agode1bc96
  29. feat(api): zero data retention (ENG-2376) (#1687)Gergő Móricz committed 1 year ago4506b21
  30. feat: enhance metadata extraction by including 'itemprop' attribute in HTML (#1624)Ademílson Tonato committed 1 year ago4189713
  31. fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) (#1597)Gergő Móricz committed 1 year ago8389a1a
  32. feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) (#1592)Gergő Móricz committed 1 year agob03670a
  33. Fix: Concatenate metadata arrays into strings with exceptions (#1574)devin-ai-integration[bot] committed 1 year agoa5a915d
  34. feat: add swoogo classes to force include main tagsGergő Móricz committed 1 year ago2e2c3d5
  35. fix(html-transformer): Update free_string function parameter type (#1163)Carter LaSalle committed 1 year ago78094e2