Code

History

Commits on main to apps/api/src/scraper/WebScraper/utils35+
  1. Format api and add pre-commit hooks (ENG-3408) (#2088)tom committed 1 year agoc527796
  2. Add blocked domain exceptionMicah Stairs committed 1 year ago8e125b7
  3. Add blocked domain exceptionMicah Stairs committed 1 year ago5d378e6
  4. Add another blocked URL exceptionMicah Stairs committed 1 year agoe8531b9
  5. Add a couple exceptions to our blocked list (#1816)Micah Stairs committed 1 year agoe8d4e51
  6. feat(acuc): propagate team flags (FIR-1879) (#1522)Gergő Móricz committed 1 year agofa58199
  7. Update blocklist.tsNicolas committed 1 year agoae010a7
  8. fix(blocklist): behavioural fixesGergő Móricz committed 1 year agobc787de
  9. test(blocklist): Enhance URL blocking test coverage with decrypted domain validationAdemílson F. Tonato committed 1 year agobcd7449
  10. feat(blocklist): Improve URL blocking with tldts parsingAdemílson F. Tonato committed 1 year ago24e8aaf
  11. Update blocklist.tsNicolas committed 1 year agoc88176a
  12. fix(blocklist): unblockGergő Móricz committed 1 year agoec039dc
  13. chore: formattingNicolas committed 1 year ago588f747
  14. Nick: random fixesNicolas committed 1 year ago4987880
  15. Hash UrlsEric Ciarla committed 1 year agoa2998d4
  16. Nick: revert trailing commaNicolas committed 1 year ago8a1c404
  17. Nick: fixed prettierNicolas committed 1 year ago00335e2
  18. added library.tiktok to allowedKeywordsrafaelmmiller committed 1 year agob69c6f9
  19. fix(crawl/maxDepth): fix maxDepth behaviourGergő Móricz committed 2 years ago68c9615
  20. `WebScraper` refactor into `scrapeURL` (#714)Gergő Móricz committed 2 years ago8d467c8
  21. added trycatch and removed redundancyrafaelmmiller committed 2 years agof07bbef
  22. added remove base64 images options (true by default)rafaelmmiller committed 2 years ago4c5bb21
  23. Iframe supportThomas Kosmas committed 2 years agofe02101
  24. Merge pull request #785 from mendableai/nsc/support-for-all-metadataNicolas committed 2 years agoa73b065
  25. Update metadata.tsNicolas committed 2 years ago2ac50a1
  26. Nick: formatting + error handlingNicolas committed 2 years ago8974230
  27. Update metadata.tsNicolas committed 2 years ago417c769
  28. Update excludeTags.tsNicolas committed 2 years agoff906f7
  29. Update excludeTags.tsNicolas committed 2 years ago2c1a98f
  30. Update metadata.tsNicolas committed 2 years ago795e5a9
  31. fixed file blocking processrafaelsideguide committed 2 years ago2bf7b43
  32. Update blocklist.tsNicolas committed 2 years ago064ce48
  33. Update website_params.tsNicolas committed 2 years ago5c0c952
  34. Merge pull request #685 from devflowinc/mainNicolas committed 2 years agoff4b7a8
  35. Update website_params.tsNicolas committed 2 years agob8266cc