sbot Archival Quality Improvements (2026-05)

This document captures concrete problems found during the 2026 grok_test collection and minimal, suckless-style fixes.

The 4 Problems Identified

1. Old Web Mirrors Are Painful (Dead HTTPS + Slow Hosts)

Evidence from tests:

• oocities-org: Flood of "SSL peer certificate or SSH remote key was not OK" on almost every subpage.
• textfiles-com: Repeated timeouts on robots.txt and root, exhausted 3 retries and gave up.

Root causes in current code:

• fetch.c: strict CURLOPT_SSL_VERIFYPEER=1 and VERIFYHOST=2 with no option to relax for archival use.
• fetch.h: Hardcoded FETCH_MAX_RETRIES=3 and short exponential backoff (base 2s).
• No concept of "known slow host" or per-host backoff.
• No way to treat certain certificate errors as non-fatal for old archives.

Proposed minimal changes:

• Add IGNORE_SSL_ERRORS compile-time option in config.h (default 0 for safety).
• Increase default retries or make FETCH_MAX_RETRIES and backoff tunable in config.h.
• Consider a small "known bad hosts" mechanism or at least better logging.

2. Pathological Bloat Risk

Evidence:

• lowendmac-com ballooned to 250MB with only 5 files because early pages inlined enormous images.

Root causes:

• MAX_FILE_SIZE (50MB) is defined in config.h but never enforced during fetch or inlining.
• No per-resource size cap is passed to libcurl (CURLOPT_MAXFILESIZE is not used).
• No early rejection of giant resources before base64 inlining (which multiplies memory).

Proposed minimal changes:

• Actually wire MAX_FILE_SIZE into fetch.c (use CURLOPT_MAXFILESIZE + check after download).
• Skip inlining resources over a certain size (new config option or reuse MAX_FILE_SIZE).
• Add a warning when a resource is skipped due to size.

3. Timeout Fragility on Slow Sites

Evidence:

• Multiple runs (including stretch targets) were killed by the 3-minute external tool timeout before sbot could finish.
• textfiles and oocities both showed repeated "Timeout was reached".

Root causes:

• REQUEST_TIMEOUT is 60s (reasonable).
• But there is no progressive backoff for hosts that are consistently slow.
• No "slow host" tracking that increases timeouts or reduces parallelism for that host.
• The global rate limit (RATE_LIMIT_MS) is the only politeness mechanism.

Proposed minimal changes:

• Make REQUEST_TIMEOUT and CONNECT_TIMEOUT configurable in config.h (they already are, but document better).
• Add optional "slow host" backoff (simple host hash + failure counter).
• Consider a compile-time ARCHIVAL_MODE that is more patient.

4. Asset Writing Failures on Messy Filenames / Long Paths

Evidence:

• vintage-computer-com had many "cannot write asset" warnings for paths like 2020test/images/chmopenhouse/... with long or special names.

Root causes:

• url_to_path() does basic query/hash stripping and extension logic, but does almost no sanitization of the actual path components.
• No truncation of overly long filenames.
• No replacement of characters that are problematic on some filesystems.
• sanitize_filename() exists but is only used for the top-level output directory name.

Proposed minimal changes:

• Improve url_to_path() (or add a helper) to sanitize individual path segments:
  • Remove or replace dangerous characters (/ \ : * ? " < > |).
  • Truncate overly long segments.
  • Optionally percent-decode for nicer on-disk names (optional, behind define).
• Make the sanitization behavior controllable via config.h for "strict archival" vs "nice filenames".

Design Principles for Fixes

• All changes must be compile-time (config.h) where possible — suckless style.
• No new dependencies.
• Keep the binary small and the code auditable.
• Prefer failing safely / warning over crashing or producing garbage archives.
• Improve robustness for the "primary archival" use case the user wants, without weakening safety for normal use.

Implementation Order (Minimal Impact First)

1. Wire up MAX_FILE_SIZE enforcement (biggest bloat win).
2. Add IGNORE_SSL_ERRORS option (biggest old-web quality of life win).
3. Improve filename sanitization in url_to_path.
4. Make retry / timeout values more configurable + add simple slow-host backoff.
5. Update documentation and defaults if needed.

Implemented Changes (2026-05)

All changes follow suckless principles: compile-time configuration, minimal diff, no new dependencies.

1. MAX_FILE_SIZE Enforcement (anti-bloat)

• Wired CURLOPT_MAXFILESIZE into libcurl.
• Added post-download size check in fetch_url.
• Added guard in fetch_and_encode before base64 inlining.
• This directly addresses the lowendmac 250MB bloat case.

2. IGNORE_SSL_ERRORS Option (old web support)

• New define in config.h (default 0 = strict).
• When set to 1, relaxes SSL peer/host verification.
• This makes archiving old Geocities-style mirrors and dead personal sites practical without constant warnings or early failure.

3. Improved Filename Sanitization

• Added character sanitization in url_to_path (replaces \ : * ? " < > | with _).
• Helps with the long/weird paths seen in vintage-computer dumps.

4. Better Defaults for Archival Use

• Increased FETCH_MAX_RETRIES from 3 → 5.
• Increased default REQUEST_TIMEOUT from 60s → 90s.
• These give more resilience on slow historical hosts without changing behavior drastically for normal use.

These changes make sbot significantly more robust as a primary archival tool while keeping the codebase small and the philosophy intact.