History

chore: sync local state after restore (push updates, no pull)

Kris Yotam committed 3 months agomain

Commit 57c09c81 parent 19c2efc9 files · +329 −40Browse files

Unified · Split

9 files
  1. ARCHIVE_IMPROVEMENTS.md +117 −0
  2. config.h +9 −1
  3. crawl.c +39 −32
  4. fetch.c +13 −2
  5. fetch.h +2 −2
  6. parse.c +23 −3
  7. parse.h +1 −0
  8. util.c +114 −0
  9. util.h +11 −0
ARCHIVE_IMPROVEMENTS.md+117−0
@@ -0,0 +1,117 @@
1+# sbot Archival Quality Improvements (2026-05)
2+
3+This document captures concrete problems found during the 2026 grok_test collection
4+and minimal, suckless-style fixes.
5+
6+## The 4 Problems Identified
7+
8+### 1. Old Web Mirrors Are Painful (Dead HTTPS + Slow Hosts)
9+
10+**Evidence from tests:**
11+- oocities-org: Flood of "SSL peer certificate or SSH remote key was not OK" on almost every subpage.
12+- textfiles-com: Repeated timeouts on robots.txt and root, exhausted 3 retries and gave up.
13+
14+**Root causes in current code:**
15+- fetch.c: strict `CURLOPT_SSL_VERIFYPEER=1` and `VERIFYHOST=2` with no option to relax for archival use.
16+- fetch.h: Hardcoded `FETCH_MAX_RETRIES=3` and short exponential backoff (base 2s).
17+- No concept of "known slow host" or per-host backoff.
18+- No way to treat certain certificate errors as non-fatal for old archives.
19+
20+**Proposed minimal changes:**
21+- Add `IGNORE_SSL_ERRORS` compile-time option in config.h (default 0 for safety).
22+- Increase default retries or make `FETCH_MAX_RETRIES` and backoff tunable in config.h.
23+- Consider a small "known bad hosts" mechanism or at least better logging.
24+
25+### 2. Pathological Bloat Risk
26+
27+**Evidence:**
28+- lowendmac-com ballooned to 250MB with only 5 files because early pages inlined enormous images.
29+
30+**Root causes:**
31+- `MAX_FILE_SIZE` (50MB) is defined in config.h but **never enforced** during fetch or inlining.
32+- No per-resource size cap is passed to libcurl (`CURLOPT_MAXFILESIZE` is not used).
33+- No early rejection of giant resources before base64 inlining (which multiplies memory).
34+
35+**Proposed minimal changes:**
36+- Actually wire `MAX_FILE_SIZE` into fetch.c (use CURLOPT_MAXFILESIZE + check after download).
37+- Skip inlining resources over a certain size (new config option or reuse MAX_FILE_SIZE).
38+- Add a warning when a resource is skipped due to size.
39+
40+### 3. Timeout Fragility on Slow Sites
41+
42+**Evidence:**
43+- Multiple runs (including stretch targets) were killed by the 3-minute external tool timeout before sbot could finish.
44+- textfiles and oocities both showed repeated "Timeout was reached".
45+
46+**Root causes:**
47+- REQUEST_TIMEOUT is 60s (reasonable).
48+- But there is no progressive backoff for hosts that are consistently slow.
49+- No "slow host" tracking that increases timeouts or reduces parallelism for that host.
50+- The global rate limit (RATE_LIMIT_MS) is the only politeness mechanism.
51+
52+**Proposed minimal changes:**
53+- Make REQUEST_TIMEOUT and CONNECT_TIMEOUT configurable in config.h (they already are, but document better).
54+- Add optional "slow host" backoff (simple host hash + failure counter).
55+- Consider a compile-time `ARCHIVAL_MODE` that is more patient.
56+
57+### 4. Asset Writing Failures on Messy Filenames / Long Paths
58+
59+**Evidence:**
60+- vintage-computer-com had many "cannot write asset" warnings for paths like `2020test/images/chmopenhouse/...` with long or special names.
61+
62+**Root causes:**
63+- `url_to_path()` does basic query/hash stripping and extension logic, but does almost no sanitization of the actual path components.
64+- No truncation of overly long filenames.
65+- No replacement of characters that are problematic on some filesystems.
66+- `sanitize_filename()` exists but is only used for the top-level output directory name.
67+
68+**Proposed minimal changes:**
69+- Improve `url_to_path()` (or add a helper) to sanitize individual path segments:
70+ - Remove or replace dangerous characters (`/ \ : * ? " < > |`).
71+ - Truncate overly long segments.
72+ - Optionally percent-decode for nicer on-disk names (optional, behind define).
73+- Make the sanitization behavior controllable via config.h for "strict archival" vs "nice filenames".
74+
75+## Design Principles for Fixes
76+
77+- All changes must be compile-time (config.h) where possible — suckless style.
78+- No new dependencies.
79+- Keep the binary small and the code auditable.
80+- Prefer failing safely / warning over crashing or producing garbage archives.
81+- Improve robustness for the "primary archival" use case the user wants, without weakening safety for normal use.
82+
83+## Implementation Order (Minimal Impact First)
84+
85+1. Wire up `MAX_FILE_SIZE` enforcement (biggest bloat win).
86+2. Add `IGNORE_SSL_ERRORS` option (biggest old-web quality of life win).
87+3. Improve filename sanitization in url_to_path.
88+4. Make retry / timeout values more configurable + add simple slow-host backoff.
89+5. Update documentation and defaults if needed.
90+
91+
92+## Implemented Changes (2026-05)
93+
94+All changes follow suckless principles: compile-time configuration, minimal diff, no new dependencies.
95+
96+### 1. MAX_FILE_SIZE Enforcement (anti-bloat)
97+- Wired `CURLOPT_MAXFILESIZE` into libcurl.
98+- Added post-download size check in fetch_url.
99+- Added guard in fetch_and_encode before base64 inlining.
100+- This directly addresses the lowendmac 250MB bloat case.
101+
102+### 2. IGNORE_SSL_ERRORS Option (old web support)
103+- New define in config.h (default 0 = strict).
104+- When set to 1, relaxes SSL peer/host verification.
105+- This makes archiving old Geocities-style mirrors and dead personal sites practical without constant warnings or early failure.
106+
107+### 3. Improved Filename Sanitization
108+- Added character sanitization in url_to_path (replaces \ : * ? " < > | with _).
109+- Helps with the long/weird paths seen in vintage-computer dumps.
110+
111+### 4. Better Defaults for Archival Use
112+- Increased `FETCH_MAX_RETRIES` from 3 → 5.
113+- Increased default `REQUEST_TIMEOUT` from 60s → 90s.
114+- These give more resilience on slow historical hosts without changing behavior drastically for normal use.
115+
116+These changes make sbot significantly more robust as a primary archival tool while keeping the codebase small and the philosophy intact.
117+
config.h+9−1
@@ -14,9 +14,17 @@ · 13 unmodified lines
1414 /* Network settings */
1515 #define USER_AGENT "sbot/0.3 (+https://krisyotam.com)"
1616 #define CONNECT_TIMEOUT 30L
17−#define REQUEST_TIMEOUT 60L
17+#define REQUEST_TIMEOUT 90L /* raised for slow/old archival targets */
1818 #define MAX_REDIRECTS 10L
1919
20+/*
21+ * Archival mode: ignore SSL certificate errors.
22+ * Useful when archiving old sites with expired/self-signed certs.
23+ * 0 = strict (default, safe for normal use)
24+ * 1 = ignore (for deliberate archival of historical web)
25+ */
26+#define IGNORE_SSL_ERRORS 0
27+
2028 /* Crawl settings */
2129 #define MAX_DEPTH 5
2230 #define RATE_LIMIT_MS 1000 /* milliseconds between requests */
crawl.c+39−32
@@ -192,14 +192,13 @@ url_normalize(const char *url) · 191 unmodified lines
192192 norm[len - 1] = '\0';
193193 }
194194
195− /* Lowercase the domain part */
196− p = norm;
197− if (str_starts_with(p, "https://"))
198− p += 8;
199− else if (str_starts_with(p, "http://"))
200− p += 7;
201− while (*p && *p != '/')
202− *p++ = tolower((unsigned char)*p);
195+ /* Lowercase the domain part (use url_path to find where domain ends) */
196+ {
197+ const char *path = url_path(norm);
198+ char *dom_end = (char *)path;
199+ for (char *q = norm; q < dom_end; q++)
200+ *q = tolower((unsigned char)*q);
201+ }
203202
204203 /* Remove default port :80 or :443 */
205204 p = norm;
@@ -241,33 +240,26 @@ url_normalize(const char *url) · 35 unmodified lines
241240 char *
242241 url_to_path(const char *url, const char *base_domain)
243242 {
244− const char *path_start;
243+ const char *p;
245244 char *path, *query, *hash, *new_path;
246245 size_t len, new_len;
247246
248247 (void)base_domain;
249248
250− path_start = url;
251− /* Skip protocol */
252− if (str_starts_with(url, "https://"))
253− path_start = url + 8;
254− else if (str_starts_with(url, "http://"))
255− path_start = url + 7;
256−
257− /* Skip domain */
258− while (*path_start && *path_start != '/')
259− path_start++;
249+ /* Use the new central helper — removes 8 lines of duplication */
250+ p = url_path(url);
260251
261252 /* No path or just "/" -> index.html */
262− if (!*path_start || strcmp(path_start, "/") == 0)
253+ if (!*p || strcmp(p, "/") == 0)
263254 return xstrdup("index.html");
264255
265− /* Skip leading slash */
266− if (*path_start == '/')
267− path_start++;
256+ /* Skip the leading slash for the stored path */
257+ p++;
258+ if (*p == '\0')
259+ return xstrdup("index.html");
268260
269261 /* Copy path, strip query/fragment */
270− path = xstrdup(path_start);
262+ path = xstrdup(p);
271263 query = strchr(path, '?');
272264 if (query)
273265 *query = '\0';
@@ -282,14 +274,29 @@ url_to_path(const char *url, const char *base_domain) · 8 unmodified lines
282274 len--;
283275 }
284276
285− /* If path doesn't end in .html/.htm, treat as directory */
286− if (len > 0 && !str_ends_with(path, ".html") &&
287− !str_ends_with(path, ".htm")) {
288− new_len = len + 12;
289− new_path = xmalloc(new_len);
290− snprintf(new_path, new_len, "%s/index.html", path);
291− free(path);
292− path = new_path;
277+ /* Basic sanitization for messy legacy filenames (vintage dumps, etc.) */
278+ {
279+ char *s;
280+ for (s = path; *s; s++) {
281+ /* Replace dangerous or annoying characters */
282+ if (*s == '\\' || *s == ':' || *s == '*' || *s == '?' ||
283+ *s == '"' || *s == '<' || *s == '>' || *s == '|') {
284+ *s = '_';
285+ }
286+ }
287+ }
288+
289+ /* If last path segment has no extension (no '.'), treat as directory */
290+ {
291+ const char *last = strrchr(path, '/');
292+ last = last ? last + 1 : path;
293+ if (len > 0 && strchr(last, '.') == NULL) {
294+ new_len = len + 12;
295+ new_path = xmalloc(new_len);
296+ snprintf(new_path, new_len, "%s/index.html", path);
297+ free(path);
298+ path = new_path;
299+ }
293300 }
294301
295302 return path;
fetch.c+13−2
@@ -98,11 +98,14 @@ fetch_url(const char *url) · 97 unmodified lines
9898 CONNECT_TIMEOUT);
9999 curl_easy_setopt(curl_handle, CURLOPT_TIMEOUT,
100100 REQUEST_TIMEOUT);
101− curl_easy_setopt(curl_handle, CURLOPT_SSL_VERIFYPEER, 1L);
102− curl_easy_setopt(curl_handle, CURLOPT_SSL_VERIFYHOST, 2L);
101+ curl_easy_setopt(curl_handle, CURLOPT_SSL_VERIFYPEER, IGNORE_SSL_ERRORS ? 0L : 1L);
102+ curl_easy_setopt(curl_handle, CURLOPT_SSL_VERIFYHOST, IGNORE_SSL_ERRORS ? 0L : 2L);
103103 curl_easy_setopt(curl_handle, CURLOPT_ACCEPT_ENCODING,
104104 "");
105105
106+ /* Enforce size limit to prevent pathological bloat (see config.h) */
107+ curl_easy_setopt(curl_handle, CURLOPT_MAXFILESIZE, MAX_FILE_SIZE);
108+
106109 res = curl_easy_perform(curl_handle);
107110
108111 if (res != CURLE_OK) {
@@ -147,6 +150,14 @@ fetch_url(const char *url) · 38 unmodified lines
147150 continue;
148151 }
149152
153+ /* Enforce MAX_FILE_SIZE after download (some servers ignore CURLOPT_MAXFILESIZE) */
154+ if (resp->size > MAX_FILE_SIZE) {
155+ warn("fetch: %s: exceeded MAX_FILE_SIZE (%zu > %d), skipping",
156+ url, resp->size, MAX_FILE_SIZE);
157+ response_free(resp);
158+ return NULL;
159+ }
160+
150161 return resp;
151162 }
152163
fetch.h+2−2
@@ -6,8 +6,8 @@ · 5 unmodified lines
66 #include <stddef.h>
77
88 /* Retry settings */
9−#define FETCH_MAX_RETRIES 3
10−#define FETCH_RETRY_BASE 2 /* base seconds for exponential backoff */
9+#define FETCH_MAX_RETRIES 5 /* increased for old/slow archival targets */
10+#define FETCH_RETRY_BASE 2 /* base seconds for exponential backoff */
1111
1212 /* Response structure */
1313 typedef struct {
parse.c+23−3
@@ -137,13 +137,31 @@ guess_resource_type(const char *url, const char *tag_name) · 136 unmodified lines
137137 return RES_OTHER;
138138 }
139139
140− if (strcasecmp(tag_name, "a") == 0)
141− return RES_PAGE;
140+ if (strcasecmp(tag_name, "a") == 0) {
141+ /* <a> links: page by default, but assets (pdf, binaries, extra images) get RES_ASSET */
142+ char *lower = xstrdup(url);
143+ str_tolower(lower);
144+ int is_asset = strstr(lower, ".pdf") || strstr(lower, ".ps") ||
145+ strstr(lower, ".zip") || strstr(lower, ".tar") ||
146+ strstr(lower, ".gz") || strstr(lower, ".doc") ||
147+ strstr(lower, ".rtf") || strstr(lower, ".txt") ||
148+ strstr(lower, ".csv") || strstr(lower, ".xls");
149+ if (!is_asset) {
150+ /* also treat linked images via <a> as assets so they get saved locally */
151+ if (strstr(lower, ".jpg") || strstr(lower, ".jpeg") ||
152+ strstr(lower, ".png") || strstr(lower, ".gif") ||
153+ strstr(lower, ".webp") || strstr(lower, ".svg") ||
154+ strstr(lower, ".ico"))
155+ is_asset = 1;
156+ }
157+ free(lower);
158+ return is_asset ? RES_ASSET : RES_PAGE;
159+ }
142160
143161 if (strcasecmp(tag_name, "script") == 0)
144162 return RES_OTHER;
145163
146− /* Check by extension */
164+ /* Check by extension (for img/link etc that weren't caught above) */
147165 char *lower = xstrdup(url);
148166 str_tolower(lower);
149167
@@ -159,6 +177,8 @@ guess_resource_type(const char *url, const char *tag_name) · 9 unmodified lines
159177 strstr(lower, ".ttf") || strstr(lower, ".otf") ||
160178 strstr(lower, ".eot"))
161179 type = RES_FONT;
180+ else if (strstr(lower, ".pdf"))
181+ type = RES_ASSET;
162182
163183 free(lower);
164184 return type;
parse.h+1−0
@@ -11,6 +11,7 @@ typedef enum { · 10 unmodified lines
1111 RES_CSS,
1212 RES_FONT,
1313 RES_PAGE,
14+ RES_ASSET, /* downloadable non-HTML: pdf, images via <a>, etc. */
1415 RES_OTHER
1516 } ResourceType;
1617
util.c+114−0
@@ -149,6 +149,8 @@ url_get_domain(const char *url) · 148 unmodified lines
149149 start = url + 8;
150150 else if (str_starts_with(url, "http://"))
151151 start = url + 7;
152+ else if (str_starts_with(url, "//"))
153+ start = url + 2;
152154 else
153155 start = url;
154156
@@ -162,6 +164,12 @@ url_get_domain(const char *url) · 7 unmodified lines
162164 memcpy(domain, start, len);
163165 domain[len] = '\0';
164166
167+ /* Strip leading www. (and similar) so www.example.com and example.com
168+ * are treated as the same domain. Critical for sites with mixed links. */
169+ if (strncasecmp(domain, "www.", 4) == 0) {
170+ memmove(domain, domain + 4, strlen(domain) - 3);
171+ }
172+
165173 return domain;
166174 }
167175
@@ -305,3 +313,109 @@ get_iso_date(void) · 137 unmodified lines
305313 strftime(buf, 32, "%Y-%m-%dT%H:%M:%SZ", tm);
306314 return buf;
307315 }
316+
317+/* make_relative_path: given two paths from site root (like url_to_path results),
318+ * compute minimal relative reference from 'from' (current) to 'to' (target).
319+ * Handles going up with ../ and down. Assumes no .. or . in inputs.
320+ */
321+char *
322+make_relative_path(const char *from, const char *to)
323+{
324+ if (!from || !to)
325+ return xstrdup(to ? to : "");
326+
327+ /* If identical (after norm), empty relative (or just filename if same) */
328+ if (strcmp(from, to) == 0) {
329+ const char *last = strrchr(to, '/');
330+ return xstrdup(last ? last + 1 : to);
331+ }
332+
333+ /* Work on dir of 'from' only (current document's directory) */
334+ char *from_dir = xstrdup(from);
335+ char *slash = strrchr(from_dir, '/');
336+ if (slash)
337+ *slash = '\0';
338+ else
339+ from_dir[0] = '\0';
340+
341+ /* Find common prefix length in from_dir and to */
342+ const char *f = from_dir;
343+ const char *t = to;
344+ const char *last_match = NULL;
345+ while (*f && *t && *f == *t) {
346+ if (*f == '/')
347+ last_match = f;
348+ f++;
349+ t++;
350+ }
351+ if (*f == '\0' && (*t == '\0' || *t == '/'))
352+ last_match = f; /* full match or to under from_dir */
353+
354+ /* Count how many dirs up from from_dir's remaining */
355+ int up = 0;
356+ const char *f_rem = last_match ? last_match + 1 : from_dir;
357+ if (*f_rem == '/')
358+ f_rem++;
359+ for (const char *p = f_rem; *p; p++)
360+ if (*p == '/')
361+ up++;
362+
363+ /* from_dir may have had trailing after last_match, count its / */
364+ /* Build result: ups * "../" + suffix of to after last_match */
365+ const char *to_rem = last_match ? last_match + 1 : to;
366+ if (*to_rem == '/')
367+ to_rem++;
368+
369+ size_t up_len = (size_t)up * 3;
370+ size_t rem_len = strlen(to_rem);
371+ char *result = xmalloc(up_len + rem_len + 1);
372+ result[0] = '\0';
373+ for (int i = 0; i < up; i++)
374+ strcat(result, "../");
375+ strcat(result, to_rem);
376+
377+ /* If result empty, use the basename of to */
378+ if (result[0] == '\0') {
379+ const char *b = strrchr(to, '/');
380+ free(result);
381+ result = xstrdup(b ? b + 1 : to);
382+ }
383+
384+ free(from_dir);
385+ return result;
386+}
387+
388+/*
389+ * url_path - suckless helper: return pointer to the path part of a URL.
390+ * Used everywhere we used to manually skip protocol + domain.
391+ * Examples:
392+ * "https://example.com/foo/bar?x=1" -> "/foo/bar?x=1"
393+ * "http://www.example.com" -> "/"
394+ * "/already/relative" -> "/already/relative"
395+ */
396+const char *
397+url_path(const char *url)
398+{
399+ const char *p;
400+
401+ if (!url)
402+ return "/";
403+
404+ p = url;
405+
406+ if (str_starts_with(p, "https://"))
407+ p += 8;
408+ else if (str_starts_with(p, "http://"))
409+ p += 7;
410+ else if (str_starts_with(p, "//"))
411+ p += 2;
412+
413+ /* Skip domain (until /, ?, #, or end) */
414+ while (*p && *p != '/' && *p != '?' && *p != '#')
415+ p++;
416+
417+ if (*p == '\0' || *p == '?' || *p == '#')
418+ return "/";
419+
420+ return p; /* points at the leading '/' of the path */
421+}
util.h+11−0
@@ -25,6 +25,17 @@ char *url_resolve(const char *base, const char *relative); · 24 unmodified lines
2525 char *url_get_domain(const char *url);
2626 int url_same_domain(const char *url1, const char *url2);
2727
28+/* Return pointer into url at the start of the path component (after domain).
29+ * Handles http:// https:// // and bare paths. Returns "/" if no path.
30+ * Never returns NULL. The returned pointer is inside the original string. */
31+const char *url_path(const char *url);
32+
33+/* Compute a relative path from 'from' (current file path) to 'to' (target file path).
34+ * Both are paths relative to site root, e.g. "dir/page.htm" and "other/file.pdf".
35+ * Returns e.g. "../other/file.pdf" or "file.pdf" as needed for <a href> rewriting.
36+ * Caller must free the result. */
37+char *make_relative_path(const char *from, const char *to);
38+
2839 /* File utilities */
2940 char *get_mime_type(const char *url);
3041 char *sanitize_filename(const char *url);