History

Merge pull request #25 from davis7dotsh/agent/arbitrary-services

Ben Davis committed 3 months agomain

Commit a6e4f072 parents 4d1f99b 3e7089b18 files · +1674 −45Browse files

Unified · Split

18 files
  1. Makefile +1 −1
  2. README.md +50 −0
  3. docker/entrypoint.sh +4 −3
  4. docs/tx9-cli-design.md +15 −1
  5. guest/hb +53 −6
  6. guest/hb-workload +31 −1
  7. guest/tx9-logs +187 −26
  8. guest/tx9-services +462 −0
  9. internal/assets/assets.go +1 −1
  10. internal/assets/assets_test.go +4 −1
  11. internal/cli/cmd_logs.go +2 −2
  12. internal/cli/dispatch.go +1 −1
  13. internal/cli/dispatch_test.go +8 −0
  14. provision/provision.sh +2 −1
  15. tests/regressions-hb-workload.sh +107 −0
  16. tests/regressions-logs.sh +106 −0
  17. tests/regressions-services.sh +630 −0
  18. tests/static.sh +10 −1
Makefile+1−1
@@ -2,7 +2,7 @@ · 1 unmodified line
22
33 # tests/regressions-*.sh is globbed so new split-out regression files don't
44 # require touching this Makefile.
5−SHELL_FILES := guest/hb guest/hb-workload guest/lib-mcp.sh guest/profile.sh guest/agent-bash-profile.sh provision/provision.sh docker/entrypoint.sh docker/executor-entrypoint.sh tests/lib.sh tests/static.sh tests/hermes-state.sh $(wildcard tests/regressions-*.sh)
5+SHELL_FILES := guest/hb guest/hb-workload guest/tx9-services guest/lib-mcp.sh guest/profile.sh guest/agent-bash-profile.sh provision/provision.sh docker/entrypoint.sh docker/executor-entrypoint.sh tests/lib.sh tests/static.sh tests/hermes-state.sh $(wildcard tests/regressions-*.sh)
66
77 syntax:
88 bash -n $(SHELL_FILES)
README.md+50−0
@@ -72,6 +72,8 @@ Inside the agent container: · 71 unmodified lines
7272 hb status / hb doctor / hb versions
7373 hb down # durable pause; the reconcile loop won't restart services
7474 hb up
75+hb services # status for portable custom service drop-ins
76+hb services-reload # reconcile custom services immediately
7577 hb gateway-enable --confirm-single-writer I_CONFIRM_NO_OTHER_GATEWAY_USES_THIS_IDENTITY
7678 hb gateway-disable
7779 hb verify-state
@@ -87,6 +89,54 @@ tx9 gateway enable <box> --confirm-single-writer · 9 unmodified lines
8789 tx9 gateway disable <box>
8890 ```
8991
92+### Portable custom services
93+
94+An executable regular file placed directly in
95+`~/.config/hermes-box/services.d/` defines a custom service. Names must match
96+`[a-z0-9][a-z0-9_-]{0,62}`; directories, symlinks, non-executable files, and
97+unsafe names are ignored and reported by `hb services`. Definitions live on
98+the portable `/data` volume, so they survive backup, import, and upgrade.
99+
100+Each file is executed directly by absolute path, without shell sourcing,
101+evaluation, arguments, or interpolation. Scripts must remain in the
102+foreground and should `exec` their daemon. For example:
103+
104+```bash
105+install -d -m 700 ~/.config/hermes-box/services.d
106+cat >~/.config/hermes-box/services.d/signal <<'EOF'
107+#!/usr/bin/env bash
108+exec signal-cli daemon
109+EOF
110+chmod 700 ~/.config/hermes-box/services.d/signal
111+hb services-reload
112+```
113+
114+The existing 20-second workload loop also reconciles definitions
115+automatically. Each service is supervised independently with a bounded
116+restart delay; one crashing service cannot block the others. `hb pause` and
117+`hb down` synchronously stop all custom services and prevent restart, while
118+`hb up`/`hb resume` start them again. Disabling only the Hermes gateway does
119+not affect custom services. Container termination is forwarded through the
120+log supervisor to each foreground process.
121+
122+`hb services` distinguishes a stable foreground child (`running`) from a
123+live capture wrapper waiting to restart it (`restarting`). This is process
124+state, not an application health check; `hb doctor` cannot infer a generic
125+health contract for arbitrary service code.
126+
127+Service output uses the same rotating, redacted durable log pipeline as other
128+tx9-owned processes. Sources are distinct and collision-free:
129+
130+```bash
131+tx9 logs media-bot --source service-signal
132+tx9 logs media-bot --source all
133+```
134+
135+Drop-ins have the same permissions and network/filesystem access as the
136+`agent` user. Treat installing one as installing executable code in the box;
137+the filename and regular non-symlink checks prevent accidental shell
138+evaluation or symlink execution, but do not sandbox trusted service code.
139+
90140 ## Logs and resource allocation
91141
92142 Runtime output from the agent supervisor, Hermes gateway, and Executor is
docker/entrypoint.sh+4−3
@@ -1,8 +1,9 @@
11 #!/usr/bin/env bash
22 # Container PID-1 workload (run under docker --init so signals behave).
3−# Reuses guest/hb-workload verbatim: it reconciles Executor, the Hermes
4−# gateway, and the 0.0.0.0 socat bridges every 20s. tx9-logs supervises the
5−# loop process itself; Docker's restart policy supervises the container.
3+# Reuses guest/hb-workload verbatim: it reconciles custom services, Executor,
4+# the Hermes gateway, and the 0.0.0.0 socat bridges every 20s. tx9-logs
5+# supervises the loop process itself; Docker's restart policy supervises the
6+# container.
67 set -uo pipefail
78
89 trap 'exit 143' TERM
docs/tx9-cli-design.md+15−1
@@ -52,7 +52,7 @@ Docker overview can be read. `tx9 help` remains Docker-independent. · 51 unmodified lines
5252 | `tx9 backup <box>` (aliases `export`, `save`) | Flags: `--path` (default `~/Downloads`), `--password`/env/prompt, `--no-encrypt`. Quiesce → archive agent /data → validate → (encrypt) → verify → `<box>-<timestamp>.tx9`. |
5353 | `tx9 import <file.tx9>` (aliases `load`, `restore`) | Flags: `--name`, `--password`/env/prompt. Validate before creating anything; restore staged; arrive quiesced + gateway-disabled + fresh token; fail on name collision. |
5454 | `tx9 mount <add\|list\|remove> ...` | Persist host-directory bind mounts for an agent and recreate only its disposable container. Targets must be below `/mnt`, outside the portable `/data` volume. `add` supports `--read-only` and `--require-mountpoint`. |
55−| `tx9 logs <box>` | Query durable agent, Executor, Hermes, Codex, and Claude events. Filters include source, age, text, severity (`--level`, this level and above; unleveled events count as info), count, and normalized JSONL. `tx9 logs export <box>` creates a mode-0600 portable log bundle from both isolated volumes. |
55+| `tx9 logs <box>` | Query durable agent, Executor, Hermes, Codex, Claude, and custom-service events. Filters include source, age, text, severity (`--level`, this level and above; unleveled events count as info), count, and normalized JSONL. Custom sources use `service-<name>`. `tx9 logs export <box>` creates a mode-0600 portable log bundle from both isolated volumes. |
5656 | `tx9 resources <box>` | Show actual container CPU/RAM limits and volume usage versus advisory budgets. `resources set` updates limits live and persists them; `resources reset` restores 4 CPU/8 GiB (agent) and 2 CPU/2 GiB (Executor). |
5757 | `tx9 gateway <status\|enable\|disable> <box>` | Inspect or control the container-supervised Hermes gateway. Enable requires `--confirm-single-writer`. |
5858 | `tx9 open <box>` | Print (or open) the authenticated dashboard URL (`?_token=`). |
@@ -108,6 +108,20 @@ the port: · 49 unmodified lines
108108 SQLite histories. Executor data is never mounted into the agent container.
109109 This provides complete tx9-owned runtime output, not an independent audit
110110 trail for operations that Executor does not emit to any durable sink.
111+- **Portable custom services**: direct executable regular-file children of
112+ `${XDG_CONFIG_HOME:-$HOME/.config}/hermes-box/services.d` are supervised as
113+ the agent. Names are limited to `[a-z0-9][a-z0-9_-]{0,62}`. The absolute
114+ file path is passed as argv directly, never sourced or evaluated; scripts
115+ stay foreground and should `exec` their daemon. Each service has an
116+ independent `tx9-logs` capture and bounded restart loop under source
117+ `service-<name>`. Runtime PID/lock state stays outside `/data`, while the
118+ definitions and logs remain on `/data` and therefore travel in backups.
119+ `hb services` reports status and ignored entries; `hb services-reload`
120+ reconciles immediately in addition to the normal 20-second loop. Quiesce
121+ synchronously stops every custom service and blocks restart until
122+ `hb up`/`hb resume`; Hermes gateway enable/disable is independent. Drop-ins
123+ run with the agent's ordinary access and are trusted executable code, not a
124+ sandbox boundary.
111125 - **Gateway single-writer**: restored boxes never auto-enable the Hermes
112126 gateway. `tx9 gateway enable <box> --confirm-single-writer` delegates to
113127 `hb gateway-enable` and stays the only host-side path.
guest/hb+53−6
@@ -4,8 +4,9 @@ set -uo pipefail · 3 unmodified lines
44 # shellcheck disable=SC1091
55 [ -f /etc/profile.d/hermes-box.sh ] && . /etc/profile.d/hermes-box.sh
66 export TX9_LOG_MAX_BYTES TX9_LOG_MAX_FILES
7+HB_BIN_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
78 # shellcheck source=guest/lib-mcp.sh
8−. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib-mcp.sh"
9+. "$HB_BIN_DIR/lib-mcp.sh"
910
1011 DATA="${HB_DATA:-/data}"
1112 AGENT_HOME="$DATA/home/agent"
@@ -20,6 +21,7 @@ GATEWAY_LOCK="$STATE_DIR/gateway.lock" · 8 unmodified lines
2021 GATEWAY_RELOAD_REQUESTS="$STATE_DIR/gateway-reload-requests"
2122 GATEWAY_RELOAD_LOCK="$STATE_DIR/gateway-reload.lock"
2223 EXECUTOR_LOCK="$STATE_DIR/executor.lock"
24+SERVICES_DIR="$STATE_DIR/services.d"
2325 WIRE_VERSION=http-v3
2426 WIRED="$AGENT_HOME/.hermes-box-wired"
2527 TOKEN_FILE="$AGENT_HOME/.executor/server-control/auth.json"
@@ -64,9 +66,11 @@ init() { · 38 unmodified lines
6466 local managed=(
6567 "$AGENT_HOME" "$AGENT_HOME/.claude" "$AGENT_HOME/.codex" "$AGENT_HOME/.hermes"
6668 "$AGENT_HOME/workspace" "$AGENT_HOME/.config" "$AGENT_HOME/.config/hermes-box"
67− "$AGENT_HOME/.local" "$AGENT_HOME/.local/share" "$AGENT_HOME/.local/state" "$LOGS"
69+ "$SERVICES_DIR" "$AGENT_HOME/.local" "$AGENT_HOME/.local/share"
70+ "$AGENT_HOME/.local/state" "$LOGS"
6871 )
6972 mkdir -p "${managed[@]}"
73+ chmod 0700 "$SERVICES_DIR" 2>/dev/null || true
7074 id agent >/dev/null 2>&1 && chown agent:agent "${managed[@]}" 2>/dev/null || true
7175 if [[ -e "$LEGACY_QUIESCE_FILE" ]]; then
7276 touch "$QUIESCE_FILE" || return 1
@@ -81,6 +85,20 @@ init() { · 8 unmodified lines
8185 id agent >/dev/null 2>&1 && chown agent:agent "$GATEWAY_DISABLED" "$GATEWAY_POLICY" 2>/dev/null || true
8286 }
8387
88+_services_helper() {
89+ local helper
90+ helper="$(command -v tx9-services 2>/dev/null || true)"
91+ [[ -n "$helper" ]] || helper="$HB_BIN_DIR/tx9-services"
92+ [[ -x "$helper" ]] || {
93+ echo "tx9-services is unavailable" >&2
94+ return 1
95+ }
96+ TX9_SERVICES_CONFIG_ROOT="$STATE_DIR" TX9_SERVICES_LOG_DIR="$LOGS" "$helper" "$@"
97+}
98+
99+_services_reconcile() { _services_helper reconcile; }
100+_services_stop() { _services_helper stop-all; }
101+
84102 _port_open() {
85103 (exec 3<>"/dev/tcp/$EXECUTOR_HOST/${EXECUTOR_PORT:-4788}") 2>/dev/null && { exec 3>&-; return 0; }
86104 return 1
@@ -333,15 +351,29 @@ _start_gateway() { · 246 unmodified lines
333351 }
334352
335353 up() {
354+ local services_failed=0
355+ local gateway_status
356+
336357 init
337358 rm -f "$QUIESCE_FILE"
359+ _services_reconcile || {
360+ services_failed=1
361+ echo "warning: custom service reconciliation failed; run 'hb services-reload' for details" >&2
362+ }
338363 _executor_up || return 1
339364 _start_gateway
365+ gateway_status=$?
366+ [[ "$gateway_status" == 0 ]] || return "$gateway_status"
367+ [[ "$services_failed" == 0 ]]
340368 }
341369
342370 reconcile() {
343371 init
344− [[ ! -e "$QUIESCE_FILE" ]] || return 0
372+ if [[ -e "$QUIESCE_FILE" ]]; then
373+ _services_stop
374+ return
375+ fi
376+ _services_reconcile || echo "warning: custom service reconciliation failed; continuing core reconciliation" >&2
345377 HB_STEADY_QUIET=1 _executor_up || return 1
346378 _start_gateway
347379 }
@@ -372,13 +404,14 @@ pause() { · 24 unmodified lines
372404 local failed=0
373405 init
374406 touch "$QUIESCE_FILE"
407+ _services_stop || failed=1
375408 _stop_gateway || failed=1
376409 _stop_executor || failed=1
377410 if command -v hermes-state >/dev/null 2>&1; then
378411 hermes-state verify --checkpoint >/dev/null || failed=1
379412 fi
380413 [[ "$failed" == 0 ]] || return 1
381− echo "Hermes gateway and Executor: quiesced"
414+ echo "Custom services, Hermes gateway, and Executor: quiesced"
382415 }
383416
384417 resume() {
@@ -729,7 +762,7 @@ write_manifest() { · 344 unmodified lines
729762 }
730763
731764 status() {
732− local gateway_status
765+ local gateway_status services_count
733766 if [[ -e "$GATEWAY_DISABLED" ]]; then
734767 gateway_status=disabled
735768 elif _gateway_running; then
@@ -747,9 +780,20 @@ status() { · 11 unmodified lines
747780 printf 'platform: no runtime state\n'
748781 fi
749782 printf 'mcp: %s\n' "$([[ -f "$WIRED" ]] && cat "$WIRED" || echo unwired)"
783+ services_count="$(find "$SERVICES_DIR" -mindepth 1 -maxdepth 1 -printf x 2>/dev/null | wc -c | tr -d ' ')"
784+ printf "services: %s drop-in entries (run 'hb services')\n" "$services_count"
750785 printf 'data: %s\n' "$(du -sh "$DATA" 2>/dev/null | cut -f1)"
751786 }
752787
788+services() { _services_helper status; }
789+
790+services_reload() {
791+ local failed=0
792+ _services_reconcile || failed=1
793+ services || failed=1
794+ return "$failed"
795+}
796+
753797 versions() {
754798 echo "node: $(node --version 2>/dev/null || echo -)"
755799 # sed -n 1p, not head -1: vp prints more after the version line, and head
@@ -822,6 +866,7 @@ doctor() { · 66 unmodified lines
822866 _check "Codex HTTP MCP config" grep -qF "url = \"$(_executor_url)\"" "$CODEX_HOME/config.toml" || failed=1
823867 _check "Hermes HTTP MCP config" _hermes_http_config || failed=1
824868 fi
869+ echo "info custom services have no generic health contract; inspect process state with 'hb services'"
825870 [[ "$failed" == 0 ]]
826871 }
827872
@@ -861,9 +906,11 @@ main() { · 33 unmodified lines
861906 write-manifest) write_manifest ;;
862907 doctor) doctor ;;
863908 status) status ;;
909+ services) services ;;
910+ services-reload) services_reload ;;
864911 versions) versions ;;
865912 logs) shift; logs "${1:-executor}" ;;
866− *) echo "usage: hb {init|up|reconcile|down|pause|resume|gateway-enable|gateway-disable|gateway-reload-if-requested|verify-state|acknowledge-active-paths|cutover-ready|web|wire-mcp|doctor|status|versions|logs}"; return 1 ;;
913+ *) echo "usage: hb {init|up|reconcile|down|pause|resume|services|services-reload|gateway-enable|gateway-disable|gateway-reload-if-requested|verify-state|acknowledge-active-paths|cutover-ready|web|wire-mcp|doctor|status|versions|logs}"; return 1 ;;
867914 esac
868915 }
869916
guest/hb-workload+31−1
@@ -17,6 +17,8 @@ EXECUTOR_TARGET_PORT="${EXECUTOR_PORT:-4788}" · 16 unmodified lines
1717 HERMES_TARGET_PORT="${HERMES_API_PORT:-8642}"
1818 LOGS="${HB_WORKLOAD_LOGS:-/data/logs}"
1919 BOX_ENV="${HB_BOX_ENV:-/etc/hermes-box.env}"
20+SERVICES_HELPER="${HB_SERVICES_HELPER:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/tx9-services}"
21+WORKLOAD_SLEEP_PID=''
2022
2123 _exec_up() { (exec 3<>"/dev/tcp/127.0.0.1/$EXECUTOR_TARGET_PORT") 2>/dev/null && { exec 3>&-; return 0; }; return 1; }
2224 _api_up() { (exec 3<>"/dev/tcp/127.0.0.1/$HERMES_TARGET_PORT") 2>/dev/null && { exec 3>&-; return 0; }; return 1; }
@@ -54,6 +56,22 @@ _spawn_logged() { · 31 unmodified lines
5456 fi
5557 }
5658
59+_stop_custom_services() {
60+ [[ -x "$SERVICES_HELPER" ]] || return 0
61+ TX9_SERVICES_CONFIG_ROOT="$(dirname "$QUIESCE_FILE")" \
62+ TX9_SERVICES_LOG_DIR="$LOGS" "$SERVICES_HELPER" stop-all
63+}
64+
65+_shutdown() {
66+ local exit_status="$1"
67+ trap - TERM INT
68+ [[ -z "$WORKLOAD_SLEEP_PID" ]] || kill -TERM "$WORKLOAD_SLEEP_PID" 2>/dev/null || true
69+ _stop_custom_services || echo "custom service shutdown failed during workload termination" >&2
70+ _stop_executor_bridge
71+ _stop_api_bridge
72+ exit "$exit_status"
73+}
74+
5775 _gateway_reload_pending() {
5876 [ -d "$GATEWAY_RELOAD_REQUESTS" ] &&
5977 [ -n "$(find "$GATEWAY_RELOAD_REQUESTS" -maxdepth 1 -type f -name 'request.*' -print -quit)" ]
@@ -89,6 +107,10 @@ reconcile_once() { · 29 unmodified lines
89107 return
90108 fi
91109 if [ -e "$QUIESCE_FILE" ] || [ -e "$LEGACY_QUIESCE_FILE" ]; then
110+ # Reconciliation while quiesced is a stop-only operation for portable
111+ # custom services. Explicit hb pause performs the synchronous gate; this
112+ # also covers a durable marker restored or created between loop cycles.
113+ _stop_custom_services || echo "custom service shutdown reconciliation failed" >&2
92114 _stop_executor_bridge
93115 _stop_api_bridge
94116 return
@@ -121,11 +143,19 @@ reconcile_once() { · 26 unmodified lines
121143 }
122144
123145 main() {
146+ # Install lifecycle traps only for the executed workload. Regression tests
147+ # source this file to exercise reconcile_once and must retain their own
148+ # process-level trap policy.
149+ trap '_shutdown 143' TERM
150+ trap '_shutdown 130' INT
124151 mkdir -p "$LOGS"
125152 while true; do
126153 reconcile_once
127154 [[ "${HB_WORKLOAD_ONCE:-0}" != 1 ]] || break
128− sleep 20
155+ sleep 20 &
156+ WORKLOAD_SLEEP_PID=$!
157+ wait "$WORKLOAD_SLEEP_PID" 2>/dev/null || true
158+ WORKLOAD_SLEEP_PID=''
129159 done
130160 }
131161
guest/tx9-logs+187−26
@@ -10,7 +10,9 @@ from __future__ import annotations · 9 unmodified lines
1010
1111 import argparse
1212 import base64
13+import ctypes
1314 import datetime as dt
15+import errno
1416 import hashlib
1517 import heapq
1618 import io
@@ -51,6 +53,8 @@ CAPTURE_READER_DRAIN_SECONDS = 0.5 · 34 unmodified lines
5153 CAPTURE_DEDUP_WINDOW_SECONDS = 60.0
5254 CAPTURE_DEDUP_MAX_REPEATS = 1000
5355 MAX_RAW_CONTINUATION_LINES = 1000
56+PARENT_DEATH_EXEC_ACTION = "__parent-death-exec"
57+PR_SET_PDEATHSIG = 1
5458 # Numeric values below this many epoch seconds (2001-09-09) cannot be event
5559 # times; loadavg fragments and counters otherwise parse as 1970 timestamps.
5660 MIN_PLAUSIBLE_EPOCH_SECONDS = 1_000_000_000
@@ -74,6 +78,7 @@ SOURCE_ALIASES = { · 17 unmodified lines
7478 "socat": "agent",
7579 "hermes-api-socat": "hermes",
7680 }
81+SERVICE_SOURCE_RE = re.compile(r"service-[a-z0-9][a-z0-9_-]{0,62}")
7782
7883 BEARER_RE = re.compile(r"(?i)(\bbearer[ \t]+)([^\s,;\"']+)")
7984 AUTHORIZATION_RE = re.compile(r"(?i)(\b(?:proxy-)?authorization\b[ \t]*[=:][ \t]*)([^\r\n]+)")
@@ -512,6 +517,71 @@ def nonnegative_float(value: str) -> float: · 432 unmodified lines
512517 return parsed
513518
514519
520+def write_exec_failure(status_fd: int, stage: str, error: int) -> None:
521+ payload = f"{stage}:{error}\n".encode("ascii")
522+ try:
523+ os.write(status_fd, payload)
524+ except OSError:
525+ pass
526+
527+
528+def parent_death_exec(arguments: Sequence[str]) -> int:
529+ """Arm Linux parent-death SIGKILL, close the setup race, then exec.
530+
531+ Capture's supported lifecycle contract is the direct foreground child. A
532+ command that daemonizes descendants outside that child's process group is
533+ outside this guarantee.
534+ """
535+
536+ if len(arguments) < 4 or arguments[2] != "--":
537+ print("tx9-logs: invalid internal parent-death invocation", file=sys.stderr)
538+ return 126
539+ try:
540+ expected_parent = positive_int(arguments[0])
541+ status_fd = positive_int(arguments[1])
542+ except argparse.ArgumentTypeError as exc:
543+ print(f"tx9-logs: invalid internal parent-death argument: {exc}", file=sys.stderr)
544+ return 126
545+ command = list(arguments[3:])
546+ if not command:
547+ print("tx9-logs: internal parent-death exec requires a command", file=sys.stderr)
548+ return 126
549+
550+ try:
551+ # Popen clears CLOEXEC for pass_fds so this fresh interpreter can
552+ # report setup/exec errors. Restore it before the target exec: EOF on
553+ # the parent end is then the unambiguous success handshake.
554+ os.set_inheritable(status_fd, False)
555+ libc = ctypes.CDLL(None, use_errno=True)
556+ prctl = libc.prctl
557+ prctl.argtypes = [
558+ ctypes.c_int,
559+ ctypes.c_ulong,
560+ ctypes.c_ulong,
561+ ctypes.c_ulong,
562+ ctypes.c_ulong,
563+ ]
564+ prctl.restype = ctypes.c_int
565+ if prctl(PR_SET_PDEATHSIG, signal.SIGKILL, 0, 0, 0) != 0:
566+ error = ctypes.get_errno()
567+ raise OSError(error, os.strerror(error))
568+ except OSError as exc:
569+ write_exec_failure(status_fd, "setup", exc.errno or 0)
570+ return 126
571+
572+ # PR_SET_PDEATHSIG is not retroactive. If capture died between Popen's
573+ # fork and the prctl call, fail closed instead of execing an orphan.
574+ if os.getppid() != expected_parent:
575+ os.kill(os.getpid(), signal.SIGKILL)
576+ os._exit(128 + signal.SIGKILL)
577+
578+ try:
579+ os.execvpe(command[0], command, os.environ)
580+ except OSError as exc:
581+ write_exec_failure(status_fd, "exec", exc.errno or 0)
582+ return 127 if isinstance(exc, FileNotFoundError) else 126
583+
584+
515585 def path_parts_under(root: Path, path: Path) -> tuple[Path, tuple[str, ...]]:
516586 root = root.absolute()
517587 path = path.absolute()
@@ -1027,51 +1097,123 @@ class Capture: · 509 unmodified lines
10271097 child_env = dict(os.environ)
10281098 child_env["TX9_LOG_MIRROR_STDOUT_FD"] = str(self.mirror_out)
10291099 child_env["TX9_LOG_MIRROR_STDERR_FD"] = str(self.mirror_err)
1100+ # Re-exec this module as a tiny setup shim instead of using
1101+ # subprocess.preexec_fn, whose Python callback is unsafe around the
1102+ # reader threads used by capture. Popen creates the new session first;
1103+ # the fresh interpreter then arms parent death and execs the command.
1104+ status_read_fd = -1
1105+ status_write_fd = -1
1106+ reader_stop = threading.Event()
1107+ reader_cancelled = threading.Event()
1108+ readers: list[threading.Thread] = []
10301109 try:
1110+ status_read_fd, status_write_fd = os.pipe()
1111+ launch_command = [
1112+ sys.executable,
1113+ str(Path(__file__).resolve()),
1114+ PARENT_DEATH_EXEC_ACTION,
1115+ str(os.getpid()),
1116+ str(status_write_fd),
1117+ "--",
1118+ *command,
1119+ ]
10311120 child = subprocess.Popen(
1032− command,
1121+ launch_command,
10331122 stdout=subprocess.PIPE,
10341123 stderr=subprocess.PIPE,
10351124 env=child_env,
10361125 start_new_session=True,
1037− pass_fds=(self.mirror_out, self.mirror_err),
1126+ pass_fds=(self.mirror_out, self.mirror_err, status_write_fd),
10381127 )
10391128 self.child = child
10401129 self.active_pgid = child.pid
1130+ os.close(status_write_fd)
1131+ status_write_fd = -1
1132+ assert self.child.stdout is not None
1133+ assert self.child.stderr is not None
1134+ readers = [
1135+ threading.Thread(
1136+ target=self._reader,
1137+ args=("stdout", self.child.stdout, reader_stop, reader_cancelled),
1138+ daemon=True,
1139+ ),
1140+ threading.Thread(
1141+ target=self._reader,
1142+ args=("stderr", self.child.stderr, reader_stop, reader_cancelled),
1143+ daemon=True,
1144+ ),
1145+ ]
1146+ for reader in readers:
1147+ reader.start()
10411148 # A signal can arrive after Popen creates the process group but
10421149 # before the assignment above. Honor it before waiting so that
10431150 # startup cannot lose Docker's shutdown request.
10441151 if self.received_signal is not None:
10451152 self._forward_signal(self.received_signal, None)
1153+ read_fd = status_read_fd
1154+ status_read_fd = -1
1155+ with os.fdopen(read_fd, "rb", buffering=0) as status_pipe:
1156+ exec_status = status_pipe.read(128)
10461157 except OSError as exc:
1158+ if status_read_fd >= 0:
1159+ os.close(status_read_fd)
1160+ if status_write_fd >= 0:
1161+ os.close(status_write_fd)
1162+ if self.child is not None:
1163+ self._signal_process_group(signal.SIGKILL)
1164+ self.child.wait()
1165+ reader_stop.set()
1166+ for reader in readers:
1167+ reader.join()
1168+ if reader_cancelled.is_set():
1169+ self._signal_process_group(signal.SIGKILL)
1170+ self.active_pgid = None
1171+ self.child = None
1172+ self.flush_repeats()
10471173 message = f"could not start {Path(command[0]).name}: {exc}"
10481174 self.output("stderr", (message + "\n").encode())
10491175 self.structured("process_start_failed", message)
1050− return 127 if isinstance(exc, FileNotFoundError) else 126
1176+ status = 127 if isinstance(exc, FileNotFoundError) else 126
1177+ if self.received_signal is not None:
1178+ return 128 + self.received_signal
1179+ return status
1180+
1181+ if exec_status:
1182+ try:
1183+ stage, error_text = exec_status.decode("ascii").strip().split(":", 1)
1184+ error = int(error_text)
1185+ if stage not in {"setup", "exec"} or error < 0:
1186+ raise ValueError
1187+ except (UnicodeDecodeError, ValueError):
1188+ stage = "setup"
1189+ error = 0
1190+ self.child.wait()
1191+ reader_stop.set()
1192+ for reader in readers:
1193+ reader.join()
1194+ if reader_cancelled.is_set():
1195+ self._signal_process_group(signal.SIGKILL)
1196+ self.active_pgid = None
1197+ self.child = None
1198+ self.flush_repeats()
1199+ if stage == "exec":
1200+ detail = OSError(error, os.strerror(error), command[0])
1201+ status = 127 if error == errno.ENOENT else 126
1202+ else:
1203+ detail = OSError(error, os.strerror(error))
1204+ status = 126
1205+ message = f"could not start {Path(command[0]).name}: {detail}"
1206+ self.output("stderr", (message + "\n").encode())
1207+ self.structured("process_start_failed", message)
1208+ if self.received_signal is not None:
1209+ return 128 + self.received_signal
1210+ return status
10511211
10521212 self.structured(
10531213 "process_start",
10541214 f"{Path(command[0]).name} started",
10551215 {"pid": self.child.pid, "command": Path(command[0]).name},
10561216 )
1057− assert self.child.stdout is not None
1058− assert self.child.stderr is not None
1059− reader_stop = threading.Event()
1060− reader_cancelled = threading.Event()
1061− readers = [
1062− threading.Thread(
1063− target=self._reader,
1064− args=("stdout", self.child.stdout, reader_stop, reader_cancelled),
1065− daemon=True,
1066− ),
1067− threading.Thread(
1068− target=self._reader,
1069− args=("stderr", self.child.stderr, reader_stop, reader_cancelled),
1070− daemon=True,
1071− ),
1072− ]
1073− for reader in readers:
1074− reader.start()
10751217 return_code = self.child.wait()
10761218 # The direct child owns the capture lifecycle. Stop background
10771219 # descendants in its process group, then let readers drain buffered
@@ -1146,7 +1288,12 @@ def selected_sources(value: str) -> set[str] | None: · 68 unmodified lines
11461288 if not values or "all" in values:
11471289 return None
11481290 normalized = {normalize_source(item) for item in values}
1149− unknown = normalized - {"agent", "executor", "hermes", "codex", "claude"}
1291+ unknown = {
1292+ source
1293+ for source in normalized
1294+ if source not in {"agent", "executor", "hermes", "codex", "claude"}
1295+ and SERVICE_SOURCE_RE.fullmatch(source) is None
1296+ }
11501297 if unknown:
11511298 raise ValueError("unknown source(s): " + ", ".join(sorted(unknown)))
11521299 return normalized
@@ -1616,7 +1763,10 @@ def runtime_source(stem: str, scope: str) -> str: · 463 unmodified lines
16161763 return "executor"
16171764 stem = stem.partition(".legacy-")[0]
16181765 source = normalize_source(stem)
1619− if source not in {"agent", "hermes", "codex", "claude"}:
1766+ if (
1767+ source not in {"agent", "hermes", "codex", "claude"}
1768+ and SERVICE_SOURCE_RE.fullmatch(source) is None
1769+ ):
16201770 return "agent"
16211771 return source
16221772
@@ -1641,7 +1791,10 @@ def read_runtime_logs(root: Path, *, scope: str, box: str) -> Iterator[dict[str, · 18 unmodified lines
16411791 if replacement and replacement in names:
16421792 continue
16431793 source = runtime_source(active.stem, scope)
1644− if source not in {"agent", "executor", "hermes", "codex", "claude"}:
1794+ if (
1795+ source not in {"agent", "executor", "hermes", "codex", "claude"}
1796+ and SERVICE_SOURCE_RE.fullmatch(source) is None
1797+ ):
16451798 source = "agent" if scope == "agent" else "executor"
16461799 yield from read_raw(active, root=root, box=box, source=source)
16471800
@@ -1842,7 +1995,9 @@ def filtered_events(args: argparse.Namespace) -> Iterator[tuple[dt.datetime, int · 194 unmodified lines
18421995 agent_home = home_for_root(agent_root)
18431996
18441997 streams: list[Iterable[dict[str, Any]]] = []
1845− if selected is None or selected & {"agent", "hermes", "codex", "claude"}:
1998+ if selected is None or selected & {"agent", "hermes", "codex", "claude"} or any(
1999+ SERVICE_SOURCE_RE.fullmatch(source) is not None for source in selected
2000+ ):
18462001 streams.append(read_runtime_logs(agent_root, scope="agent", box=args.box))
18472002 if source_wanted("executor", selected):
18482003 streams.append(read_runtime_logs(executor_root, scope="executor", box=args.box))
@@ -2012,7 +2167,11 @@ def add_query_options(parser: argparse.ArgumentParser, *, default_tail: int | No · 163 unmodified lines
20122167 parser.add_argument("--agent-root", required=True, help="read-only agent volume root")
20132168 parser.add_argument("--executor-root", required=True, help="read-only Executor volume root")
20142169 parser.add_argument("--box", required=True, help="box name used in normalized events")
2015− parser.add_argument("--source", default="all", help="comma-separated agent,executor,hermes,codex,claude")
2170+ parser.add_argument(
2171+ "--source",
2172+ default="all",
2173+ help="comma-separated agent,executor,hermes,codex,claude,service-<name>",
2174+ )
20162175 parser.add_argument("--tail", type=lambda value: positive_int(value), default=default_tail)
20172176 parser.add_argument("--since", default="", help="duration such as 24h or RFC3339 timestamp")
20182177 parser.add_argument("--grep", default="", help="case-insensitive text filter")
@@ -2058,6 +2217,8 @@ def build_parser() -> argparse.ArgumentParser: · 39 unmodified lines
20582217
20592218
20602219 def main() -> int:
2220+ if sys.argv[1:2] == [PARENT_DEATH_EXEC_ACTION]:
2221+ return parent_death_exec(sys.argv[2:])
20612222 parser = build_parser()
20622223 args = parser.parse_args()
20632224 return int(args.handler(args))
guest/tx9-services+462−0
@@ -0,0 +1,462 @@
1+#!/usr/bin/env bash
2+# Portable guest-side service supervisor for executable drop-ins.
3+set -uo pipefail
4+
5+CONFIG_ROOT="${TX9_SERVICES_CONFIG_ROOT:-${XDG_CONFIG_HOME:-$HOME/.config}/hermes-box}"
6+DEFINITIONS="$CONFIG_ROOT/services.d"
7+STATE_DIR="${TX9_SERVICES_STATE_DIR:-${XDG_RUNTIME_DIR:-/tmp}/hermes-box-services-$(id -u)}"
8+LOG_DIR="${TX9_SERVICES_LOG_DIR:-/data/logs}"
9+QUIESCE_FILE="${TX9_SERVICES_QUIESCE_FILE:-$CONFIG_ROOT/quiesced}"
10+RESTART_DELAY="${TX9_SERVICES_RESTART_DELAY:-5}"
11+STOP_TIMEOUT="${TX9_SERVICES_STOP_TIMEOUT:-5}"
12+LOCK_WAIT="${TX9_SERVICES_LOCK_WAIT:-10}"
13+LOCK_FILE="$STATE_DIR/reconcile.lock"
14+LOCK_FD=''
15+
16+_safe_name() { [[ "$1" =~ ^[a-z0-9][a-z0-9_-]{0,62}$ ]]; }
17+
18+_ensure_dirs() {
19+ mkdir -p "$DEFINITIONS" "$STATE_DIR" "$LOG_DIR" || return 1
20+ LOG_DIR="$(cd -- "$LOG_DIR" && pwd -P)" || return 1
21+ chmod 0700 "$DEFINITIONS" 2>/dev/null || true
22+ chmod 0700 "$STATE_DIR" 2>/dev/null || true
23+}
24+
25+_log_helper() {
26+ local helper="${TX9_SERVICES_LOG_HELPER:-}"
27+ if [[ -z "$helper" ]]; then
28+ helper="$(command -v tx9-logs 2>/dev/null || true)"
29+ fi
30+ [[ -n "$helper" && "$helper" == /* && -x "$helper" ]] || {
31+ echo "tx9-services: tx9-logs is unavailable" >&2
32+ return 1
33+ }
34+ printf '%s\n' "$helper"
35+}
36+
37+_acquire_lock() {
38+ _ensure_dirs || return 1
39+ exec {LOCK_FD}>"$LOCK_FILE"
40+ if ! flock -w "$LOCK_WAIT" "$LOCK_FD"; then
41+ echo "tx9-services: could not acquire reconciliation lock" >&2
42+ eval "exec ${LOCK_FD}>&-"
43+ LOCK_FD=''
44+ return 1
45+ fi
46+}
47+
48+_release_lock() {
49+ [[ -n "$LOCK_FD" ]] || return 0
50+ flock -u "$LOCK_FD" 2>/dev/null || true
51+ eval "exec ${LOCK_FD}>&-" 2>/dev/null || true
52+ LOCK_FD=''
53+}
54+
55+_definition_fingerprint() {
56+ local path="$1" metadata digest
57+ metadata="$(stat -c '%d:%i:%s:%a' -- "$path" 2>/dev/null)" || return 1
58+ digest="$(sha256sum -- "$path" 2>/dev/null)" || return 1
59+ digest="${digest%% *}"
60+ digest="${digest#\\}"
61+ [[ "$digest" =~ ^[a-f0-9]{64}$ ]] || return 1
62+ printf '%s:%s\n' "$metadata" "$digest"
63+}
64+
65+_valid_definition() {
66+ local path="$1"
67+ [[ ! -L "$path" && -f "$path" && -x "$path" ]]
68+}
69+
70+_process_start_time() {
71+ local pid="$1" line rest
72+ local -a fields
73+ [[ "$pid" =~ ^[1-9][0-9]*$ ]] || return 1
74+ IFS= read -r line 2>/dev/null <"/proc/$pid/stat" || return 1
75+ rest="${line##*) }"
76+ read -r -a fields <<<"$rest"
77+ ((${#fields[@]} >= 20)) || return 1
78+ printf '%s\n' "${fields[19]}"
79+}
80+
81+_read_state() {
82+ local state_file="$1"
83+ IFS=$'\t' read -r STATE_PID STATE_START STATE_FINGERPRINT STATE_HELPER STATE_LOG_DIR STATE_DELAY 2>/dev/null <"$state_file" || return 1
84+ [[ "$STATE_PID" =~ ^[1-9][0-9]*$ && "$STATE_START" =~ ^[0-9]+$ && -n "$STATE_FINGERPRINT" && "$STATE_HELPER" == /* && "$STATE_LOG_DIR" == /* && -n "$STATE_DELAY" ]]
85+}
86+
87+_capture_argv_matches() {
88+ local pid="$1" name="$2" definition="$3" helper="$4" log_dir="$5" delay="$6" index
89+ local -a argv expected
90+ mapfile -d '' -t argv <"/proc/$pid/cmdline" 2>/dev/null || return 1
91+ ((${#argv[@]} > 0)) || return 1
92+ if [[ "${argv[0]}" == "$helper" ]]; then
93+ index=0
94+ elif ((${#argv[@]} > 1)) && [[ "${argv[1]}" == "$helper" ]]; then
95+ index=1
96+ else
97+ return 1
98+ fi
99+ expected=(
100+ "$helper" capture --source "service-$name" --log-dir "$log_dir"
101+ --restart-delay "$delay" -- "$definition"
102+ )
103+ ((${#argv[@]} - index == ${#expected[@]})) || return 1
104+ local offset
105+ for ((offset = 0; offset < ${#expected[@]}; offset++)); do
106+ [[ "${argv[index + offset]}" == "${expected[offset]}" ]] || return 1
107+ done
108+}
109+
110+_state_process_matches() {
111+ local state_file="$1" name="$2" definition="$3" current_start current_uid
112+ _read_state "$state_file" || return 1
113+ current_start="$(_process_start_time "$STATE_PID")" || return 1
114+ [[ "$current_start" == "$STATE_START" ]] || return 1
115+ current_uid="$(stat -c '%u' -- "/proc/$STATE_PID" 2>/dev/null)" || return 1
116+ [[ "$current_uid" == "$(id -u)" ]] || return 1
117+ _capture_argv_matches "$STATE_PID" "$name" "$definition" \
118+ "$STATE_HELPER" "$STATE_LOG_DIR" "$STATE_DELAY"
119+}
120+
121+_write_state() {
122+ local state_file="$1" pid="$2" start="$3" fingerprint="$4" helper="$5" log_dir="$6" delay="$7" tmp
123+ tmp="$state_file.tmp.$$"
124+ umask 077
125+ printf '%s\t%s\t%s\t%s\t%s\t%s\n' \
126+ "$pid" "$start" "$fingerprint" "$helper" "$log_dir" "$delay" >"$tmp" || return 1
127+ mv "$tmp" "$state_file"
128+}
129+
130+_start_one() {
131+ local name="$1" definition="$2" fingerprint="$3" helper="$4" pid start attempt state_file
132+ state_file="$STATE_DIR/$name.state"
133+ [[ -n "$helper" ]] || return 1
134+ (
135+ # flock locks are tied to open file descriptions and survive fork. The
136+ # long-lived capture must never inherit the reconciliation lock.
137+ [[ -z "$LOCK_FD" ]] || eval "exec ${LOCK_FD}>&-"
138+ exec "$helper" capture --source "service-$name" --log-dir "$LOG_DIR" \
139+ --restart-delay "$RESTART_DELAY" -- "$definition"
140+ ) </dev/null >/dev/null 2>&1 &
141+ pid=$!
142+ start="$(_process_start_time "$pid")" || {
143+ wait "$pid" 2>/dev/null || true
144+ echo "tx9-services: $name failed before supervision started" >&2
145+ return 1
146+ }
147+ _write_state "$state_file" "$pid" "$start" "$fingerprint" \
148+ "$helper" "$LOG_DIR" "$RESTART_DELAY" || {
149+ kill -TERM "$pid" 2>/dev/null || true
150+ wait "$pid" 2>/dev/null || true
151+ return 1
152+ }
153+ for ((attempt = 0; attempt < 100; attempt++)); do
154+ if _state_process_matches "$state_file" "$name" "$definition"; then
155+ echo "tx9-services: started $name"
156+ return 0
157+ fi
158+ kill -0 "$pid" 2>/dev/null || break
159+ sleep 0.02
160+ done
161+ kill -TERM "$pid" 2>/dev/null || true
162+ wait "$pid" 2>/dev/null || true
163+ rm -f "$state_file"
164+ echo "tx9-services: $name capture failed to start" >&2
165+ return 1
166+}
167+
168+_direct_children() {
169+ local pid="$1" child current_uid
170+ [[ -r "/proc/$pid/task/$pid/children" ]] || return 0
171+ for child in $(<"/proc/$pid/task/$pid/children"); do
172+ [[ "$child" =~ ^[1-9][0-9]*$ ]] || continue
173+ current_uid="$(stat -c '%u' -- "/proc/$child" 2>/dev/null)" || continue
174+ [[ "$current_uid" == "$(id -u)" ]] || continue
175+ printf '%s\n' "$child"
176+ done
177+}
178+
179+_pid_alive() {
180+ local state
181+ kill -0 "$1" 2>/dev/null || return 1
182+ state="$(ps -o stat= -p "$1" 2>/dev/null)" || return 1
183+ [[ "$state" != Z* ]]
184+}
185+
186+_pid_identity_alive() {
187+ local pid="$1" expected_start="$2" current_start
188+ _pid_alive "$pid" || return 1
189+ current_start="$(_process_start_time "$pid")" || return 1
190+ [[ "$current_start" == "$expected_start" ]]
191+}
192+
193+_process_group_identity_alive() {
194+ local leader="$1" expected_start="$2" expected_group="$3" current_group
195+ _pid_identity_alive "$leader" "$expected_start" || return 1
196+ current_group="$(ps -o pgid= -p "$leader" 2>/dev/null)" || return 1
197+ current_group="${current_group//[[:space:]]/}"
198+ [[ "$current_group" == "$expected_group" ]]
199+}
200+
201+_supervised_child_running() {
202+ local capture_pid="$1" child start current_start
203+ local -a children
204+ mapfile -t children < <(_direct_children "$capture_pid")
205+ ((${#children[@]} == 1)) || return 1
206+ child="${children[0]}"
207+ _pid_alive "$child" || return 1
208+ start="$(_process_start_time "$child")" || return 1
209+ # A child that merely flashes into existence during a crash loop is not a
210+ # running service. Require the same direct child to remain present across a
211+ # short observation window; this is process-state reporting, not a generic
212+ # application health check.
213+ sleep 0.05
214+ current_start="$(_process_start_time "$child")" || return 1
215+ [[ "$current_start" == "$start" ]] || return 1
216+ _pid_alive "$child" || return 1
217+ [[ "$(sed -n 's/^PPid:[[:space:]]*//p' "/proc/$child/status" 2>/dev/null)" == "$capture_pid" ]]
218+}
219+
220+_stop_one() {
221+ local name="$1" definition="$2" attempt child start group index alive failed=0
222+ local state_file="$STATE_DIR/$name.state"
223+ local -a child_groups=()
224+ local -a child_starts=()
225+ if ! _state_process_matches "$state_file" "$name" "$definition"; then
226+ rm -f "$state_file"
227+ return 0
228+ fi
229+
230+ while IFS= read -r child; do
231+ start="$(_process_start_time "$child")" || continue
232+ group="$(ps -o pgid= -p "$child" 2>/dev/null)" || continue
233+ group="${group//[[:space:]]/}"
234+ [[ "$group" == "$child" ]] || continue
235+ _process_group_identity_alive "$child" "$start" "$group" || continue
236+ child_groups+=("$group")
237+ child_starts+=("$start")
238+ done < <(_direct_children "$STATE_PID")
239+
240+ if ! _pid_identity_alive "$STATE_PID" "$STATE_START"; then
241+ rm -f "$state_file"
242+ return 0
243+ fi
244+ kill -TERM "$STATE_PID" 2>/dev/null || true
245+ for ((attempt = 0; attempt < STOP_TIMEOUT * 10; attempt++)); do
246+ alive=0
247+ _pid_identity_alive "$STATE_PID" "$STATE_START" && alive=1
248+ for ((index = 0; index < ${#child_groups[@]}; index++)); do
249+ _process_group_identity_alive \
250+ "${child_groups[index]}" "${child_starts[index]}" "${child_groups[index]}" && alive=1
251+ done
252+ [[ "$alive" == 1 ]] || break
253+ sleep 0.1
254+ done
255+
256+ for ((index = 0; index < ${#child_groups[@]}; index++)); do
257+ if _process_group_identity_alive \
258+ "${child_groups[index]}" "${child_starts[index]}" "${child_groups[index]}"; then
259+ kill -KILL -- "-${child_groups[index]}" 2>/dev/null || true
260+ fi
261+ done
262+ if _pid_identity_alive "$STATE_PID" "$STATE_START"; then
263+ kill -KILL "$STATE_PID" 2>/dev/null || true
264+ fi
265+
266+ for ((attempt = 0; attempt < 20; attempt++)); do
267+ alive=0
268+ _pid_identity_alive "$STATE_PID" "$STATE_START" && alive=1
269+ for ((index = 0; index < ${#child_groups[@]}; index++)); do
270+ _process_group_identity_alive \
271+ "${child_groups[index]}" "${child_starts[index]}" "${child_groups[index]}" && alive=1
272+ done
273+ [[ "$alive" == 1 ]] || break
274+ sleep 0.05
275+ done
276+
277+ if _pid_identity_alive "$STATE_PID" "$STATE_START"; then
278+ echo "tx9-services: $name capture survived stop" >&2
279+ failed=1
280+ fi
281+ for ((index = 0; index < ${#child_groups[@]}; index++)); do
282+ if _process_group_identity_alive \
283+ "${child_groups[index]}" "${child_starts[index]}" "${child_groups[index]}"; then
284+ echo "tx9-services: $name process group survived stop" >&2
285+ failed=1
286+ fi
287+ done
288+ if [[ "$failed" == 0 ]]; then
289+ rm -f "$state_file"
290+ echo "tx9-services: stopped $name"
291+ fi
292+ return "$failed"
293+}
294+
295+_definition_reason() {
296+ local path="$1" name="$2"
297+ if ! _safe_name "$name"; then printf 'unsafe name'; return; fi
298+ if [[ -L "$path" ]]; then printf 'symlink'; return; fi
299+ if [[ ! -f "$path" ]]; then printf 'not a regular file'; return; fi
300+ if [[ ! -x "$path" ]]; then printf 'not executable'; return; fi
301+ printf 'valid'
302+}
303+
304+_reconcile_locked() {
305+ local path name fingerprint state_file current_helper='' helper_available=1 failed=0
306+ local -A desired=()
307+ while IFS= read -r -d '' path; do
308+ name="${path##*/}"
309+ if ! _valid_definition "$path" || ! _safe_name "$name"; then
310+ echo "tx9-services: ignored $name ($(_definition_reason "$path" "$name"))" >&2
311+ continue
312+ fi
313+ fingerprint="$(_definition_fingerprint "$path")" || {
314+ echo "tx9-services: ignored $name (could not inspect definition)" >&2
315+ continue
316+ }
317+ desired["$name"]="$fingerprint"
318+ done < <(find "$DEFINITIONS" -mindepth 1 -maxdepth 1 -print0 2>/dev/null)
319+
320+ if ((${#desired[@]} > 0)); then
321+ if ! current_helper="$(_log_helper)"; then
322+ helper_available=0
323+ failed=1
324+ fi
325+ fi
326+
327+ shopt -s nullglob
328+ for state_file in "$STATE_DIR"/*.state; do
329+ name="${state_file##*/}"
330+ name="${name%.state}"
331+ path="$DEFINITIONS/$name"
332+ if [[ -z "${desired[$name]+yes}" ]]; then
333+ _stop_one "$name" "$path" || failed=1
334+ continue
335+ fi
336+ # A helper is required to replace a configured supervisor. Preserve the
337+ # existing launch when helper resolution is transiently unavailable;
338+ # otherwise stopping it here would turn a recoverable error into downtime.
339+ [[ "$helper_available" == 1 ]] || continue
340+ if _state_process_matches "$state_file" "$name" "$path" &&
341+ [[ "$STATE_FINGERPRINT" == "${desired[$name]}" &&
342+ "$STATE_HELPER" == "$current_helper" && "$STATE_LOG_DIR" == "$LOG_DIR" &&
343+ "$STATE_DELAY" == "$RESTART_DELAY" ]]; then
344+ continue
345+ fi
346+ _stop_one "$name" "$path" || failed=1
347+ done
348+ shopt -u nullglob
349+
350+ [[ ! -e "$QUIESCE_FILE" ]] || return "$failed"
351+ for name in "${!desired[@]}"; do
352+ state_file="$STATE_DIR/$name.state"
353+ path="$DEFINITIONS/$name"
354+ [[ "$helper_available" == 1 ]] || continue
355+ if _state_process_matches "$state_file" "$name" "$path" &&
356+ [[ "$STATE_FINGERPRINT" == "${desired[$name]}" &&
357+ "$STATE_HELPER" == "$current_helper" && "$STATE_LOG_DIR" == "$LOG_DIR" &&
358+ "$STATE_DELAY" == "$RESTART_DELAY" ]]; then
359+ continue
360+ fi
361+ _start_one "$name" "$path" "${desired[$name]}" "$current_helper" || failed=1
362+ done
363+ return "$failed"
364+}
365+
366+reconcile() {
367+ _acquire_lock || return 1
368+ _reconcile_locked
369+ local status=$?
370+ _release_lock
371+ return "$status"
372+}
373+
374+_stop_all_locked() {
375+ local state_file name definition failed=0
376+ shopt -s nullglob
377+ for state_file in "$STATE_DIR"/*.state; do
378+ name="${state_file##*/}"
379+ name="${name%.state}"
380+ definition="$DEFINITIONS/$name"
381+ _stop_one "$name" "$definition" || failed=1
382+ done
383+ shopt -u nullglob
384+ return "$failed"
385+}
386+
387+stop_all() {
388+ _acquire_lock || return 1
389+ _stop_all_locked
390+ local status=$?
391+ _release_lock
392+ return "$status"
393+}
394+
395+_status_locked() {
396+ local path name reason fingerprint state_file current_helper rows=0
397+ local -A seen=()
398+ current_helper="$(_log_helper 2>/dev/null || true)"
399+ printf 'NAME\tSTATUS\tDEFINITION\n'
400+ while IFS= read -r -d '' path; do
401+ name="${path##*/}"
402+ seen["$name"]=1
403+ rows=$((rows + 1))
404+ reason="$(_definition_reason "$path" "$name")"
405+ if [[ "$reason" != valid ]]; then
406+ printf '%s\tignored (%s)\t%s\n' "$name" "$reason" "$path"
407+ continue
408+ fi
409+ state_file="$STATE_DIR/$name.state"
410+ fingerprint="$(_definition_fingerprint "$path" 2>/dev/null || true)"
411+ if _state_process_matches "$state_file" "$name" "$path" && _read_state "$state_file"; then
412+ if [[ "$STATE_FINGERPRINT" == "$fingerprint" &&
413+ "$STATE_HELPER" == "$current_helper" && "$STATE_LOG_DIR" == "$LOG_DIR" &&
414+ "$STATE_DELAY" == "$RESTART_DELAY" ]]; then
415+ if _supervised_child_running "$STATE_PID"; then
416+ printf '%s\trunning\t%s\n' "$name" "$path"
417+ else
418+ printf '%s\trestarting\t%s\n' "$name" "$path"
419+ fi
420+ else
421+ printf '%s\treload pending\t%s\n' "$name" "$path"
422+ fi
423+ elif [[ -e "$QUIESCE_FILE" ]]; then
424+ printf '%s\tstopped (quiesced)\t%s\n' "$name" "$path"
425+ else
426+ printf '%s\tstopped\t%s\n' "$name" "$path"
427+ fi
428+ done < <(find "$DEFINITIONS" -mindepth 1 -maxdepth 1 -print0 2>/dev/null)
429+
430+ shopt -s nullglob
431+ for state_file in "$STATE_DIR"/*.state; do
432+ name="${state_file##*/}"
433+ name="${name%.state}"
434+ if [[ -z "${seen[$name]+yes}" ]]; then
435+ rows=$((rows + 1))
436+ printf '%s\tstale state\t%s\n' "$name" "$DEFINITIONS/$name"
437+ fi
438+ done
439+ shopt -u nullglob
440+ if [[ "$rows" == 0 ]]; then
441+ printf 'No custom services configured. Add executable files to %s\n' "$DEFINITIONS"
442+ fi
443+}
444+
445+status() {
446+ _acquire_lock || return 1
447+ _status_locked
448+ local result=$?
449+ _release_lock
450+ return "$result"
451+}
452+
453+main() {
454+ case "${1:-status}" in
455+ reconcile | reload) reconcile ;;
456+ stop-all) stop_all ;;
457+ status) status ;;
458+ *) echo 'usage: tx9-services {status|reconcile|reload|stop-all}' >&2; return 2 ;;
459+ esac
460+}
461+
462+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then main "$@"; fi
internal/assets/assets.go+1−1
@@ -102,7 +102,7 @@ func fileMode(p string) int64 { · 101 unmodified lines
102102 switch {
103103 case strings.HasSuffix(p, ".sh"):
104104 return 0o755
105− case p == "guest/hb" || strings.HasPrefix(p, "guest/hb-"):
105+ case p == "guest/hb" || strings.HasPrefix(p, "guest/hb-") || p == "guest/tx9-services":
106106 return 0o755
107107 case p == "guest/hermes-state":
108108 return 0o755
internal/assets/assets_test.go+4−1
@@ -6,6 +6,7 @@ import ( · 5 unmodified lines
66 "io/fs"
77 "os"
88 "path/filepath"
9+ "strings"
910 "testing"
1011 )
1112
@@ -51,9 +52,11 @@ func TestBuildContextTar(t *testing.T) { · 39 unmodified lines
5152 if mode != 0o755 {
5253 t.Errorf("guest/hb mode = %o, want 0755", mode)
5354 }
54− for _, want := range []string{"box.env", "provision/provision.sh", "docker/entrypoint.sh", "docker/executor-entrypoint.sh", "guest/hb-workload", "guest/hermes-state"} {
55+ for _, want := range []string{"box.env", "provision/provision.sh", "docker/entrypoint.sh", "docker/executor-entrypoint.sh", "guest/hb-workload", "guest/tx9-services", "guest/hermes-state"} {
5556 if _, ok := modes[want]; !ok {
5657 t.Errorf("%s missing from build context tar", want)
58+ } else if strings.HasPrefix(want, "guest/") && modes[want] != 0o755 {
59+ t.Errorf("%s mode = %o, want 0755", want, modes[want])
5760 }
5861 }
5962 for name := range modes {
internal/cli/cmd_logs.go+2−2
@@ -42,7 +42,7 @@ func cmdLogs(args []string) error { · 41 unmodified lines
4242
4343 fs := flag.NewFlagSet("logs", flag.ContinueOnError)
4444 opts := logsQueryOptions{}
45− fs.StringVar(&opts.Source, "source", "all", "comma-separated sources: agent,executor,hermes,codex,claude")
45+ fs.StringVar(&opts.Source, "source", "all", "comma-separated sources: agent,executor,hermes,codex,claude,service-<name>")
4646 fs.IntVar(&opts.Tail, "tail", 200, "show the newest N matching events")
4747 fs.StringVar(&opts.Since, "since", "", "only events after a duration or RFC3339 timestamp (for example 24h)")
4848 fs.StringVar(&opts.Contains, "grep", "", "only events containing text")
@@ -89,7 +89,7 @@ func cmdLogsExport(args []string) error { · 40 unmodified lines
8989 fs := flag.NewFlagSet("logs export", flag.ContinueOnError)
9090 opts := logsQueryOptions{}
9191 output := fs.String("output", "", "output .tar.gz path (default: <box>-logs-<timestamp>.tar.gz)")
92− fs.StringVar(&opts.Source, "source", "all", "comma-separated sources: agent,executor,hermes,codex,claude")
92+ fs.StringVar(&opts.Source, "source", "all", "comma-separated sources: agent,executor,hermes,codex,claude,service-<name>")
9393 fs.StringVar(&opts.Since, "since", "", "only events after a duration or RFC3339 timestamp (for example 24h)")
9494 fs.StringVar(&opts.Contains, "grep", "", "only events containing text")
9595 fs.StringVar(&opts.Level, "level", "", "only events at this level or above: debug|info|warn|error (unleveled events count as info)")
internal/cli/dispatch.go+1−1
@@ -41,7 +41,7 @@ var commandSpecs = []commandSpec{ · 40 unmodified lines
4141 {name: "enter", help: "exec into a box's agent container (--executor for the executor container)", aliases: []string{"ssh", "shell"}, run: cmdEnter},
4242 {name: "start", help: "start a stopped box", run: cmdStart},
4343 {name: "stop", help: "stop a running box", run: cmdStop},
44− {name: "logs", help: "query/export durable agent, Executor, Hermes, Codex, and Claude events", run: cmdLogs},
44+ {name: "logs", help: "query/export durable agent, Executor, Hermes, Codex, Claude, and custom-service events", run: cmdLogs},
4545 {name: "resources", help: "show/update container limits and volume budgets", run: cmdResources},
4646 {name: "backup", help: "archive a box to a .tx9 file", aliases: []string{"export", "save"}, run: cmdBackup},
4747 {name: "import", help: "restore a box from a .tx9 file", aliases: []string{"load", "restore"}, run: cmdImport},
internal/cli/dispatch_test.go+8−0
@@ -59,6 +59,14 @@ func TestUsageShowsAliasesFromCommandSpecs(t *testing.T) { · 58 unmodified lines
5959 }
6060 }
6161
62+func TestUsageMentionsCustomServiceLogs(t *testing.T) {
63+ var usage bytes.Buffer
64+ printUsage(&usage)
65+ if !strings.Contains(usage.String(), "custom-service events") {
66+ t.Fatalf("logs help does not mention custom services:\n%s", usage.String())
67+ }
68+}
69+
6270 func TestNoArgumentsShowsOverviewAndCommands(t *testing.T) {
6371 var stdout, stderr bytes.Buffer
6472 status := runWithOverview([]string{"tx9"}, nil, func(w io.Writer) error {
provision/provision.sh+2−1
@@ -220,12 +220,13 @@ install_config() { · 219 unmodified lines
220220 }
221221
222222 place_assets() {
223− log "profile, tmux config, hb and log helpers"
223+ log "profile, tmux config, hb, service, and log helpers"
224224 install -m 0644 "$CTX/guest/profile.sh" /etc/profile.d/hermes-box.sh
225225 install -m 0644 "$CTX/guest/tmux.conf" "$OPT/tmux.conf"
226226 install -m 0644 "$CTX/guest/lib-mcp.sh" "$OPT/bin/lib-mcp.sh"
227227 install -m 0755 "$CTX/guest/hb" "$OPT/bin/hb"
228228 install -m 0755 "$CTX/guest/hb-workload" "$OPT/bin/hb-workload"
229+ install -m 0755 "$CTX/guest/tx9-services" "$OPT/bin/tx9-services"
229230 install -m 0755 "$CTX/guest/hermes-state" "$OPT/bin/hermes-state"
230231 install -m 0755 "$CTX/guest/tx9-logs" "$OPT/bin/tx9-logs"
231232 # agent login shell auto-attaches tmux; see guest/agent-bash-profile.sh
tests/regressions-hb-workload.sh+107−0
@@ -359,6 +359,92 @@ HB_DATA="$gateway_data" "$PROJECT_ROOT/guest/hb" gateway-enable \ · 358 unmodified lines
359359 [[ ! -e "$tmp/up-gateway-started" ]]
360360 )
361361
362+# Interactive startup reports a custom-service failure after still starting
363+# Executor and Hermes. Resume preserves the same contract via hb up.
364+(
365+ HB_DATA="$tmp/up-service-failure-data"
366+ HOME="$HB_DATA/home/agent"
367+ mkdir -p "$HOME"
368+ # shellcheck disable=SC1090
369+ source "$PROJECT_ROOT/guest/hb"
370+
371+ calls="$tmp/up-service-failure.calls"
372+ init() { mkdir -p "$STATE_DIR"; }
373+ _services_reconcile() {
374+ echo services >>"$calls"
375+ return 1
376+ }
377+ _executor_up() {
378+ echo executor >>"$calls"
379+ return 0
380+ }
381+ _start_gateway() {
382+ echo gateway >>"$calls"
383+ return 0
384+ }
385+
386+ if up >"$tmp/up-service-failure.stdout" 2>"$tmp/up-service-failure.stderr"; then
387+ echo "hb up masked custom-service reconciliation failure" >&2
388+ exit 1
389+ fi
390+ [[ "$(cat "$calls")" == $'services\nexecutor\ngateway' ]]
391+ grep -q 'custom service reconciliation failed' "$tmp/up-service-failure.stderr"
392+
393+ : >"$calls"
394+ touch "$QUIESCE_FILE"
395+ if resume >"$tmp/resume-service-failure.stdout" 2>"$tmp/resume-service-failure.stderr"; then
396+ echo "hb resume masked custom-service reconciliation failure" >&2
397+ exit 1
398+ fi
399+ [[ ! -e "$QUIESCE_FILE" ]]
400+ [[ "$(cat "$calls")" == $'services\nexecutor\ngateway' ]]
401+ grep -q 'custom service reconciliation failed' "$tmp/resume-service-failure.stderr"
402+)
403+
404+# Custom-service failures are isolated from the first-party lifecycle. Both
405+# core reconciliation steps still run, hb reconcile returns their success,
406+# and hb-workload is allowed to bring up the Hermes API bridge.
407+(
408+ HB_DATA="$tmp/service-isolation-data"
409+ HOME="$HB_DATA/home/agent"
410+ mkdir -p "$HOME/.config/hermes-box" "$tmp/service-isolation-logs"
411+ # shellcheck disable=SC1090
412+ source "$PROJECT_ROOT/guest/hb"
413+ init() { :; }
414+ _services_reconcile() { touch "$tmp/service-isolation-services"; return 1; }
415+ _executor_up() { touch "$tmp/service-isolation-executor"; return 0; }
416+ _start_gateway() { touch "$tmp/service-isolation-gateway"; return 0; }
417+ reconcile >"$tmp/service-isolation.stdout" 2>"$tmp/service-isolation.stderr"
418+ [[ -e "$tmp/service-isolation-services" ]]
419+ [[ -e "$tmp/service-isolation-executor" ]]
420+ [[ -e "$tmp/service-isolation-gateway" ]]
421+ grep -q 'continuing core reconciliation' "$tmp/service-isolation.stderr"
422+ services() { touch "$tmp/service-isolation-status"; return 0; }
423+ if services_reload; then
424+ echo "services-reload masked custom reconciliation failure" >&2
425+ exit 1
426+ fi
427+ [[ -e "$tmp/service-isolation-status" ]]
428+
429+ # shellcheck disable=SC1090
430+ source "$PROJECT_ROOT/guest/hb-workload"
431+ PORT=4788
432+ HERMES_BRIDGE_PORT=8659
433+ LOGS="$tmp/service-isolation-logs"
434+ BOX_ENV="$tmp/service-isolation-box.env"
435+ LEGACY_QUIESCE_FILE="$tmp/service-isolation-legacy"
436+ printf 'EXECUTOR_PORT=4788\nHERMES_API_PORT=8642\n' >"$BOX_ENV"
437+ rm -f "$GATEWAY_DISABLED" "$QUIESCE_FILE"
438+ hb() { [[ "$1" == reconcile ]] && reconcile; }
439+ pgrep() { return 1; }
440+ pkill() { :; }
441+ _wait_api_ready() { return 0; }
442+ socat() { printf '%s\n' "$*" >>"$tmp/service-isolation-bridge"; }
443+ reconcile_once
444+ wait
445+ grep -q 'TCP-LISTEN:8659,' "$tmp/service-isolation-bridge"
446+)
447+
362448 # The reconcile loop re-checks Executor every cycle; unchanged reachability
363449 # must log only on state changes so the durable agent stream is not flooded
364450 # with one identical line per cycle. Interactive paths always report.
@@ -415,6 +501,27 @@ HB_DATA="$gateway_data" "$PROJECT_ROOT/guest/hb" gateway-enable \ · 50 unmodified lines
415501 [[ ! -e "$tmp/hermes-state-called" ]]
416502 )
417503
504+# Top-level status links to custom-service detail, while doctor explicitly
505+# avoids presenting arbitrary processes as generically health-checkable.
506+(
507+ HB_DATA="$tmp/services-discoverability-data"
508+ HERMES_HOME="$HB_DATA/home/agent/.hermes"
509+ # shellcheck disable=SC1090
510+ source "$PROJECT_ROOT/guest/hb"
511+ init
512+ _port_open() { return 1; }
513+ _gateway_running() { return 1; }
514+ status >"$tmp/services-discoverability.status"
515+ grep -Fq "services: 0 drop-in entries (run 'hb services')" \
516+ "$tmp/services-discoverability.status"
517+ _check() { return 0; }
518+ INSTALL_HERMES=0 INSTALL_EXECUTOR=0 WIRE_EXECUTOR_MCP=0 \
519+ doctor >"$tmp/services-discoverability.doctor"
520+ grep -Fq "custom services have no generic health contract" \
521+ "$tmp/services-discoverability.doctor"
522+ grep -Fq "'hb services'" "$tmp/services-discoverability.doctor"
523+)
524+
418525 # Doctor validates Codex MCP wiring against the configured Executor port.
419526 (
420527 HB_DATA="$tmp/doctor-port-data"
tests/regressions-logs.sh+106−0
@@ -51,6 +51,80 @@ args = parser.parse_args(["capture", "--source", "agent", "--", "true"]) · 50 unmodified lines
5151 assert args.restart_delay is None
5252 PY
5353
54+# The parent-death re-exec shim preserves Popen's lifecycle contract: actual
55+# exec failures are process_start_failed events, never successful starts.
56+start_failure_root="$tmp/start-failures"
57+mkdir -p "$start_failure_root"
58+printf '#!/usr/bin/env bash\nexit 0\n' >"$start_failure_root/non-executable"
59+chmod 0600 "$start_failure_root/non-executable"
60+printf '#!/definitely/missing/tx9-interpreter\nexit 0\n' \
61+ >"$start_failure_root/bad-shebang"
62+chmod 0700 "$start_failure_root/bad-shebang"
63+
64+assert_start_failed() {
65+ local label="$1" expected_status="$2" status log_dir
66+ shift 2
67+ log_dir="$start_failure_root/$label-logs"
68+ set +e
69+ "$helper" capture --source agent --log-dir "$log_dir" -- "$@" \
70+ >"$start_failure_root/$label.stdout" 2>"$start_failure_root/$label.stderr"
71+ status=$?
72+ set -e
73+ [[ "$status" == "$expected_status" ]]
74+ jq -e 'select(.type == "process_start_failed")' "$log_dir/agent.jsonl" >/dev/null
75+ if jq -e 'select(.type == "process_start")' "$log_dir/agent.jsonl" >/dev/null; then
76+ echo "$label exec failure was reported as a successful process start" >&2
77+ exit 1
78+ fi
79+}
80+
81+assert_start_failed missing 127 tx9-definitely-missing-command
82+assert_start_failed non-executable 126 "$start_failure_root/non-executable"
83+assert_start_failed bad-shebang 127 "$start_failure_root/bad-shebang"
84+
85+# The shim can emit more than a pipe buffer before exec (for example Python's
86+# verbose import trace). Readers must drain concurrently with the handshake.
87+verbose_start_dir="$start_failure_root/verbose-start-logs"
88+if ! timeout 15 env PYTHONVERBOSE=2 \
89+ "$helper" capture --source agent --log-dir "$verbose_start_dir" -- true \
90+ >"$start_failure_root/verbose-start.stdout" \
91+ 2>"$start_failure_root/verbose-start.stderr"; then
92+ echo "capture deadlocked while the exec shim filled stderr" >&2
93+ exit 1
94+fi
95+jq -e 'select(.type == "process_start")' "$verbose_start_dir/agent.jsonl" >/dev/null
96+jq -e 'select(.type == "process_exit" and .data.status == 0)' \
97+ "$verbose_start_dir/agent.jsonl" >/dev/null
98+
99+# A signal received while startup is resolving overrides an exec failure's
100+# 126/127 status just as it overrides a normal child exit.
101+python3 - "$helper" "$start_failure_root/signal-override-logs" \
102+ >"$start_failure_root/signal-override.stdout" \
103+ 2>"$start_failure_root/signal-override.stderr" <<'PY'
104+import importlib.machinery
105+import importlib.util
106+import pathlib
107+import signal
108+import sys
109+
110+loader = importlib.machinery.SourceFileLoader("tx9_logs_signal_override", sys.argv[1])
111+spec = importlib.util.spec_from_loader(loader.name, loader)
112+assert spec is not None
113+module = importlib.util.module_from_spec(spec)
114+loader.exec_module(module)
115+capture = module.Capture("agent", pathlib.Path(sys.argv[2]), 1024 * 1024, 2)
116+capture.received_signal = signal.SIGTERM
117+# Preserve the already-recorded signal without killing the short-lived shim,
118+# making the exec-failure branch deterministic.
119+capture._forward_signal = lambda _signum, _frame: None
120+try:
121+ assert capture.run_once(["tx9-definitely-missing-on-signal"]) == 143
122+finally:
123+ capture.close()
124+PY
125+jq -e 'select(.type == "process_start_failed")' \
126+ "$start_failure_root/signal-override-logs/agent.jsonl" >/dev/null
127+
54128 # Source reads stop at the first unterminated snapshot boundary instead of
55129 # reframing bytes appended during the query as a separate record. A record that
56130 # exactly fills the byte budget is retained; only a larger record is oversized.
@@ -278,6 +352,38 @@ signal_status=$? · 221 unmodified lines
278352 set -e
279353 [[ "$signal_status" == 143 ]]
280354
355+# Capture arms its direct foreground child with a Linux parent-death SIGKILL.
356+# Abrupt wrapper death therefore cannot leave the supported command orphaned.
357+pdeath_dir="$tmp/parent-death"
358+pdeath_pid_file="$tmp/parent-death.pid"
359+PDEATH_PID_FILE="$pdeath_pid_file" \
360+ "$helper" capture --source agent --log-dir "$pdeath_dir" -- \
361+ python3 -c 'import os, pathlib, signal; pathlib.Path(os.environ["PDEATH_PID_FILE"]).write_text(str(os.getpid())); signal.pause()' \
362+ >/dev/null 2>&1 &
363+pdeath_capture_pid=$!
364+pids+=("$pdeath_capture_pid")
365+wait_for_file "$pdeath_pid_file"
366+pdeath_child_pid="$(cat "$pdeath_pid_file")"
367+kill -0 "$pdeath_child_pid"
368+kill -KILL "$pdeath_capture_pid"
369+set +e
370+wait "$pdeath_capture_pid" 2>/dev/null
371+pdeath_capture_status=$?
372+set -e
373+[[ "$pdeath_capture_status" == 137 ]]
374+for _ in {1..150}; do
375+ if ! kill -0 "$pdeath_child_pid" 2>/dev/null ||
376+ [[ "$(ps -o stat= -p "$pdeath_child_pid" 2>/dev/null)" == Z* ]]; then
377+ break
378+ fi
379+ sleep 0.02
380+done
381+if kill -0 "$pdeath_child_pid" 2>/dev/null &&
382+ [[ "$(ps -o stat= -p "$pdeath_child_pid" 2>/dev/null)" != Z* ]]; then
383+ echo "captured foreground child survived SIGKILL of its wrapper" >&2
384+ exit 1
385+fi
386+
281387 # Internal cleanup gives a same-group descendant a bounded TERM grace before
282388 # cancellation/escalation, so normal shutdown handlers can flush state.
283389 graceful_dir="$tmp/graceful-descendant"
tests/regressions-services.sh+630−0
@@ -0,0 +1,630 @@
1+#!/usr/bin/env bash
2+# shellcheck disable=SC2016,SC2030,SC2031,SC2329
3+set -euo pipefail
4+
5+# shellcheck source=tests/lib.sh
6+source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib.sh"
7+
8+data="$tmp/data"
9+home="$data/home/agent"
10+config_root="$home/.config/hermes-box"
11+definitions="$config_root/services.d"
12+runtime="$tmp/runtime"
13+logs="$data/logs"
14+events="$tmp/events"
15+mkdir -p "$definitions" "$runtime" "$logs" "$events"
16+
17+export TX9_SERVICES_CONFIG_ROOT="$config_root"
18+export TX9_SERVICES_STATE_DIR="$runtime"
19+export TX9_SERVICES_LOG_DIR="$logs"
20+export TX9_SERVICES_LOG_HELPER="$PROJECT_ROOT/guest/tx9-logs"
21+export TX9_SERVICES_RESTART_DELAY=0.2
22+export TX9_SERVICES_STOP_TIMEOUT=2
23+export SERVICE_TEST_EVENTS="$events"
24+services="$PROJECT_ROOT/guest/tx9-services"
25+
26+wait_for_count() {
27+ local expected="$1" file="$2" attempt count
28+ for ((attempt = 0; attempt < 150; attempt++)); do
29+ if [[ -f "$file" ]]; then count="$(wc -l <"$file")"; else count=0; fi
30+ [[ "$count" -ge "$expected" ]] && return 0
31+ sleep 0.02
32+ done
33+ echo "timed out waiting for $expected records in $file" >&2
34+ return 1
35+}
36+
37+wait_for_absent() {
38+ local path="$1" attempt
39+ for ((attempt = 0; attempt < 150; attempt++)); do
40+ [[ ! -e "$path" ]] && return 0
41+ sleep 0.02
42+ done
43+ echo "timed out waiting for $path to disappear" >&2
44+ return 1
45+}
46+
47+wait_for_process_exit() {
48+ local pid="$1" attempt state
49+ for ((attempt = 0; attempt < 150; attempt++)); do
50+ if ! kill -0 "$pid" 2>/dev/null; then return 0; fi
51+ state="$(ps -o stat= -p "$pid" 2>/dev/null || true)"
52+ [[ "$state" == Z* ]] && return 0
53+ sleep 0.02
54+ done
55+ echo "timed out waiting for process $pid to exit" >&2
56+ return 1
57+}
58+
59+cat >"$definitions/healthy" <<'EOF'
60+#!/usr/bin/env bash
61+printf 'start %s\n' "$$" >>"$SERVICE_TEST_EVENTS/healthy.starts"
62+printf 'healthy-log\n'
63+trap 'printf "term %s\n" "$$" >>"$SERVICE_TEST_EVENTS/healthy.terms"; exit 0' TERM INT
64+while :; do sleep 1; done
65+EOF
66+chmod 0700 "$definitions/healthy"
67+
68+cat >"$definitions/broken" <<'EOF'
69+#!/usr/bin/env bash
70+printf 'start %s\n' "$$" >>"$SERVICE_TEST_EVENTS/broken.starts"
71+printf 'broken-log\n'
72+exit 17
73+EOF
74+chmod 0700 "$definitions/broken"
75+
76+# Invalid direct children are ignored and reported: unsafe name, non-executable
77+# regular file, directory, and symlink.
78+printf '#!/usr/bin/env bash\nexit 0\n' >"$definitions/BadName"
79+chmod 0700 "$definitions/BadName"
80+printf '#!/usr/bin/env bash\nexit 0\n' >"$definitions/disabled"
81+mkdir "$definitions/not-a-file"
82+ln -s "$definitions/healthy" "$definitions/linked"
83+
84+"$services" reconcile
85+wait_for_count 1 "$events/healthy.starts"
86+wait_for_count 2 "$events/broken.starts"
87+[[ -s "$runtime/healthy.state" ]]
88+[[ -s "$runtime/broken.state" ]]
89+if ! flock -n "$runtime/reconcile.lock" -c true; then
90+ echo "service capture inherited and pinned the reconciliation lock" >&2
91+ exit 1
92+fi
93+
94+# A stale or forged PID record is never authority to kill an arbitrary
95+# same-user process. PID identity and the exact capture argv must both match.
96+sleep 30 &
97+decoy_pid=$!
98+pids+=("$decoy_pid")
99+decoy_start="$(awk '{print $22}' "/proc/$decoy_pid/stat")"
100+printf '%s\t%s\tstale\t%s\t%s\t%s\n' "$decoy_pid" "$decoy_start" \
101+ "$TX9_SERVICES_LOG_HELPER" "$TX9_SERVICES_LOG_DIR" \
102+ "$TX9_SERVICES_RESTART_DELAY" \
103+ >"$runtime/ghost.state"
104+"$services" reconcile >/dev/null 2>&1
105+kill -0 "$decoy_pid"
106+[[ ! -e "$runtime/ghost.state" ]]
107+kill "$decoy_pid"
108+wait "$decoy_pid" 2>/dev/null || true
109+
110+# Stop escalation remains bound to the exact capture and service identities
111+# observed before TERM. Reused numeric PIDs/PGIDs are never sent SIGKILL.
112+(
113+ # shellcheck disable=SC1090
114+ source "$services"
115+ STATE_DIR="$tmp/reused-stop-runtime"
116+ STOP_TIMEOUT=1
117+ mkdir -p "$STATE_DIR"
118+ touch "$STATE_DIR/reused.state"
119+ signal_log="$tmp/reused-stop.signals"
120+ capture_reads="$tmp/reused-capture.reads"
121+ child_reads="$tmp/reused-child.reads"
122+
123+ _state_process_matches() {
124+ STATE_PID=4242
125+ STATE_START=100
126+ return 0
127+ }
128+ _direct_children() { printf '6262\n'; }
129+ _pid_alive() { return 0; }
130+ _process_start_time() {
131+ local pid="$1" reads_file count=0
132+ if [[ "$pid" == 4242 ]]; then
133+ reads_file="$capture_reads"
134+ [[ -f "$reads_file" ]] && count="$(wc -l <"$reads_file")"
135+ printf 'read\n' >>"$reads_file"
136+ if [[ "$count" == 0 ]]; then printf '100\n'; else printf '200\n'; fi
137+ return
138+ fi
139+ reads_file="$child_reads"
140+ [[ -f "$reads_file" ]] && count="$(wc -l <"$reads_file")"
141+ printf 'read\n' >>"$reads_file"
142+ if ((count < 2)); then printf '300\n'; else printf '400\n'; fi
143+ }
144+ ps() { printf '6262\n'; }
145+ kill() { printf '%s\n' "$*" >>"$signal_log"; }
146+ sleep() { :; }
147+
148+ _stop_one reused "$definitions/reused" >/dev/null
149+ grep -Fxq -- '-TERM 4242' "$signal_log"
150+ if grep -Fq -- '-KILL' "$signal_log"; then
151+ echo "stop sent SIGKILL after capture or service PID identity changed" >&2
152+ exit 1
153+ fi
154+ [[ ! -e "$STATE_DIR/reused.state" ]]
155+)
156+
157+# A genuinely stuck capture and same-identity service group are both
158+# escalated. If they survive SIGKILL, stop fails and retains state for retry.
159+(
160+ # shellcheck disable=SC1090
161+ source "$services"
162+ STATE_DIR="$tmp/stuck-stop-runtime"
163+ STOP_TIMEOUT=0
164+ mkdir -p "$STATE_DIR"
165+ touch "$STATE_DIR/stuck.state"
166+ signal_log="$tmp/stuck-stop.signals"
167+
168+ _state_process_matches() {
169+ STATE_PID=5252
170+ STATE_START=500
171+ return 0
172+ }
173+ _direct_children() { printf '7272\n'; }
174+ _pid_alive() { return 0; }
175+ _process_start_time() {
176+ if [[ "$1" == 5252 ]]; then printf '500\n'; else printf '700\n'; fi
177+ }
178+ ps() { printf '7272\n'; }
179+ kill() { printf '%s\n' "$*" >>"$signal_log"; }
180+ sleep() { :; }
181+
182+ if _stop_one stuck "$definitions/stuck" \
183+ >"$tmp/stuck-stop.stdout" 2>"$tmp/stuck-stop.stderr"; then
184+ echo "stop succeeded after same-identity processes survived SIGKILL" >&2
185+ exit 1
186+ fi
187+ : "$STOP_TIMEOUT" "$STATE_PID" "$STATE_START"
188+ grep -Fxq -- '-TERM 5252' "$signal_log"
189+ grep -Fxq -- '-KILL -- -7272' "$signal_log"
190+ grep -Fxq -- '-KILL 5252' "$signal_log"
191+ grep -Fq 'capture survived stop' "$tmp/stuck-stop.stderr"
192+ grep -Fq 'process group survived stop' "$tmp/stuck-stop.stderr"
193+ [[ -e "$STATE_DIR/stuck.state" ]]
194+)
195+
196+"$services" status >"$tmp/services.status"
197+grep -Fq $'healthy\trunning' "$tmp/services.status"
198+grep -Fq $'broken\trestarting' "$tmp/services.status"
199+if grep -Fq $'broken\trunning' "$tmp/services.status"; then
200+ echo "crash-looping service was reported as running" >&2
201+ exit 1
202+fi
203+grep -Fq $'BadName\tignored (unsafe name)' "$tmp/services.status"
204+grep -Fq $'disabled\tignored (not executable)' "$tmp/services.status"
205+grep -Fq $'not-a-file\tignored (not a regular file)' "$tmp/services.status"
206+grep -Fq $'linked\tignored (symlink)' "$tmp/services.status"
207+
208+# The flock and runtime identity record serialize concurrent reconciliations;
209+# the already-running foreground service is not launched twice.
210+for _ in 1 2 3 4 5 6 7 8; do
211+ "$services" reconcile >"$tmp/concurrent.$_.out" 2>"$tmp/concurrent.$_.err" &
212+done
213+wait
214+[[ "$(wc -l <"$events/healthy.starts" | tr -d ' ')" == 1 ]]
215+
216+# A transiently missing current helper cannot replace configured services, so
217+# reconcile fails without stopping or duplicating their existing supervisors.
218+# Removed definitions and explicit stop-all still use stored identity to stop.
219+helper_failure_config="$tmp/helper-failure-config"
220+helper_failure_definitions="$helper_failure_config/services.d"
221+helper_failure_runtime="$tmp/helper-failure-runtime"
222+helper_failure_logs="$tmp/helper-failure-logs"
223+helper_failure_events="$tmp/helper-failure-events"
224+missing_helper="$tmp/missing-tx9-logs"
225+mkdir -p "$helper_failure_definitions" "$helper_failure_runtime" \
226+ "$helper_failure_logs" "$helper_failure_events"
227+cat >"$helper_failure_definitions/keep" <<'EOF'
228+#!/usr/bin/env bash
229+name="${0##*/}"
230+printf 'start %s\n' "$$" >>"$HELPER_FAILURE_EVENTS/$name.starts"
231+trap 'printf "term %s\n" "$$" >>"$HELPER_FAILURE_EVENTS/$name.terms"; exit 0' TERM INT
232+while :; do sleep 1; done
233+EOF
234+cp "$helper_failure_definitions/keep" "$helper_failure_definitions/remove"
235+chmod 0700 "$helper_failure_definitions/keep" "$helper_failure_definitions/remove"
236+
237+helper_failure_services() {
238+ local helper="$1"
239+ shift
240+ env HELPER_FAILURE_EVENTS="$helper_failure_events" \
241+ TX9_SERVICES_CONFIG_ROOT="$helper_failure_config" \
242+ TX9_SERVICES_STATE_DIR="$helper_failure_runtime" \
243+ TX9_SERVICES_LOG_DIR="$helper_failure_logs" \
244+ TX9_SERVICES_LOG_HELPER="$helper" \
245+ TX9_SERVICES_RESTART_DELAY=0.2 \
246+ TX9_SERVICES_STOP_TIMEOUT=2 \
247+ "$services" "$@"
248+}
249+
250+helper_failure_services "$PROJECT_ROOT/guest/tx9-logs" reconcile >/dev/null
251+wait_for_count 1 "$helper_failure_events/keep.starts"
252+wait_for_count 1 "$helper_failure_events/remove.starts"
253+keep_state="$(cat "$helper_failure_runtime/keep.state")"
254+remove_state="$(cat "$helper_failure_runtime/remove.state")"
255+keep_capture="${keep_state%%$'\t'*}"
256+remove_capture="${remove_state%%$'\t'*}"
257+keep_child="$(awk 'NR == 1 { print $2 }' "$helper_failure_events/keep.starts")"
258+remove_child="$(awk 'NR == 1 { print $2 }' "$helper_failure_events/remove.starts")"
259+pids+=("$keep_capture" "$remove_capture")
260+
261+if helper_failure_services "$missing_helper" reconcile \
262+ >"$tmp/helper-failure.stdout" 2>"$tmp/helper-failure.stderr"; then
263+ echo "reconcile succeeded without an available log helper" >&2
264+ exit 1
265+fi
266+grep -Fq 'tx9-logs is unavailable' "$tmp/helper-failure.stderr"
267+[[ "$(cat "$helper_failure_runtime/keep.state")" == "$keep_state" ]]
268+[[ "$(cat "$helper_failure_runtime/remove.state")" == "$remove_state" ]]
269+kill -0 "$keep_capture"
270+kill -0 "$remove_capture"
271+kill -0 "$keep_child"
272+kill -0 "$remove_child"
273+[[ "$(wc -l <"$helper_failure_events/keep.starts" | tr -d ' ')" == 1 ]]
274+[[ "$(wc -l <"$helper_failure_events/remove.starts" | tr -d ' ')" == 1 ]]
275+[[ ! -e "$helper_failure_events/keep.terms" ]]
276+[[ ! -e "$helper_failure_events/remove.terms" ]]
277+
278+rm -f "$helper_failure_definitions/remove"
279+if helper_failure_services "$missing_helper" reconcile >/dev/null 2>&1; then
280+ echo "reconcile masked helper failure after removing a definition" >&2
281+ exit 1
282+fi
283+wait_for_absent "$helper_failure_runtime/remove.state"
284+wait_for_count 1 "$helper_failure_events/remove.terms"
285+[[ "$(cat "$helper_failure_runtime/keep.state")" == "$keep_state" ]]
286+kill -0 "$keep_capture"
287+kill -0 "$keep_child"
288+[[ "$(wc -l <"$helper_failure_events/keep.starts" | tr -d ' ')" == 1 ]]
289+
290+helper_failure_services "$missing_helper" stop-all >/dev/null
291+wait_for_absent "$helper_failure_runtime/keep.state"
292+wait_for_count 1 "$helper_failure_events/keep.terms"
293+
294+# Unexpected SIGKILL of a capture wrapper cannot orphan its direct foreground
295+# service. Reconcile replaces the stale wrapper with one service, and stop-all
296+# leaves no managed capture or foreground service behind.
297+pdeath_config="$tmp/pdeath-config"
298+pdeath_definitions="$pdeath_config/services.d"
299+pdeath_runtime="$tmp/pdeath-runtime"
300+pdeath_logs="$tmp/pdeath-logs"
301+pdeath_events="$tmp/pdeath-events"
302+mkdir -p "$pdeath_definitions" "$pdeath_runtime" "$pdeath_logs" "$pdeath_events"
303+cat >"$pdeath_definitions/guarded" <<'EOF'
304+#!/usr/bin/env bash
305+printf 'start %s\n' "$$" >>"$PDEATH_SERVICE_EVENTS/starts"
306+exec python3 -c 'import signal; signal.pause()' >/dev/null 2>&1
307+EOF
308+chmod 0700 "$pdeath_definitions/guarded"
309+
310+pdeath_services() {
311+ env PDEATH_SERVICE_EVENTS="$pdeath_events" \
312+ TX9_SERVICES_CONFIG_ROOT="$pdeath_config" \
313+ TX9_SERVICES_STATE_DIR="$pdeath_runtime" \
314+ TX9_SERVICES_LOG_DIR="$pdeath_logs" \
315+ TX9_SERVICES_LOG_HELPER="$PROJECT_ROOT/guest/tx9-logs" \
316+ TX9_SERVICES_RESTART_DELAY=0.2 \
317+ TX9_SERVICES_STOP_TIMEOUT=2 \
318+ "$services" "$@"
319+}
320+
321+pdeath_services reconcile >/dev/null
322+wait_for_count 1 "$pdeath_events/starts"
323+old_pdeath_capture="$(cut -f1 "$pdeath_runtime/guarded.state")"
324+old_pdeath_service="$(awk 'NR == 1 { print $2 }' "$pdeath_events/starts")"
325+pids+=("$old_pdeath_capture")
326+kill -KILL "$old_pdeath_capture"
327+wait_for_process_exit "$old_pdeath_capture"
328+wait_for_process_exit "$old_pdeath_service"
329+
330+pdeath_services reconcile >/dev/null
331+wait_for_count 2 "$pdeath_events/starts"
332+new_pdeath_capture="$(cut -f1 "$pdeath_runtime/guarded.state")"
333+new_pdeath_service="$(awk 'NR == 2 { print $2 }' "$pdeath_events/starts")"
334+pids+=("$new_pdeath_capture")
335+[[ "$new_pdeath_capture" != "$old_pdeath_capture" ]]
336+[[ "$new_pdeath_service" != "$old_pdeath_service" ]]
337+kill -0 "$new_pdeath_capture"
338+kill -0 "$new_pdeath_service"
339+[[ "$(wc -l <"$pdeath_events/starts" | tr -d ' ')" == 2 ]]
340+
341+pdeath_services stop-all >/dev/null
342+wait_for_absent "$pdeath_runtime/guarded.state"
343+wait_for_process_exit "$new_pdeath_capture"
344+wait_for_process_exit "$new_pdeath_service"
345+
346+# Runtime logging configuration is part of supervisor identity. Status reports
347+# drift in each setting, and reconcile uses the stored settings to stop the old
348+# process before starting exactly one replacement with the current settings.
349+drift_config="$tmp/drift-config"
350+drift_definitions="$drift_config/services.d"
351+drift_runtime="$tmp/drift-runtime"
352+drift_logs_a="$tmp/drift-logs-a"
353+drift_logs_b="$tmp/drift-logs-b"
354+drift_events="$tmp/drift-events"
355+drift_helper_b="$tmp/tx9-logs-b"
356+mkdir -p "$drift_definitions" "$drift_runtime" "$drift_logs_a" \
357+ "$drift_logs_b" "$drift_events"
358+cp "$PROJECT_ROOT/guest/tx9-logs" "$drift_helper_b"
359+chmod 0700 "$drift_helper_b"
360+cat >"$drift_definitions/drift" <<'EOF'
361+#!/usr/bin/env bash
362+printf 'start %s\n' "$$" >>"$SERVICE_DRIFT_EVENTS/starts"
363+printf 'drift-log %s\n' "$$"
364+trap 'printf "term %s\n" "$$" >>"$SERVICE_DRIFT_EVENTS/terms"; exit 0' TERM INT
365+while :; do sleep 1; done
366+EOF
367+chmod 0700 "$drift_definitions/drift"
368+
369+drift_services() {
370+ local log_dir="$1" helper="$2" delay="$3"
371+ shift 3
372+ env SERVICE_DRIFT_EVENTS="$drift_events" \
373+ TX9_SERVICES_CONFIG_ROOT="$drift_config" \
374+ TX9_SERVICES_STATE_DIR="$drift_runtime" \
375+ TX9_SERVICES_LOG_DIR="$log_dir" \
376+ TX9_SERVICES_LOG_HELPER="$helper" \
377+ TX9_SERVICES_RESTART_DELAY="$delay" \
378+ TX9_SERVICES_STOP_TIMEOUT=2 \
379+ "$services" "$@"
380+}
381+
382+drift_services "$drift_logs_a" "$PROJECT_ROOT/guest/tx9-logs" 0.2 reconcile >/dev/null
383+wait_for_count 1 "$drift_events/starts"
384+IFS=$'\t' read -r old_capture _old_start _old_fingerprint old_helper \
385+ old_log_dir old_delay <"$drift_runtime/drift.state"
386+pids+=("$old_capture")
387+[[ "$old_helper" == "$PROJECT_ROOT/guest/tx9-logs" ]]
388+[[ "$old_log_dir" == "$drift_logs_a" ]]
389+[[ "$old_delay" == 0.2 ]]
390+
391+drift_services "$drift_logs_b" "$PROJECT_ROOT/guest/tx9-logs" 0.2 status \
392+ >"$tmp/drift-log-dir.status"
393+grep -Fq $'drift\treload pending' "$tmp/drift-log-dir.status"
394+drift_services "$drift_logs_a" "$drift_helper_b" 0.2 status \
395+ >"$tmp/drift-helper.status"
396+grep -Fq $'drift\treload pending' "$tmp/drift-helper.status"
397+drift_services "$drift_logs_a" "$PROJECT_ROOT/guest/tx9-logs" 0.7 status \
398+ >"$tmp/drift-delay.status"
399+grep -Fq $'drift\treload pending' "$tmp/drift-delay.status"
400+
401+drift_services "$drift_logs_b" "$drift_helper_b" 0.7 reconcile >/dev/null
402+wait_for_count 1 "$drift_events/terms"
403+wait_for_count 2 "$drift_events/starts"
404+wait_for_pattern drift-log "$drift_logs_b/service-drift.jsonl"
405+IFS=$'\t' read -r new_capture _new_start _new_fingerprint new_helper \
406+ new_log_dir new_delay <"$drift_runtime/drift.state"
407+pids+=("$new_capture")
408+[[ "$new_capture" != "$old_capture" ]]
409+[[ "$new_helper" == "$drift_helper_b" ]]
410+[[ "$new_log_dir" == "$drift_logs_b" ]]
411+[[ "$new_delay" == 0.7 ]]
412+old_service="$(awk 'NR == 1 { print $2 }' "$drift_events/starts")"
413+new_service="$(awk 'NR == 2 { print $2 }' "$drift_events/starts")"
414+for _ in {1..100}; do
415+ if ! kill -0 "$old_capture" 2>/dev/null ||
416+ [[ "$(ps -o stat= -p "$old_capture" 2>/dev/null)" == Z* ]]; then
417+ break
418+ fi
419+ sleep 0.02
420+done
421+if kill -0 "$old_capture" 2>/dev/null &&
422+ [[ "$(ps -o stat= -p "$old_capture" 2>/dev/null)" != Z* ]]; then
423+ echo "old service supervisor survived runtime-config reconciliation" >&2
424+ exit 1
425+fi
426+if kill -0 "$old_service" 2>/dev/null &&
427+ [[ "$(ps -o stat= -p "$old_service" 2>/dev/null)" != Z* ]]; then
428+ echo "old custom service survived runtime-config reconciliation" >&2
429+ exit 1
430+fi
431+kill -0 "$new_capture"
432+kill -0 "$new_service"
433+[[ "$(wc -l <"$drift_events/starts" | tr -d ' ')" == 2 ]]
434+[[ "$(wc -l <"$drift_events/terms" | tr -d ' ')" == 1 ]]
435+drift_services "$drift_logs_b" "$drift_helper_b" 0.7 stop-all >/dev/null
436+wait_for_count 2 "$drift_events/terms"
437+wait_for_absent "$drift_runtime/drift.state"
438+
439+# Definition identity includes the content digest, not just whole-second stat
440+# metadata. Rewrite a same-size script in place, restore its original mtime,
441+# and require immediate replacement of the old foreground child.
442+cat >"$definitions/reloadable" <<'EOF'
443+#!/usr/bin/env bash
444+printf 'old\n' >>"$SERVICE_TEST_EVENTS/reloadable.starts"
445+trap 'printf "old\n" >>"$SERVICE_TEST_EVENTS/reloadable.terms"; exit 0' TERM INT
446+while :; do sleep 1; done
447+EOF
448+chmod 0700 "$definitions/reloadable"
449+"$services" reconcile >/dev/null
450+wait_for_pattern old "$events/reloadable.starts"
451+reload_metadata="$(stat -c '%d:%i:%Y:%s:%a' "$definitions/reloadable")"
452+reload_mtime="$(stat -c '%Y' "$definitions/reloadable")"
453+cat >"$definitions/reloadable" <<'EOF'
454+#!/usr/bin/env bash
455+printf 'new\n' >>"$SERVICE_TEST_EVENTS/reloadable.starts"
456+trap 'printf "new\n" >>"$SERVICE_TEST_EVENTS/reloadable.terms"; exit 0' TERM INT
457+while :; do sleep 1; done
458+EOF
459+touch -d "@$reload_mtime" "$definitions/reloadable"
460+[[ "$(stat -c '%d:%i:%Y:%s:%a' "$definitions/reloadable")" == "$reload_metadata" ]]
461+"$services" reconcile >/dev/null
462+wait_for_pattern old "$events/reloadable.terms"
463+wait_for_pattern new "$events/reloadable.starts"
464+rm -f "$definitions/reloadable"
465+"$services" reconcile >/dev/null
466+
467+# A crashing service restarts with the configured bounded delay while the
468+# healthy service stays up and reconciliation remains responsive.
469+before="$(wc -l <"$events/broken.starts" | tr -d ' ')"
470+sleep 0.1
471+after_early="$(wc -l <"$events/broken.starts" | tr -d ' ')"
472+[[ "$after_early" -le $((before + 1)) ]]
473+wait_for_count $((before + 1)) "$events/broken.starts"
474+[[ "$(wc -l <"$events/healthy.starts" | tr -d ' ')" == 1 ]]
475+
476+# Each custom source is independently queryable, and `all` includes it. The
477+# crash-looping broken service floods the shared stream, so the all-source
478+# check filters before the tail; an unfiltered tail legitimately evicts old
479+# events. Assertions aggregate the stream: a matching record must exist even
480+# when later records differ.
481+wait_for_pattern healthy-log "$logs/service-healthy.jsonl"
482+"$PROJECT_ROOT/guest/tx9-logs" query --agent-root "$data" \
483+ --executor-root "$tmp/empty-executor" --box fixture --source service-healthy \
484+ --tail 100 --json >"$tmp/service-query.jsonl"
485+jq -se 'any(.[]; .source == "service-healthy" and .message == "healthy-log")' \
486+ "$tmp/service-query.jsonl" >/dev/null
487+"$PROJECT_ROOT/guest/tx9-logs" query --agent-root "$data" \
488+ --executor-root "$tmp/empty-executor" --box fixture --source all \
489+ --grep healthy-log --tail 100 --json >"$tmp/all-query.jsonl"
490+jq -se 'any(.[]; .source == "service-healthy" and .message == "healthy-log")' \
491+ "$tmp/all-query.jsonl" >/dev/null
492+
493+# Removing execute permission removes the desired service and synchronously
494+# stops its capture and foreground child. Restoring it starts one fresh copy.
495+chmod 0600 "$definitions/healthy"
496+"$services" reconcile
497+wait_for_absent "$runtime/healthy.state"
498+wait_for_count 1 "$events/healthy.terms"
499+chmod 0700 "$definitions/healthy"
500+"$services" reconcile
501+wait_for_count 2 "$events/healthy.starts"
502+
503+# Direct capture termination reaches the service process group, including its
504+# foreground shell trap. Reconciliation replaces the now-stale runtime record.
505+capture_pid="$(cut -f1 "$runtime/healthy.state")"
506+kill -TERM "$capture_pid"
507+wait_for_count 2 "$events/healthy.terms"
508+for _ in {1..100}; do
509+ kill -0 "$capture_pid" 2>/dev/null || break
510+ [[ "$(ps -o stat= -p "$capture_pid" 2>/dev/null)" == Z* ]] && break
511+ sleep 0.02
512+done
513+"$services" reconcile
514+wait_for_count 3 "$events/healthy.starts"
515+
516+# hb owns the durable lifecycle gate: pause waits for every service, a
517+# quiesced reload cannot restart them, resume starts them, and gateway-disable
518+# remains independent.
519+(
520+ export HB_DATA="$data" HOME="$home"
521+ # shellcheck disable=SC1090
522+ source "$PROJECT_ROOT/guest/hb"
523+ _stop_gateway() { return 0; }
524+ _stop_executor() { return 0; }
525+ _executor_up() { return 0; }
526+ _start_gateway() { return 0; }
527+ pause >/dev/null
528+)
529+wait_for_absent "$runtime/healthy.state"
530+wait_for_absent "$runtime/broken.state"
531+healthy_before_resume="$(wc -l <"$events/healthy.starts" | tr -d ' ')"
532+"$services" reconcile
533+sleep 0.1
534+[[ "$(wc -l <"$events/healthy.starts" | tr -d ' ')" == "$healthy_before_resume" ]]
535+(
536+ export HB_DATA="$data" HOME="$home"
537+ # shellcheck disable=SC1090
538+ source "$PROJECT_ROOT/guest/hb"
539+ _executor_up() { return 0; }
540+ _start_gateway() { return 0; }
541+ resume >/dev/null
542+)
543+wait_for_count $((healthy_before_resume + 1)) "$events/healthy.starts"
544+healthy_capture="$(cut -f1 "$runtime/healthy.state")"
545+(
546+ export HB_DATA="$data" HOME="$home"
547+ # shellcheck disable=SC1090
548+ source "$PROJECT_ROOT/guest/hb"
549+ _stop_gateway() { return 0; }
550+ gateway_disable >/dev/null
551+)
552+kill -0 "$healthy_capture"
553+
554+# Deletion stops a service, while the other definition remains independently
555+# supervised. Finish by proving stop-all leaves no managed capture alive.
556+rm -f "$definitions/broken"
557+"$services" reconcile
558+wait_for_absent "$runtime/broken.state"
559+"$services" stop-all
560+wait_for_absent "$runtime/healthy.state"
561+
562+# An empty status includes the exact installation path instead of presenting
563+# a bare table with no next step.
564+empty_config="$tmp/empty-config"
565+TX9_SERVICES_CONFIG_ROOT="$empty_config" \
566+ TX9_SERVICES_STATE_DIR="$tmp/empty-runtime" \
567+ TX9_SERVICES_LOG_DIR="$tmp/empty-logs" \
568+ "$services" status >"$tmp/empty.status"
569+grep -Fq "No custom services configured. Add executable files to $empty_config/services.d" \
570+ "$tmp/empty.status"
571+
572+# A service started by a separate reload/session is outside the workload's
573+# process group. TERM on the workload must still synchronously stop it via the
574+# runtime identity record and preserve signal-derived exit status.
575+term_data="$tmp/term-data"
576+term_home="$term_data/home/agent"
577+term_config="$term_home/.config/hermes-box"
578+term_definitions="$term_config/services.d"
579+term_runtime="$tmp/term-runtime"
580+term_logs="$term_data/logs"
581+term_events="$tmp/term-events"
582+mkdir -p "$term_definitions" "$term_runtime" "$term_logs" "$term_events" "$tmp/term-bin"
583+cat >"$term_definitions/separate" <<'EOF'
584+#!/usr/bin/env bash
585+printf 'started\n' >>"$SERVICE_TERM_EVENTS/starts"
586+trap 'printf "terminated\n" >>"$SERVICE_TERM_EVENTS/terms"; exit 0' TERM INT
587+while :; do sleep 1; done
588+EOF
589+chmod 0700 "$term_definitions/separate"
590+cat >"$tmp/term-bin/hb" <<'EOF'
591+#!/usr/bin/env bash
592+exit 0
593+EOF
594+chmod 0700 "$tmp/term-bin/hb"
595+setsid env HOME="$term_home" SERVICE_TERM_EVENTS="$term_events" \
596+ TX9_SERVICES_CONFIG_ROOT="$term_config" \
597+ TX9_SERVICES_STATE_DIR="$term_runtime" \
598+ TX9_SERVICES_LOG_DIR="$term_logs" \
599+ TX9_SERVICES_LOG_HELPER="$PROJECT_ROOT/guest/tx9-logs" \
600+ TX9_SERVICES_RESTART_DELAY=0.2 TX9_SERVICES_STOP_TIMEOUT=2 \
601+ "$services" reconcile >/dev/null
602+wait_for_pattern started "$term_events/starts"
603+separate_capture="$(cut -f1 "$term_runtime/separate.state")"
604+pids+=("$separate_capture")
605+setsid env HOME="$term_home" PATH="$tmp/term-bin:$PATH" \
606+ TX9_SERVICES_STATE_DIR="$term_runtime" \
607+ TX9_SERVICES_LOG_HELPER="$PROJECT_ROOT/guest/tx9-logs" \
608+ TX9_SERVICES_RESTART_DELAY=0.2 TX9_SERVICES_STOP_TIMEOUT=2 \
609+ HB_WORKLOAD_LOGS="$term_logs" HB_BOX_ENV="$tmp/missing-box.env" \
610+ "$PROJECT_ROOT/guest/hb-workload" 4788 0 \
611+ >"$tmp/term-workload.stdout" 2>"$tmp/term-workload.stderr" &
612+workload_pid=$!
613+pids+=("$workload_pid")
614+for _ in {1..100}; do
615+ kill -0 "$workload_pid" 2>/dev/null && break
616+ sleep 0.02
617+done
618+separate_pgid="$(ps -o pgid= -p "$separate_capture" | tr -d ' ')"
619+workload_pgid="$(ps -o pgid= -p "$workload_pid" | tr -d ' ')"
620+[[ -n "$separate_pgid" && -n "$workload_pgid" && "$separate_pgid" != "$workload_pgid" ]]
621+kill -TERM "$workload_pid"
622+set +e
623+wait "$workload_pid"
624+workload_status=$?
625+set -e
626+[[ "$workload_status" == 143 ]]
627+wait_for_pattern terminated "$term_events/terms"
628+wait_for_absent "$term_runtime/separate.state"
629+
630+echo "custom service regression checks passed"
tests/static.sh+10−1
@@ -12,6 +12,7 @@ regression_files=(tests/regressions-*.sh) · 11 unmodified lines
1212 files=(
1313 guest/hb
1414 guest/hb-workload
15+ guest/tx9-services
1516 guest/lib-mcp.sh
1617 guest/profile.sh
1718 guest/agent-bash-profile.sh
@@ -26,7 +27,7 @@ files=( · 8 unmodified lines
2627 bash -n "${files[@]}"
2728 python3 -c 'compile(open("guest/hermes-state", encoding="utf-8").read(), "guest/hermes-state", "exec")'
2829 python3 -c 'compile(open("guest/tx9-logs", encoding="utf-8").read(), "guest/tx9-logs", "exec")'
29−for file in guest/hb guest/hb-workload guest/hermes-state guest/tx9-logs provision/provision.sh tests/hermes-state.sh "${regression_files[@]}"; do
30+for file in guest/hb guest/hb-workload guest/tx9-services guest/hermes-state guest/tx9-logs provision/provision.sh tests/hermes-state.sh "${regression_files[@]}"; do
3031 [[ -x "$file" ]] || { echo "not executable: $file" >&2; exit 1; }
3132 done
3233
@@ -66,6 +67,13 @@ grep -q '_refresh_targets' guest/hb-workload · 33 unmodified lines
6667 grep -q '\. "$BOX_ENV"' guest/hb-workload
6768 grep -q '_stop_executor_bridge' guest/hb-workload
6869
70+# --- guest/tx9-services invariants ---------------------------------------
71+grep -q 'services.d' guest/tx9-services
72+grep -q -- '--source "service-\$name"' guest/tx9-services
73+grep -q -- '--restart-delay' guest/tx9-services
74+grep -q '_services_reconcile' guest/hb
75+grep -q '_services_stop' guest/hb
76+
6977 # --- provisioning / config invariants ------------------------------------
7078 grep -q 'tools) tools_only ;;' provision/provision.sh
7179 grep -q 'assets) assets_only ;;' provision/provision.sh
@@ -92,6 +100,7 @@ grep -q 'tx9-logs capture' docker/entrypoint.sh · 20 unmodified lines
92100 grep -q -- '--source agent' docker/entrypoint.sh
93101 grep -q 'tx9-logs capture --source executor' docker/executor-entrypoint.sh
94102 grep -q 'tx9-logs.*OPT/bin/tx9-logs' provision/provision.sh
103+grep -q 'tx9-services.*OPT/bin/tx9-services' provision/provision.sh
95104 grep -Fq '[ -t 0 ] && [ -t 1 ]' guest/agent-bash-profile.sh
96105
97106 grep -qi 'networking is always enabled' README.md