History

Productionize Linux jemalloc heap profiling (#12265)

## Description Productionizes automatic jemalloc heap profiling on Linux so high-memory events upload a heap profile to Sentry, matching what macOS already does. The key difference from macOS: **the Linux profile is uploaded unsymbolized (raw pprof)**. It carries sample addresses + memory mappings + the GNU build-id, and is symbolized **offline** against the debug-info file (DIF) the release pipeline already uploads to Sentry — the same artifact used to symbolize panics, matched by build-id. This lets the shipped binary stay fully stripped (no symbol-table bloat) while still producing usable, symbolizable profiles. ### What changed - **`app/src/profiling.rs`**: on Linux, dump the gzipped pprof **in-process** via `jemalloc_pprof` (`dump_jemalloc_pprof_bytes`) — no external `pprof` binary, HTTP server, or fixed port (essential for the headless remote-server daemon). The dump is now **raw / unsymbolized**. The external-`pprof` path is kept for macOS, where in-process symbolization stays. - **`app/Cargo.toml`**: build `jemalloc_pprof` **without** the `symbolize` feature, so `dump_pprof()` returns a raw profile instead of symbolizing in-process (in-process symbolization would have required keeping the symbol table in the shipped binary). - **`Cargo.lock`**: bump `jemalloc_pprof` / `pprof_util` to `0.8.2`. This is required: `0.8.2` writes a usable mapping range (`memory_limit = u64::MAX`), whereas `0.8.1` wrote `memory_limit = 0`, leaving pprof unable to bind sample addresses to the binary — so profiles would be unsymbolizable even with the correct DIF. (See this PR: https://github.com/polarsignals/rust-jemalloc-pprof/pull/31) - **`script/linux/bundle`**: enable `jemalloc_pprof,heap_usage_tracking` for the `dev` and `preview` channels, and keep the **normal strip behavior** (`--strip-all` for non-dev builds). The shipped binary stays small; symbols live only in the uploaded DIF. ### Symbolizing a Linux heap profile (offline, via the Sentry DIF) The release pipeline already uploads each build's debug-info file to Sentry (`script/sentry_upload_dif.sh`), keyed by GNU build-id. To analyze a `heap-profile.pb` from an "Excessive memory usage detected" event: ```bash # 0. Use the standalone pprof (the Go-bundled `go tool pprof` misreads the # build-id on large DIFs): go install github.com/google/pprof@latest # 1. Read the main binary's build-id from the raw profile's mappings: BUILD_ID=$(pprof -raw heap-profile.pb \ | sed -n '/^Mappings/,/^Locations/p' \ | grep -E 'warp-(dev|preview)' | head -1 | awk '{print $NF}') # 2. Download the matching DIF from the channel's Sentry project # (a read-only token is sufficient; dev -> warp-client-dev, # preview -> warp-client-preview): ORG=warpdotdev; PROJECT=warp-client-dev ID=$(curl -s "https://us.sentry.io/api/0/projects/$ORG/$PROJECT/files/dsyms/?query=${BUILD_ID:0:20}" \ -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" | jq -r '.[0].id') mkdir -p difs curl -sL "https://us.sentry.io/api/0/projects/$ORG/$PROJECT/files/dsyms/?id=$ID" \ -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" -o "difs/${BUILD_ID}.debug" # 3. Symbolize + analyze (pprof matches the DIF to the profile by build-id): PPROF_BINARY_PATH=./difs pprof -http=: heap-profile.pb # flame graph / top / graph ``` The DIF carries `.symtab` + DWARF, so frames resolve to function names with file:line and inlined frames. (`addr2line -e difs/$BUILD_ID.debug <addr>` works as a lower-level alternative.) ### Notes / scope - No `pprof` binary is bundled on Linux (in-process raw dump). - `stable` Linux builds are unchanged (no profiling features) — enabling for `dev` + `preview` first. ## Linked Issue N/A — infrastructure/observability follow-up. ## Testing - `cargo check` + `cargo clippy -- -D warnings` for the app lib targeting Linux with `heap_usage_tracking` (exercises the new raw-dump branch); macOS host path also checked (validates the `pprof_binary_path` cfg-gate) — passes. - `./script/format` — no changes beyond the edited files. - End-to-end validated on a build sharing this exact code path - [ ] I have manually tested my changes locally with `./script/run` ## Agent Mode - [x] Warp Agent Mode - This PR was created via Warp's AI Agent Mode CHANGELOG-NONE --- 🤖 Generated with Warp Agent Mode Conversation: https://staging.warp.dev/conversation/e6ba6343-1bb8-4cce-a155-6f0b453bad40 Plan: https://staging.warp.dev/drive/notebook/359wXMjgAzXhtOZfnSWUlA

Yunfan Yang committed 5 months ago

Commit 1ef622e1 parent 83c11f14 files · +110 −91Browse files

Showing the changes to script/linux/bundle (1 of 4 files). Show all

Unified · Split

script/linux/bundle+4−2
@@ -170,14 +170,16 @@ elif [[ $RELEASE_CHANNEL = "dev" ]]; then · 169 unmodified lines
170170 BINARY_NAME="warp-dev"
171171 APP_NAME="WarpDev"
172172 FEATURES="$FEATURES,agent_mode_debug"
173− # Enable heap profiling using jemalloc through pprof.
174− FEATURES="$FEATURES,jemalloc_pprof"
173+ # Enable heap usage tracking & profiling using jemalloc through pprof.
174+ FEATURES="$FEATURES,jemalloc_pprof,heap_usage_tracking"
175175 export HANDLE_MARKDOWN=1
176176 elif [[ $RELEASE_CHANNEL = "preview" ]]; then
177177 WARP_BIN="preview"
178178 BINARY_NAME="warp-preview"
179179 APP_NAME="WarpPreview"
180180 FEATURES="$FEATURES,preview_channel"
181+ # Enable heap usage tracking & profiling using jemalloc through pprof.
182+ FEATURES="$FEATURES,jemalloc_pprof,heap_usage_tracking"
181183 elif [[ $RELEASE_CHANNEL = "stable" ]]; then
182184 WARP_BIN="stable"
183185 BINARY_NAME="warp"

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t