CLI, configuration and release profile

CLI design

The CLI (sentinel-cli) is intentionally thin. It parses arguments with clap and delegates to sentinel-core functions. Ten subcommands are available: analyze, explain, watch, demo, bench, pg-stat, inspect, query, diff and report.

Analyze: colored report by default, JSON with --ci

perf-sentinel analyze displays a colored terminal report when run interactively (without --ci). This is the output humans see when running the tool locally. With --ci, the output switches to structured JSON for machine consumption and the process exits with code 1 if the quality gate fails.

This follows the convention of tools like cargo test (colored output by default, --format json for CI).

The --format flag provides explicit control over the output format: text (colored terminal, default), json (structured report) or sarif (SARIF v2.1.0 for code scanning). When --ci is used without --format, the output defaults to JSON for backward compatibility.

Explain: per-trace tree view

perf-sentinel explain --input FILE --trace-id ID builds a tree from parent_span_id relationships and annotates findings inline. It runs per-trace detectors only (N+1, redundant, slow, fanout), cross-trace percentile findings are not included.

Output formats: --format text (colored tree with Unicode box-drawing characters, default) or --format json (nested JSON structure). Both include a MAX_TREE_DEPTH guard of 256 levels to prevent stack overflow on deeply nested traces.

Bench: pre-cloned batches

let batches: Vec<Vec<SpanEvent>> = (0..iterations)
    .map(|_| events.clone())
    .collect();

Input batches are cloned before timing begins. This ensures the benchmark measures only pipeline::analyze performance, not Vec<SpanEvent>::clone overhead. Since analyze consumes its input (Vec<SpanEvent> is moved), each iteration needs its own copy.

Percentile computation

per_event_ns.sort_by(|a, b| a.partial_cmp(b).unwrap_or(Ordering::Equal));
let p50_idx = ((per_event_ns.len() as f64 * 0.50).ceil() as usize).saturating_sub(1);
let p99_idx = ((per_event_ns.len() as f64 * 0.99).ceil() as usize)
    .saturating_sub(1)
    .min(per_event_ns.len() - 1);

The ceiling-based index computation follows the nearest-rank method for percentiles. The .saturating_sub(1) converts from 1-based rank to 0-based index. The .min(len - 1) prevents out-of-bounds access when ceil rounds up to len.

Throughput from nanoseconds

let elapsed_nanos: u64 = durations_ns.iter().sum();
let total_seconds = elapsed_nanos as f64 / 1_000_000_000.0;
let throughput = if total_seconds > 0.0 { total_events / total_seconds } else { 0.0 };

Throughput is computed from nanosecond precision (not millisecond) to avoid division-by-zero when iterations complete in less than 1ms. The total_elapsed_ms field in the output is derived from nanoseconds for display purposes.

RSS measurement

Platform-specific memory measurement:

The macOS implementation uses unsafe for the libc::getrusage FFI call. This is justified, there is no safe Rust API for this syscall and the function is well-documented in POSIX. The return value is checked (if ret == 0) before using the result.

Colored output with TTY detection

let is_tty = force_color || std::io::stdout().is_terminal();
let (bold, cyan, red, yellow, green, dim, reset) = if is_tty {
    ("\x1b[1m", "\x1b[36m", "\x1b[31m", "\x1b[33m", "\x1b[32m", "\x1b[2m", "\x1b[0m")
} else {
    ("", "", "", "", "", "", "")
};

ANSI escape codes are suppressed when stdout is not a terminal (e.g., piped to a file or jq). The force_color parameter allows tests to exercise the color path without a real TTY. This follows the convention of tools like ls --color=auto and rustc's output.

Sink override for --output. The stdout().is_terminal() probe above is blind to the actual writer: a CLI invoked from an interactive terminal where --output path.txt redirects the sink to a File would still pick the colored palette and leak escape bytes into the file. emit_diff guards against this by forcing no_colors() whenever output.is_some(), regardless of the stdout TTY state. The palette is then threaded into write_diff_text as an explicit argument so the writer choice and the color decision stay in sync.

PgStat: pg_stat_statements hotspot analysis

perf-sentinel pg-stat --input FILE parses PostgreSQL pg_stat_statements exports (CSV or JSON, auto-detected) and produces hotspot rankings by total execution time, call count and mean execution time. The --traces flag enables cross-referencing with trace-based findings: the tool runs pipeline::analyze() on the trace file and marks pg_stat_statements entries whose normalized template also appears in trace findings.

This subcommand is intentionally separate from analyze because pg_stat_statements data has no trace_id, it cannot participate in the trace correlation pipeline. It is a complementary analysis mode.

Inspect: interactive TUI

perf-sentinel inspect --input FILE launches a terminal UI built with ratatui and crossterm. These dependencies live in sentinel-cli/Cargo.toml only (not sentinel-core) because TUI is a presentation concern.

Layout: 3-panel split, traces list (top-left, 30%), findings for selected trace (top-right, 70%), finding detail with span tree (bottom, 50%). The detail panel reuses explain::build_tree() and explain::format_tree_text() for the span tree display.

State management: the App struct holds pre-computed findings_by_trace (indexed at construction time) to avoid recomputing on every frame. Navigation state (selected_trace, selected_finding, active_panel, scroll_offset) is updated by key events.

Data loading: events are ingested once, then cloned. One copy for correlate() (needed for tree building) and one for pipeline::analyze() (consumed by the pipeline). This avoids re-reading the file.

report subcommand

perf-sentinel report --input FILE --output report.html produces a single-file HTML dashboard aimed at developers exploring a CI artifact in a browser. The pipeline is identical to analyze end-to-end, only the final sink differs. Implemented in crates/sentinel-core/src/report/html.rs with the full UI template embedded via include_str! from crates/sentinel-core/src/report/html_template.html.

Architecture: single-file, vanilla JS, no build step, no external dependencies. The output is one HTML file with all CSS and JS inlined. No <link rel="stylesheet">, no <script src="...">, no web fonts, no images. The file opens offline from a file:// URL with zero network requests, which makes it:

• Trivially auditable: one file, no minified bundles, no transpilation.
• Durable: no build toolchain to break on a CI runner upgrade, no NPM version drift on a recipe that's supposed to be reproducible for years.
• Safe to ship as a CI artifact: no lockfile to invalidate, no supply-chain vectors through a bundled minifier.
• Fast to review in PRs: the template is a single .html file that diffs cleanly.

The front-end uses DOM APIs directly (document.createElement, Element.textContent, Element.setAttribute). No framework. No Web Components (the prior plan considered them, but plain modules fit the 8.1 scope better in practice and keep the file ~15 KB smaller).

Security model: textContent only, grep-level CI check. All user-controlled data (SQL templates, URLs, service names, trace IDs, code locations, SuggestedFix text) is injected inside a <script id="report-data" type="application/json"> block and read once at boot via textContent + JSON.parse. The JS then renders via textContent and createElement exclusively. Forbidden: innerHTML, insertAdjacentHTML, document.write, eval, new Function. A unit test (no_forbidden_apis_in_template in report/html.rs) greps the template on every build and fails CI if any of those strings appear. Second-layer defense: the Rust injector escapes the substring </ to <\/ in the serialized JSON so a hostile user-controlled value cannot close the script block early. \/ is a permitted JSON escape, so JSON.parse recovers the original value unchanged.

Only reference_url from SuggestedFix becomes a hyperlink, and only when the value starts with https:// (validated client-side in safeHttpsHref). Non-HTTPS URLs render as plain text without a link.

Scope limit: post-mortem only. The dashboard is a static rendering of a completed trace set. No polling, no WebSocket, no Server-Sent Events, no refresh loop. The equivalent "live" view from a running daemon stays with perf-sentinel query inspect (TUI fed by the daemon's /api/* endpoints). Making the HTML dashboard live-capable would require a real-time backend binding, an update diffing strategy and state persistence across reloads - a different architecture that is out of scope here and would be re-evaluated only on user feedback.

Composition pattern for Tempo. Tempo-backed exploration composes via the shell rather than via an integrated --tempo flag on report: perf-sentinel tempo --endpoint ... --search ... --output traces.json && perf-sentinel report --input traces.json --output report.html. This avoids duplicating ~8 Tempo flags (endpoint, search tags, time window, auth, timeout, max-results, etc.) on report and keeps the two subcommands each responsible for one concern (ingestion vs. rendering). The same pattern applies to any other ingestion source: compose, don't plumb.

Trace embedding and size cap. Only traces referenced by a finding are embedded (the Explain tab is entry-point-driven from Findings, so free-navigation traces would bloat the file without earning their bytes). When --max-traces-embedded is unset, the sink targets a ~5 MB HTML output size, greedily dropping the lowest-IIS traces first (reuses the top_offenders ordering). A trimmed_traces: { kept, total } field in the embedded payload drives a banner in the Findings tab when trimming kicks in. Setting --max-traces-embedded explicitly honors the cap exactly, overriding the 5 MB heuristic.

Embedded report slimming. The report object serialized into the payload is a slimmed copy, not the full Report. Three sections the dashboard does not fully render are bounded so a high-cardinality run does not inflate the self-contained file, while analyze --format json keeps every one of them in full. findings are trimmed critical-first when they exceed 70% of the size budget (surfaced by trimmed_findings: { kept, total }). per_endpoint_io_ops is dropped entirely (no dashboard view reads it; the Diff tab's endpoint deltas come from the --before baseline, not this field). green_summary.top_offenders is capped to 25 rows (the dashboard renders only the top entry). The trace ranking still reads the un-slimmed top_offenders so the IIS ordering does not degrade past the embed cap. The trace candidate set, however, deliberately comes from the embedded (possibly trimmed) findings: every embedded trace keeps its finding visible in the dashboard, at the price of a conservative trimmed_traces.total when the findings trim fires.

Exit-code semantics differ from analyze --ci. report exits 0 even when the quality gate fails. The gate status is surfaced as a red/green badge in the HTML top bar. Users who need a CI exit signal keep using analyze --ci. Two subcommands, two concerns.

Optional cross-references: pg_stat, diff, correlations. Three optional tabs are added by dedicated flags:

• --pg-stat <FILE> ingests a pg_stat_statements CSV or JSON export via the same parse_pg_stat + rank_pg_stat path that the pg-stat subcommand uses. A pg_stat tab then shows the by-total-time ranking (Template, Calls, Total ms, Mean ms). The other two rankings (by calls, by mean) stay accessible via the text pg-stat subcommand and are not duplicated in the HTML.
• --pg-stat-prometheus <URL> scrapes a postgres_exporter endpoint one-shot via fetch_from_prometheus, same effect as --pg-stat without the intermediate file. Mutually exclusive with --pg-stat at the clap level (conflicts_with). This is a flag on report rather than a separate subcommand because a one-shot HTTP GET is not a streaming source that deserves its own command surface. Consistent with the rest of the CLI: if it doesn't stream, it composes.
• --before <FILE> deserializes a baseline report JSON (the output of analyze --format json), feeds it into diff::diff_runs against the current run, and embeds the DiffReport. A Diff tab then renders four sections: new findings (clickable, open Explain), resolved findings (not clickable, their traces are in the baseline which is not embedded), severity changes and endpoint metric deltas (both non-clickable tabular data).

Correlations tab. Only daemon-produced reports carry correlations. The batch pipeline does not emit them, so the tab stays hidden on batch output. The JS guards on report.correlations?.length > 0, so the tab lights up automatically when a future daemon-produced JSON is fed into perf-sentinel report --input <daemon.json>. No new field was added to the Report struct.

Cross-navigation. Two cross-navs plug the tabs together:

• Explain to pg_stat: when the active finding's trace contains a SQL span whose normalized template matches a row in pg_stat, that span gets a ps-span-pgstat-link class and a click handler. Clicking switches to the pg_stat tab with the matching row highlighted and a "Filtered from Explain" banner shown above the table. The banner has a "clear" link that hides it and removes the highlight. The span is not clickable when pg_stat is absent from the payload.
• Diff to Explain: rows in the new_findings section are clickable and delegate to the existing openExplain function. Rows in resolved_findings, severity_changes and endpoint_metric_deltas are not clickable. For a new finding whose trace_id has been trimmed by the size cap, the Explain panel shows "Trace not embedded (cap reached). Rerun with --max-traces-embedded <higher> to include it." instead of an empty tree.

Search via /. Each of Findings, pg_stat, Diff, Correlations carries a hidden <input type="search"> at the top of its panel. The global keyboard handler catches / when no input is focused and the active tab is searchable, reveals the input and focuses it. esc with the input focused clears the filter and hides the input. Filter logic walks the active panel's rows and toggles display: none based on case-insensitive substring match over textContent. State is cleared on tab switch (no cross-tab carryover). Explain and GreenOps are no-search by design (no meaningful row list). The 500-row cap on Findings still applies.

Baseline JSON round-trip. --before requires the Report struct to derive Deserialize, which the 8.1 tree did not. The cascade adds Deserialize to Report, Analysis, GreenSummary, QualityGate, QualityRule, TopOffender, CarbonReport, CarbonEstimate, RegionBreakdown and IntensitySource. Optional fields with skip_serializing_if gain a matching #[serde(default)] so round-trip parses cleanly even when the source JSON omits them. Two &'static str fields on CarbonEstimate (model, methodology) and one on RegionBreakdown (status) became String for serde round-trip. The cost is a handful of .to_string() calls on construction-site constants, invisible next to the surrounding numeric work.

Polish pass: client-side-only ergonomics. A later iteration added CSV export, deep-link hash, session-scoped persistence and a ? cheatsheet modal, all strictly client-side additions to html_template.html. No Rust sink changes, no new endpoints, no new dependencies.

• CSV export: every listable tab (Findings, pg_stat, Diff, Correlations) carries an Export CSV button above the list/table. The click handler runs the same filter predicate that renders the DOM, assembles RFC 4180-escaped rows with pure string concatenation (no innerHTML risk), and triggers a download via Blob + URL.createObjectURL + a temporary <a download>. The object URL is revoked on a 0ms setTimeout to avoid a memory leak while letting the browser complete the download. Explain (not a list) and GreenOps (single summary, regions table short enough to read in-place) do not get export buttons, on purpose.
• Deep-link hash: the URL fragment encodes tab[&search=...][&ranking=...][&severity=...][&service=...] on every tab switch, chip click and search input change. Writes go through history.replaceState so back/forward history is not polluted. Old-browser fallback assigns location.hash directly (one history push, acceptable). Reads on DOMContentLoaded validate the tab is registered; an unknown target or malformed hash silently falls through to defaults.
• sessionStorage persistence: two keys, perf-sentinel:theme (dark/light, read before first paint to avoid theme-flash) and perf-sentinel:pgstat-ranking (last-active ranking slug). Every access is wrapped in try/catch because Safari private mode and some enterprise policies throw on sessionStorage.setItem. Deliberately not localStorage: file:// shares the null origin across all local HTML files, so localStorage would collide across unrelated reports; sessionStorage is tab-scoped and collision-free. Hash wins over sessionStorage when both carry a value.
• Cheatsheet modal: a ?-triggered native <dialog> element (opened via showModal(), which implicitly applies the WAI-ARIA dialog role and traps focus for us) lists every shortcut. The ? key is ignored when a text input is focused so typing ? in the filter still works. Vim-style g-prefixed shortcuts (g f / g e / g p / g d / g c / g r) switch tabs with a 1000ms timeout on the pending g, hidden tabs are a silent no-op. Esc gains two extra priority tiers on top of the existing ladder: close the cheatsheet (highest) and clear active filter chips (lowest). Findings pagination replaces the hard 500-row cap with a Show N more findings button that reveals another 500 rows at a time.

STATIC_CSP compile-time invariant

The static-mode Content-Security-Policy is the same string the template shipped before the live mode was added. It forbids every network egress and inline-execution vector except the inline <script> and <style> blocks the report itself depends on.

The placeholder substitution pipeline in inject rewrites three tokens ({{REPORT_JSON}}, {{PAGE_TITLE}}, {{CONTENT_SECURITY_POLICY}}) in a fixed order. Any byte sequence {{ that lands in STATIC_CSP would shadow that pipeline and corrupt the substitution silently.

A const _: () = { ... while ... assert!(...) } block at compile time checks that STATIC_CSP.as_bytes() never contains {{. The runtime debug_assert! in inject catches the daemon-URL half (validated by validate_url), the const block catches the static half so a future edit that introduces a templating bracket breaks the build instead of silently corrupting the output. const _: () = ... is the canonical pattern for an anonymous compile-time check that does not warn under dead_code.

Feature flags

The workspace uses Cargo feature flags to keep daemon-only dependencies optional:

Both daemon and tui are in the default feature set for the CLI. Users of sentinel-core as a library dependency can depend on it without daemon to avoid pulling in the hyper stack:

perf-sentinel-core = { version = "0.3", default-features = false }

This compiles the full batch pipeline (normalize, correlate, detect, score, report) without any HTTP client code. Config types (ScaphandreConfig, CloudEnergyConfig) are always available so the TOML parser works regardless of features; only the runtime scrapers and state types are gated.

Source code location on findings

CodeLocation struct

When OTel spans carry source code attributes (code.function, code.filepath, code.lineno, code.namespace), these are extracted during OTLP conversion and stored on SpanEvent as four optional fields. The detection pipeline propagates them to Finding.code_location: Option<CodeLocation>:

pub struct CodeLocation {
    pub function: Option<String>,
    pub filepath: Option<String>,
    pub lineno: Option<u32>,
    pub namespace: Option<String>,
}

All four fields are optional and independently present. Most auto-instrumented OTel agents emit code.function and code.namespace but not code.filepath or code.lineno. The system degrades gracefully: findings without source attributes appear without a source line, with no noise in the output.

CLI display

When code_location is present, the CLI renders a "Source:" line below the finding's endpoint:

    Source:   com.example.OrderService.processItems (OrderService.java:42)

The format is namespace.function (filepath:lineno), with each part omitted if absent. The rendering logic builds the string incrementally: namespace and function are joined with a dot, filepath and lineno are appended in parentheses only when the name portion is also present.

SARIF physicalLocation enhancement

When a finding carries a CodeLocation with at least a filepath, the SARIF output includes a locations array with a physicalLocation entry:

{
  "physicalLocation": {
    "artifactLocation": { "uri": "OrderService.java" },
    "region": { "startLine": 42 }
  }
}

The region.startLine field is included only when lineno is available. This enables inline annotations in GitHub Code Scanning and GitLab SAST when the SARIF report is uploaded as a code scanning result.

code.filepath sanitization

The code.filepath OTel attribute is attacker-controlled (a hostile span can set it to any string). Before emitting it as the SARIF artifactLocation.uri, sanitize_sarif_filepath rejects values that could phish a viewer or bypass code scanning resolvers. The sanitizer drops the URI entirely (returns None) for any of:

• Absolute paths (POSIX /..., Windows \...).
• Any colon. Legitimate source paths in instrumented apps do not contain colons; rejecting unconditionally avoids subtle bypasses around javascript:, data:, file:, etc.
• Path traversal segments. Both literal .. and percent-encoded variants (%2e%2e, %2E%2E, mixed case, .%2e, %2e.) are caught.
• Double-encoded percent sequences (%25...) that decode to a percent on first pass and to a real character on a second pass.
• Overlong UTF-8 prefixes (%c0, %c1) that decode to non-canonical encodings of ASCII characters in lax decoders.
• Control characters (newlines, NUL, etc.) that could break a SARIF consumer's tokenizer or inject into logs.
• Unicode BiDi overrides and invisible format characters (U+061C, U+180E, U+202A..U+202E, U+2066..U+2069, U+200B..U+200F, U+FEFF) that can confuse rendered filenames (Trojan Source class of attack, CVE-2021-42574).

Findings with a rejected filepath still appear in the SARIF report; only the physicalLocations array is omitted (the logicalLocations and other fields remain).

query subcommand

perf-sentinel query queries a running daemon's HTTP API. It requires the daemon feature flag.

Sub-actions

All sub-actions except inspect accept --format text|json. The default is text (colored terminal output), matching the analyze command's default. --format json outputs raw JSON for scripting and automation.

Colored output

findings reuses the existing print_findings() function from the analyze command, so the colored output is identical: severity-colored labels, source code location, template, suggestion, green impact.

explain deserializes the daemon's JSON response into an ExplainTree and calls format_tree_text() for the colored span tree with inline findings, identical to perf-sentinel explain.

inspect fetches all findings from /api/findings?limit=10000, then for each distinct trace_id it fetches the explain tree from /api/explain/{trace_id} and deserializes it into an ExplainTree. The pre-rendered colored trees are passed to the TUI via App::with_pre_rendered_trees, so the detail panel shows real span trees (not empty stubs) for every trace still in the daemon's TraceWindow. Traces that aged out return the detail panel without a tree (silent skip, no confusing empty panel).

correlations renders a custom colored table with confidence percentage color-coded (red >= 80%, yellow >= 50%).

status renders a key-value display with version, formatted uptime (Xh Ym Zs), active traces and stored findings count.

Implementation

The cmd_query function builds a closure around http_client::fetch_get that handles connection failures with a helpful error message ("Is perf-sentinel watch running?"). Each sub-action constructs the appropriate URL path, fetches the response and renders it according to the --format flag.

The default daemon URL is http://localhost:4318, matching the daemon's default HTTP listen port. Users can override it with --daemon http://host:port.

Configuration parsing

Sectioned format (only format accepted from 0.6.0)

The config requires a sectioned form: every tunable lives under [thresholds], [detection], [green] or [daemon].

[detection]
n_plus_one_min_occurrences = 5

serde(default) produces None for absent fields. The From<RawConfig> for Config conversion is a flat .unwrap_or(default) per field:

n_plus_one_threshold: raw.detection.n_plus_one_min_occurrences
    .unwrap_or(defaults.n_plus_one_threshold),

0.6.0 breaking change: 8 legacy top-level keys removed

load_from_str runs reject_legacy_top_level_keys before the typed RawConfig parse. Eight 0.5.x top-level keys (n_plus_one_threshold, window_duration_ms, listen_addr, listen_port, max_active_traces, trace_ttl_ms, max_events_per_trace, max_payload_size) now produce a ConfigError::Validation whose message names both the removed key and its sectioned replacement, so cargo run --bin perf-sentinel watch on a 0.5.x config fails fast and tells the operator exactly what to edit. The full migration table is in Configuration.

tracing::warn!(
    "Daemon configured to listen on non-loopback address: {}. \
     Endpoints have no authentication: use a reverse proxy or \
     network policy for security.",
    self.listen_addr
);

[profile.release]
codegen-units = 1
lto = "thin"
strip = true
panic = "abort"
opt-level = 3

[target.'cfg(target_env = "musl")'.dependencies]
mimalloc = "0.1.49"

#[cfg(target_env = "musl")]
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;