Acknowledgment workflow

perf-sentinel supports two complementary acknowledgment mechanisms: TOML in-repo (CI ack, since 0.5.17) and JSONL (JSON Lines, an append-only log format where each line is a standalone JSON object) via the daemon HTTP API (daemon ack, since 0.5.20). They cover different operational scenarios and can be used side-by-side. This page explains how each works, when to pick which, and how the CLI helper introduced in 0.5.22 plugs into the daemon side.

Two things the diagram makes visible. The daemon reads both stores while batch reads only the TOML. And the HTML dashboard is always produced by batch report, the daemon serves no HTML at all: --daemon-url only makes that generated file talk to a daemon from the browser, which is what puts Ack buttons on it. A static report has no ack path, it has no one to post to.

What is a finding signature. A signature is a stable identifier for a finding, built by hashing (finding_type, service, normalised endpoint template, normalised query, URL or destination template) with SHA-256 and keeping a 32-hex prefix. The same finding produced by two daemon restarts yields the same signature, so an ack written once stays attached to its target across restarts and across analyzers. The exact serialisation rule and the 11 tests that lock it lives in Acknowledgments.

CI ack: TOML in repo

The .perf-sentinel-acknowledgments.toml file at the root of an application repository, versioned in git, modified through PR review. Use this for permanent decisions made by the team: false positives, known accepted-risk findings, intentional design choices.

Adding a TOML ack

Edit the file directly:

[[acknowledged]]
signature = "n_plus_one_sql:order-svc:_api_orders:0123456789abcdef0123456789abcdef"
acknowledged_by = "team-architecture"
acknowledged_at = "2026-05-04T13:30:00Z"
reason = "Intentional fanout for batch reporting endpoint"

Commit, open a pull request, get review, merge. The next CI run will honor the ack via analyze --acknowledgments and the CI templates bundled with the project.

Removing a TOML ack

Delete the entry, commit, PR, review, merge. Same lifecycle as adding one.

Daemon ack: JSONL via API

For temporary, runtime acks made by SREs or oncall: defer a finding while a fix ships, suppress noise during a known incident, etc. The daemon persists these in a JSONL file as append-only events, with optional expiration timestamps.

Adding a daemon ack via curl (low-level)

curl -X POST http://daemon:4318/api/findings/<sig>/ack \
  -H "Content-Type: application/json" \
  -d '{"by":"alice","reason":"deferred","expires_at":"2026-05-11T00:00:00Z"}'

When auth is enabled server-side ([daemon.ack] api_key), add -H "X-API-Key: <KEY>".

Adding a daemon ack via CLI (since 0.5.22, recommended)

perf-sentinel ack create \
  --signature "n_plus_one_sql:order-svc:_api_orders:0123456789abcdef0123456789abcdef" \
  --reason "deferred to next sprint" \
  --expires 7d

The CLI handles auth resolution, duration parsing (relative or ISO8601), daemon URL resolution and produces readable error messages. See CLI reference for the full reference, including the 1 KiB caps applied to stdin signatures and the interactive API-key prompt.

Revoking a daemon ack

perf-sentinel ack revoke \
  --signature "n_plus_one_sql:order-svc:_api_orders:0123456789abcdef0123456789abcdef"

Or via curl:

curl -X DELETE http://daemon:4318/api/findings/<sig>/ack

Listing active acks

perf-sentinel ack list                  # daemon acks, table format
perf-sentinel ack list --output json    # daemon acks, JSON

perf-sentinel ack list only enumerates daemon-side acks. TOML CI acks live in the file itself, view them with:

cat .perf-sentinel-acknowledgments.toml

Over HTTP, GET /api/acks?include_toml=true (since 0.24.0) lists both, each row naming its source, see Query API.

Interop: TOML wins on conflict

Both sources are unioned at finding-filtering time. If the same signature is acked in both TOML and daemon JSONL, the TOML version wins. The rationale: the TOML baseline is shipped via PR review and represents an immutable team-level decision; the daemon JSONL is a mutable, runtime-only override.

One boundary to have clear before reaching for either surface: a daemon ack never reaches CI. The JSONL lives on that daemon's disk, so acking from the dashboard, the TUI or ack create silences that one daemon and nothing else. A CI gate, the Diff baseline and the unmatched_acknowledgment warning all read the TOML alone. To unblock a pipeline, copy the signature into the TOML and open a PR. The signatures are identical on both sides, so the dashboard is a fine place to copy one from.

Writing a TOML entry without transcribing it

Nothing needs copying by hand. The JSON report already carries every field an entry takes, so one command appends a complete block. Adjust the select(...) to pick your finding:

perf-sentinel analyze --input traces.json --format json \
  | jq -r --arg by "$(git config user.email)" \
          --arg why "Intentional, see ADR-0042" \
          --arg on "$(date +%F)" '
      .findings[]
      | select(.type == "n_plus_one_sql" and .service == "order-svc")
      | "
[[acknowledged]]
signature = \"\(.signature)\"
acknowledged_by = \"\($by)\"
acknowledged_at = \"\($on)\"
reason = \"\($why)\"
service = \"\(.service)\"
source_endpoint = \"\(.source_endpoint)\""' \
  >> .perf-sentinel-acknowledgments.toml

Review the appended block, add an expires_at if the decision is temporary, then open the PR. Same recipe when promoting an ack made on a daemon: read the signature with perf-sentinel ack --daemon <url> list --output json and select on it. The daemon entry can stay, TOML wins on conflict.

A POST /api/findings/{sig}/ack for a signature already covered by TOML returns HTTP 409 to avoid silent shadowing. The ack create CLI maps this to exit 2 with a hint pointing at ack revoke.

Adding a daemon ack from the HTML report (since 0.5.23, browser)

The HTML report can run in live mode and drive the same daemon endpoints from the browser. Generate the report with --daemon-url, open it, click the per-finding Ack button. See HTML report for the setup, the CORS prerequisites, and the X-API-Key handling.

perf-sentinel report --input traces.json --output report.html \
  --daemon-url http://localhost:4318
open report.html

Adding a daemon ack from the TUI (since 0.5.24, terminal)

perf-sentinel query inspect opens an interactive TUI that exposes the daemon findings list, span trees, and cross-trace correlations. With 0.5.24, pressing a on the selected finding opens an acknowledgment modal (reason / expires / by) that posts to the same daemon endpoint, and u opens a revoke confirmation. The Findings panel renders an [acked by <user>] italic gray indicator next to already-acknowledged findings. See Inspect (TUI) for the keybinding map and the auth flow.

perf-sentinel query --daemon http://localhost:4318 inspect
# Press 'a' on a finding → modal → fill reason → Tab to Submit → Enter

a and u are no-op in batch mode (inspect --input) since acknowledgment requires a running daemon to persist.

Adding a daemon ack from the Hub (Hub 0.3.0, browser)

A PerfSentinelHub that holds an ack credential for a daemon relays an ack or a revoke to it from one page, in the name of the signed-in user. The Grafana findings dashboard links to that page from the Ack column of its three findings tables (<Hub URL>/?ack=<signature>), which is the way to acknowledge a finding the daemon's ring no longer holds: the write route only needs the signature. One submit writes to every checked daemon that carries the finding, because each daemon keeps its own store. A CI baseline ack shows there and can only be changed in the file. See the Hub's docs/LAUNCHER.md.

Choosing between TOML and daemon

Pull requests only in an environment

A team that wants every ack of one environment, a CI or a staging daemon for instance, to go through a reviewed pull request keeps the TOML baseline and closes the runtime writes on that daemon.

Do not set [daemon.ack] enabled = false for that. It does close the routes, but the daemon then loads no ack at all, the TOML baseline included, so every finding the baseline acknowledges comes back in GET /api/findings.

Keep enabled = true and set [daemon.ack] api_key to a secret nobody is given. POST and DELETE then answer 401 from the CLI, the HTML report and the TUI alike, the baseline keeps applying, and GET /api/acks stays readable with [daemon] read_api_key. On a Hub that relays acks (Hub 0.3.0 and later), leave that source without an ack credential: the Hub's ack page then offers nothing for it.

Observability

The daemon exposes Prometheus counters on /metrics for every ack operation it processes (perf_sentinel_ack_operations_total{action} and perf_sentinel_ack_operations_failed_total{action,reason}). See Metrics for the full schema and example PromQL queries.

Signature stability and service restarts

Acknowledgments match findings by a canonical signature:

<finding_type>:<service>:<sanitized_endpoint>:<sha256-prefix-of-template>

The signature deliberately excludes trace_id and span_id, so a single ack survives service restarts and routine traffic with varying request identifiers. The contract is locked by unit tests in crates/sentinel-core/src/acknowledgments.rs.

Critical dependency on http.route

The endpoint component is derived from the OpenTelemetry http.route attribute on the entry HTTP span or its same-service ancestors. For an explicitly named service, perf-sentinel selects the outermost route in the contiguous chain; it never adopts the caller service's route. A route template without a leading slash is canonicalized with one (api/orders/{id} becomes /api/orders/{id}), so equivalent instrumentation shapes produce one signature. Some frameworks instead put a symbolic route name in http.route and the request path in url.path. When the route contains no / and url.path is usable, perf-sentinel uses url.path; a route containing / remains authoritative, including slashless Django routes and templates. A symbolic route without url.path keeps the conservative existing behavior and is canonicalized with a leading slash.

When traced services emit http.route:

• Same finding on the same logical endpoint produces the same signature.
• Acknowledgments survive service restarts.
• Acknowledgments survive normal traffic with rotating request IDs.

When http.route is missing on an explicitly SERVER span, perf-sentinel falls back to http.url, url.full, url.path, then http.target (depending on the ingest format). URL-only CLIENT spans remain outbound operations and are not mistaken for entry points. Each unique URL yields a different signature, ack churn becomes proportional to URL cardinality, and deferred findings reappear at every new request id. The fallback exists so the operator still sees a usable endpoint string, not as a recommended posture.

Standard OpenTelemetry agents emit http.route automatically:

• Spring Boot 3+ with the OpenTelemetry Java agent.
• ASP.NET Core with the OpenTelemetry .NET SDK.
• Express.js, Fastify, Koa with @opentelemetry/instrumentation-*.
• Most modern HTTP framework auto-instrumentations.

To confirm an instrumented service emits route templates, inspect a recent finding's source_endpoint against a running daemon:

curl -s http://localhost:4318/api/findings | jq -r '.[].source_endpoint' | sort -u

Templates with placeholders (/api/orders/{id}) indicate healthy instrumentation. Instantiated URLs with hardcoded ids (/api/orders/42) indicate http.route is missing and acks will churn.

Upgrade note (0.11.2): endpoint attribution changed in three ways, on OTLP, Jaeger and Zipkin alike. It selects the outermost route in a contiguous same-service chain rather than the nearest one, and the daemon keeps valid sampled parent context across OTLP export requests, so findings that previously used an inner framework route or unknown can receive a different signature. A framework route holding no / now yields to a usable url.path, but only on the inbound side, a SERVER span for its own endpoint or a non-CLIENT ancestor in the chain, so an instrumentation that never sets a span kind keeps the route name. And a route that omits its leading slash gains one, which moves findings that were already attributed to the right route. Re-capture affected acknowledgments and persisted report baselines with 0.11.2.

Separately, a SERVER span no longer produces an outbound HTTP call, so on a fleet instrumented in legacy semantic conventions some HTTP findings disappear rather than move. A fresh batch analysis flags every ack that suppressed nothing, moved and disappeared findings alike, as unmatched_acknowledgment. Read that message before acting on it: when the endpoint still emitted I/O, and a surviving SQL child on the same handler is enough for that, the message reads the problem looks fixed and the entry can be removed, which is the wrong conclusion for an ack whose finding only vanished with the upgrade. The daemon never emits this warning, it is a batch signal only.

Service renames invalidate acks

The <service> component of the signature is the OpenTelemetry service.name resource attribute. Renaming a service (order-svc → orders-svc) produces a new signature for every existing finding, so any TOML or daemon ack matched against the old name stops applying. The findings reappear in CLI output and quality gates until you either:

• Update the signature field of each affected ack entry to use the new service name (TOML), or
• Re-create the daemon-side acks via perf-sentinel ack create against the new signature.

Same applies to non-cosmetic changes to http.route (e.g. /api/orders/{id} → /api/v2/orders/{id}) and to changes to the normalised template that the detector builds (a refactor that adds a WHERE clause changes the normalized template, hence the sha256 prefix). Plan these renames alongside an ack-file refresh PR.

Carbon scoring scope

The green_impact field on each finding is computed per detection inside a single trace. The values reported by perf-sentinel analyze or in the JSON report describe one occurrence and do not aggregate across traces.

The daemon exposes Prometheus counters (perf_sentinel_findings_total, perf_sentinel_avoidable_io_ops and, since 0.18.0, its per-service split perf_sentinel_service_avoidable_io_ops_total) that accumulate monotonically over the daemon's lifetime. Each batch contributes its own per-batch dedup, keyed on (trace_id, template, source_endpoint), which prevents counting the same pattern twice within one batch. Distinct traces, including those produced after a service restart, contribute separately because they represent distinct request executions. The counters reset only when the daemon process restarts, matching the standard Prometheus counter semantics. Use rate(...) over short windows for trend dashboards rather than reading the raw absolute value.