Architecture

Perf Sentinel is a polyglot performance anti-pattern detector built as a Rust workspace with two crates:

• sentinel-core : library containing all pipeline logic
• sentinel-cli : binary providing the CLI entry point (perf-sentinel)

Glossary

A few terms recur across the docs and code, with distinct meanings:

• Event: a single normalized SpanEvent. The pre-detection unit (input).
• Finding: a detected anti-pattern (output of detect). The product-level concept exposed in CLI output, JSON, SARIF, HTML, the daemon API and acks.
• Pattern: the normalized SQL, HTTP or messaging template that groups events for a finding (e.g. SELECT * FROM users WHERE id = ?).
• Detection: the act, or the family of detectors (n_plus_one, chatty, ...). Not a synonym of finding.

Operating modes:

• Batch mode (perf-sentinel analyze): processes a complete trace set and produces a single Report. Used in CI integration tests, ad-hoc analysis, and perf-sentinel report (HTML).
• CI mode (perf-sentinel analyze --ci): same as batch, but the quality gate failure exits with code 1.
• Daemon mode / streaming mode / watch mode (perf-sentinel watch): long-running process ingesting OTLP/JSON events in real time, with TTL-based eviction and per-trace detection. Findings are stamped daemon_staging or daemon_production based on [daemon] environment (boosts severity in downstream consumers like perf-lint).

The Confidence axis (local_batch < ci_batch < daemon_staging < daemon_production) is stamped automatically by the runtime and exposed in JSON, SARIF, HTML and the daemon query API. A batch run is stamped ci_batch when it detects a CI environment (CI set to anything but false or 0, or JENKINS_URL present) and local_batch otherwise, whether or not --ci is passed.

Pipeline overview

Operating modes

Batch mode (perf-sentinel analyze)

Processes a complete set of events and produces a single report with quality gate evaluation.

Vec<SpanEvent>
  -> normalize::normalize_all()        -> Vec<NormalizedEvent>
  -> correlate::correlate()            -> Vec<Trace>
  -> detect::detect()                  -> Vec<Finding>
  -> score::score_green()              -> (Vec<Finding>, GreenSummary, Vec<PerEndpointIoOps>)
  -> quality_gate::evaluate()          -> QualityGate
  -> Report { analysis, findings, green_summary, quality_gate }

In CI mode (--ci), the process exits with code 1 if the quality gate fails.

Streaming mode (perf-sentinel watch)

Runs as a daemon, receiving events in real time and emitting findings as they are detected.

OTLP gRPC (port 4317)  \
OTLP HTTP (port 4318)   +---> mpsc channel ---> TraceWindow (LRU + TTL)
JSON unix socket       /                               |
                                              +--------+--------+
                                              |                 |
                                        LRU eviction    TTL eviction
                                              |                 |
                                              +--------+--------+
                                                       |
                                          normalize -> detect -> score
                                                       |
                                              NDJSON findings (stdout)
                                              Prometheus /metrics

• Events are normalized outside the TraceWindow lock to minimize lock hold time.
• Valid sampled OTLP route, parent and consumer-destination context from named services is retained alongside events, so the window can resolve an outer same-service entry route, or a consumer span, that arrives in a later export. Endpoint contexts, consumer destinations and ancestry entries are independently bounded by max_events_per_trace. They share the trace LRU and TTL and never count as I/O events.
• Traces are evicted when the LRU cache is full (max_active_traces) or when TTL expires (trace_ttl_ms).
• On eviction, the trace is analyzed through detect and score stages.
• Findings are emitted as newline-delimited JSON to stdout, unless [daemon] findings_stdout = false.
• Prometheus metrics are exposed on the same HTTP port (4318) at /metrics.

Daemon query API

In watch mode, the daemon exposes its internal state via HTTP endpoints on port 4318 alongside /v1/traces and /metrics:

• GET /api/findings (filterable, capped at 1000)
• GET /api/findings/{trace_id}
• GET /api/explain/{trace_id} (tree with findings inline, from daemon memory)
• GET /api/correlations (cross-trace correlations, capped at 1000)
• GET /api/status (uptime, active traces, stored findings count)
• GET /api/config (effective [daemon] configuration, secrets summarized)
• GET /api/energy (live health of the energy and intensity backends)
• GET /api/export/report (live state as a Report JSON, pipe-compatible with report --input -)
• POST and DELETE /api/findings/{signature}/ack, GET /api/acks (runtime acknowledgments)
• POST and GET /api/incidents (Alertmanager incidents with their frozen findings)

A FindingsStore ring buffer retains recent findings for the API (sized by [daemon] max_retained_findings, default 10k). The companion CLI subcommand perf-sentinel query renders these endpoints in colored terminal output. Gated by [daemon] api_enabled (default true). See Limitations for the no-auth threat model.

Module responsibilities

Key types

Crate boundaries

sentinel-cli (binary)
  |
  +-- clap CLI: analyze / explain / diff / report / watch / query / ack / inspect / tempo / jaeger-query / pg-stat / mysql-stat / calibrate / disclose / verify-hash / hash-bake / demo / bench / man / completions
  |
  +-- depends on sentinel-core (library)
        |
        +-- All pipeline logic
        +-- Traits at borders only: IngestSource, ReportSink
        +-- Pure functions between stages

The CLI crate stays thin: it parses arguments, loads config and delegates to sentinel-core functions. All business logic lives in sentinel-core.

Cargo features

Whole subcommands are compile-time gated by Cargo features:

The published binary enables all six. Library consumers of sentinel-core start with everything off and opt in. tonic, axum, tower, prometheus and opentelemetry-proto remain unconditional dependencies of sentinel-core because report/metrics.rs and ingest/otlp/ use their types even when no listener runs.