Architecture

perf-sentinel is a polyglot performance anti-pattern detector built as a Rust workspace with two crates:

• sentinel-core : library containing all pipeline logic
• sentinel-cli : binary providing the CLI entry point (perf-sentinel)

Glossary

A few terms recur across the docs and code; the distinctions matter:

• Event — a single normalized SpanEvent. The pre-detection unit (input).
• Finding — a detected anti-pattern (output of detect). The product-level concept exposed in CLI output, JSON, SARIF, HTML, the daemon API and acks.
• Pattern — the normalized SQL/HTTP template that groups events for a finding (e.g. SELECT * FROM users WHERE id = ?).
• Detection — the act, or the family of detectors (n_plus_one, chatty, ...). Not a synonym of finding.

Operating modes:

• Batch mode (perf-sentinel analyze) — processes a complete trace set and produces a single Report. Used in CI integration tests, ad-hoc analysis, and perf-sentinel report (HTML).
• CI mode (perf-sentinel analyze --ci) — same as batch, but the quality gate failure exits with code 1. The Confidence field on each finding is stamped ci_batch (lowest confidence: limited traffic shapes).
• Daemon mode / streaming mode / watch mode (perf-sentinel watch) — long-running process ingesting OTLP/JSON events in real time, with TTL-based eviction and per-trace detection. Findings are stamped daemon_staging or daemon_production based on [daemon] daemon_environment (boosts severity in downstream consumers like perf-lint).

The Confidence axis (ci_batch < daemon_staging < daemon_production) is stamped automatically by the runtime and exposed in JSON, SARIF, HTML and the daemon query API.

Pipeline overview

Operating modes

Batch mode (perf-sentinel analyze)

Processes a complete set of events and produces a single report with quality gate evaluation.

Vec<SpanEvent>
  -> normalize::normalize_all()        -> Vec<NormalizedEvent>
  -> correlate::correlate()            -> Vec<Trace>
  -> detect::detect()                  -> Vec<Finding>
  -> score::score_green()              -> (Vec<Finding>, GreenSummary)
  -> quality_gate::evaluate()          -> QualityGate
  -> Report { analysis, findings, green_summary, quality_gate }

In CI mode (--ci), the process exits with code 1 if the quality gate fails.

Streaming mode (perf-sentinel watch)

Runs as a daemon, receiving events in real time and emitting findings as they are detected.

OTLP gRPC (port 4317)  \
OTLP HTTP (port 4318)   +---> mpsc channel ---> TraceWindow (LRU + TTL)
JSON unix socket       /                               |
                                              +--------+--------+
                                              |                 |
                                        LRU eviction    TTL eviction
                                              |                 |
                                              +--------+--------+
                                                       |
                                          normalize -> detect -> score
                                                       |
                                              NDJSON findings (stdout)
                                              Prometheus /metrics

• Events are normalized outside the TraceWindow lock to minimize lock hold time.
• Traces are evicted when the LRU cache is full (max_active_traces) or when TTL expires (trace_ttl_ms).
• On eviction, the trace is analyzed through detect and score stages.
• Findings are emitted as newline-delimited JSON to stdout.
• Prometheus metrics are exposed on the same HTTP port (4318) at /metrics.

Daemon query API

In watch mode, the daemon exposes its internal state via HTTP endpoints on port 4318 alongside /v1/traces and /metrics:

• GET /api/findings (filterable, capped at 1000)
• GET /api/findings/{trace_id}
• GET /api/explain/{trace_id} (tree with findings inline, from daemon memory)
• GET /api/correlations (cross-trace correlations, capped at 1000)
• GET /api/status (uptime, active traces, stored findings count)

A FindingsStore ring buffer retains recent findings for the API (sized by [daemon] max_retained_findings, default 10k). The companion CLI subcommand perf-sentinel query renders these endpoints in colored terminal output. Gated by [daemon] api_enabled (default true); see Limitations for the no-auth threat model.

Module responsibilities

Key types

Crate boundaries

sentinel-cli (binary)
  |
  +-- clap CLI: analyze / explain / diff / report / watch / query / ack / inspect / tempo / jaeger-query / pg-stat / calibrate / disclose / verify-hash / hash-bake / demo / bench / man / completions
  |
  +-- depends on sentinel-core (library)
        |
        +-- All pipeline logic
        +-- Traits at borders only: IngestSource, ReportSink
        +-- Pure functions between stages

The CLI crate is intentionally thin: it parses arguments, loads config and delegates to sentinel-core functions. All business logic lives in sentinel-core.

Cargo features

Whole subcommands are compile-time gated by Cargo features:

The published binary enables all four. Library consumers of sentinel-core start with everything off and opt in. Note that tonic, axum, tower, prometheus and opentelemetry-proto remain unconditional dependencies of sentinel-core because report/metrics.rs and ingest/otlp.rs use their types even when no listener runs.