// documentation

Get Perf Sentinel running

A single self-hosted binary that reads the OpenTelemetry traces your services already emit, flags wasteful I/O, and prices it in energy and carbon. This guide goes from the first run to a CI gate.

Quickstart

Five steps, from the built-in demo to a live daemon. Nothing to install on your application side, but your spans must already carry the query text (db.statement) and the target URL (http.url): spans without those attributes are dropped (the report counts them on OTLP input), so audit your own tracing first with perf-sentinel inspect. Perf Sentinel analyzes SQL queries, outbound HTTP and RPC calls, and message publishes. Non-SQL datastores such as Redis, MongoDB or Elasticsearch are skipped, and so is the consumer side of messaging.

Before installing, see how a trace becomes a findingThe animated synoptic of Perf Sentinel, from the map of environments down to the detection rules.Open the synoptic →

bash

# 1. Try the built-in demo (no app-side setup)
perf-sentinel demo                   # colored terminal report
perf-sentinel demo --tui             # interactive TUI report
perf-sentinel demo --html demo.html  # HTML dashboard

# 2. Capture traces from your test suite, no Collector needed
perf-sentinel capture --output traces.json -- mvn verify   # or pytest, npm test, go test ./...

# 3. Analyze a captured trace file
perf-sentinel analyze --input traces.json

# 4. Use it as a CI quality gate (exit 1 over threshold)
perf-sentinel analyze --input traces.json --ci --config .perf-sentinel.toml

# 5. Stream traces from your apps (daemon mode)
perf-sentinel watch

demo --html is a full showcase: every dashboard tab is populated (Overview, Findings with inline Explain, Carbon, pg_stat, mysql_stat, Diff and synthesized cross-trace correlations).

On dd-trace? Bridge it through the Collector datadogreceiver, then either feed its file exporter OTLP JSON straight to analyze --input for batch (auto-detected), or stream to the watch daemon over OTLP. See Coming from Datadog.

With Claude Code? The Perf Sentinel plugin writes performance integration tests for your current change and runs them under Perf Sentinel, and reads your production findings from the Hub to fix them in small batches, each proved by a non-regression test. It asks before every change. Install it with /plugin marketplace add Institut-du-Numerique-Responsable/perf-sentinel-claude, then /plugin install perf-sentinel@perf-sentinel. See the plugin on GitHub.

Installation

Pick crates.io, a prebuilt binary, or the container image. Linux binaries target musl, so they are static and run on any distribution, including FROM scratch images. In a CI pipeline, pin an exact version and read the CHANGELOG before each version bump. Finding identities and quality gate figures can move between versions.

bash

# from crates.io
cargo install perf-sentinel --locked

# or a prebuilt binary (Linux amd64/arm64, macOS arm64, Windows amd64)
chmod +x perf-sentinel-linux-amd64
sudo mv perf-sentinel-linux-amd64 /usr/local/bin/perf-sentinel

# or via Docker
docker run --rm -p 4317:4317 -p 4318:4318 \
  ghcr.io/Institut-du-Numerique-Responsable/perf-sentinel:latest watch --listen-address 0.0.0.0

A Helm chart is published for Kubernetes deployments. Container images are available on ghcr.io and Docker Hub.

Configuration

Drop a .perf-sentinel.toml at the repository root. Thresholds drive the CI gate, and detection tunes how anti-patterns are flagged.

.perf-sentinel.toml

[thresholds]
n_plus_one_sql_critical_max = 0   # zero-tolerance for N+1 SQL
io_waste_ratio_max = 0.30         # at most 30% avoidable I/O

[detection]
n_plus_one_min_occurrences = 5
slow_query_threshold_ms = 500

thresholds.n_plus_one_sql_critical_max

Max critical N+1 SQL findings before the gate fails. 0 = zero tolerance.

thresholds.io_waste_ratio_max

Maximum share of avoidable I/O tolerated, between 0 and 1.

detection.n_plus_one_min_occurrences

Repetitions of a query template with different parameters within a trace before it counts as N+1.

detection.slow_query_threshold_ms

Duration above which a SQL query, HTTP call or broker publish counts as slow. A finding needs 3 slow occurrences of the same template in one trace.

CLI reference

18 subcommands in one binary. Run perf-sentinel <cmd> --help for the full flag list.

The CLI level in the synoptic →

demo Built-in synthetic dataset, no capture required.

$ perf-sentinel demo --html demo.html

analyze Analyze captured traces. --ci to gate, --tui to drill down, --format json|sarif.

$ perf-sentinel analyze --input traces.json --ci

watch OTLP daemon: gRPC :4317 / HTTP :4318 ingestion, Prometheus /metrics, /health, query API.

$ perf-sentinel watch --listen-address 0.0.0.0

report Single-file offline HTML dashboard, light/dark, with CSV export.

$ perf-sentinel report --input traces.json --output report.html

// more subcommands

capture Receive OTLP into a trace file, no Collector needed.

diff Compare two analyses to catch regressions in CI.

explain Drill into one trace (--trace-id), annotated span tree.

inspect Standalone four-panel TUI.

pg-stat pg_stat_statements hotspots, ranked by total time.

mysql-stat performance_schema digests, ranked by total time.

calibrate Per-service energy factors from measured power.

disclose Verifiable periodic disclosure, signable via Sigstore.

disclose-pdf Renders a periodic report as a PDF/A-3b that carries its source files, once its fingerprint and attestation check out.

verify-hash Rechecks a report, JSON or PDF: fingerprint, attestation, signature and binary provenance.

verify-card Checks a share card, file or verify link: its signature, whether the file is the one signed, and the key its domain publishes.

card Signs and draws a share card from the figures the Hub prepared (card keygen, card render).

login / hub Signs the terminal in to a Hub, then lists findings and incidents, acknowledges them and runs analyses through it.

tempo Fetch traces straight from a Grafana Tempo backend.

jaeger-query Fetch from Jaeger or Victoria Traces.

query Query the live daemon (HTTP API and TUI).

ack Manage acknowledged, known findings.

One-liner cheat sheet for the rest of the surface

perf-sentinel explain --input traces.json --trace-id abc123        # tree view of one trace
perf-sentinel inspect --input traces.json                          # interactive TUI
perf-sentinel diff --before base.json --after head.json            # PR regression diff
perf-sentinel pg-stat --input pg_stat.csv --traces traces.json     # PostgreSQL hotspots
perf-sentinel mysql-stat --input mysql_perf.csv --traces traces.json  # MySQL hotspots
perf-sentinel tempo --endpoint http://tempo:3200 --trace-id <id>   # pull from Grafana Tempo
perf-sentinel jaeger-query --endpoint http://jaeger:16686 --service order-svc
perf-sentinel calibrate --traces traces.json --measured-energy rapl.csv
perf-sentinel completions zsh > ~/.zfunc/_perf-sentinel            # shell completions
perf-sentinel man > perf-sentinel.1                                # man page
perf-sentinel query findings --service order-svc                   # talk to a running daemon

The whole CLI surface in one map (18 subcommands, one pipeline, 11 outputs)

Perf Sentinel CLI command map: 18 subcommands, one shared pipeline, 11 output sinks

Input & output formats

Traces come in from files or live OTLP, and results go out as terminal text, machine-readable JSON/SARIF, an HTML dashboard, or live daemon streams. Output is deterministic: identical input yields byte-identical JSON and SARIF, findings sorted on a stable key rather than HashMap order, so a CI quality gate gives the same verdict on every run.

Format ingestion, stage 1 of the synoptic →

Inputs

Trace files

Native Perf Sentinel JSON, OTLP JSON, Jaeger JSON, Zipkin v2. Auto-detected from the first bytes, no --format flag.

Live OTLP

gRPC :4317 and HTTP :4318, ingested by the watch daemon from your OTel Collector or SDK. On dd-trace? Bridge it through the Collector datadogreceiver, see Coming from Datadog.

Grafana Tempo

Pull traces directly from a Tempo backend with perf-sentinel tempo.

Jaeger query API

Pull from a Jaeger or Victoria Traces backend with perf-sentinel jaeger-query.

pg_stat_statements

Rank PostgreSQL hotspots from the catalog view with perf-sentinel pg-stat.

performance_schema

Rank MySQL hotspots from the digest table with perf-sentinel mysql-stat.

Outputs

text (default)

Colored terminal output, grouped by severity.

json

Structured report with a documented schema.

sarif (v2.1.0)

GitHub/GitLab code scanning with inline PR annotations.

HTML dashboard

Single-file offline report with CSV export.

Interactive TUI

Analyze, Inspect and Explain in one keyboard drill-down.

Live daemon

NDJSON findings, Prometheus /metrics, /health probe, query API.

Periodic disclosure

Hash-verifiable perf-sentinel-report/v1.8 JSON, Sigstore-signable.

Disclosure PDF

PDF/A-3b rendering of a periodic report for readers without tools, with the JSON, the attestation and the bundle attached.

Hub export

Bounded, coalesced push of live findings, incidents and correlations to PerfSentinelHub, retried on failure. A finding it has to drop is named by the advisor and fires an alert.

GreenOps metrics

Every finding carries a GreenOps reading. The figures are directional and meant to rank and trend, not to certify (for now).

Waste, energy and carbon, stages 5 to 7 of the synoptic →

I/O intensity score (IIS) I/O ops ÷ invocations

An endpoint's total I/O operations divided by its number of invocations. The higher it is, the more I/O each call costs.

I/O waste ratio avoidable ÷ total

Avoidable operations over total operations: the share you can directly recover by fixing N+1s and redundant calls.

co2.total (E × I) + M

Reported as the Software Carbon Intensity v1.0 numerator (ISO/IEC 21031:2024): energy used (E) times grid carbon intensity (I), plus embodied hardware emissions (M), summed over the analyzed traces.

Bands (stable across versions)

healthy moderate high critical

io_intensity_band and io_waste_ratio_band enum values are stable across versions. The underlying numeric thresholds may evolve.

Which energy source on which infrastructure

No infrastructure prerequisite: the I/O proxy model and the embedded grid tables produce estimates from the first trace, and measured sources refine them where the infrastructure allows. When several sources cover one service, the most precise wins automatically, per service.

Your infrastructure

Energy refinement

Precision

Any, zero setup

I/O proxy model (default)

directional, ~2x bracket

Cloud VMs (AWS, GCP, Azure)

Cloud SPECpower (CPU% + instance type)

~±30%

Kubernetes or any Linux x86 host with RAPL

Kepler (RAPL, per container or process)

good, CPU package only by default

Bare metal x86 (incl. AWS *.metal, OVH, Hetzner, Scaleway)

Alumet (recommended) or Scaphandre (RAPL)

highest tier

Physical servers with a BMC

Redfish (chassis power read from the BMC)

node-level, periphery included

Databases and brokers, self-hosted or managed (RDS, Aurora, MSK…)

Alumet on their host, or the declared cluster priced from SPECpower (ceiling or CPU)

host energy, counted in the totals

Anywhere, on top of any row above

Electricity Maps (real-time grid intensity)

refines the I axis, not E

FR, GB, FI, DK, ES, AU, HU, NZ, on top of any row above

Grid operator feeds (RTE, NESO, Fingrid, Energinet, REE, Open Electricity, MAVIR, em6), free

refines the I axis, direct emissions

The carbon side prices the detected I/O with an activity-based methodology, region-hourly grid intensity (embedded profiles from ENTSO-E, EIA and more, or live from Electricity Maps or eight grid operators such as RTE and NESO), bottom-up embodied carbon (Boavizta and HotCarbon 2024) and Sigstore-signed, hash-verifiable disclosures.

It is suitable as a primary data source for a horizontal carbon accounting platform, or as an internal controlling tool for software-emissions KPIs and RGESN conformance.

It is not yet third-party verified for standalone CSRD or GHG Protocol Scope 2/3 inventory reporting, which requires audit by a qualified body and integration with non-IT scopes. CO₂ figures carry a ~2× uncertainty bracket in the default proxy mode, tighter with any measured-energy source (Alumet or Scaphandre RAPL, Kepler eBPF, Redfish BMC, or cloud SPECpower with calibration). Sources and bounds: estimate accuracy and the methodology.

Energy & carbon, in plain language →

Concrete pairings: pass the I/O counts and per-region energy estimates to Watershed, Sweep, Greenly or Persefoni as activity data, or use Perf Sentinel directly to demonstrate RGESN (Référentiel Général d’Écoconception de Services Numériques, ARCEP/Ademe/DINUM 2024) software-optimization conformance, where N+1 detection, redundant calls, caching and fanout reduction map onto the corresponding criteria.

For organisations who still want a non-regulatory periodic efficiency disclosure (quarterly or yearly JSON, optional Sigstore signature, a PDF for readers without tools, publishable to PerfSentinelHub, intentionally kept off the main quickstart path), the optional perf-sentinel disclose workflow is documented in the disclosure guide.

Performance

perf-sentinel bench times the analysis pipeline only (normalize → correlate → detect → score), single-threaded, on synthetic datasets: the pure pipeline cost, not end-to-end throughput nor a daemon-under-load benchmark.

Dataset (44,043 synthetic events)

Platform

Pipeline throughput

p50 / p99 per event

Repeated pattern

x86 Xeon 8481C

~576k evt/s

1.72 / 1.88 µs

Repeated pattern

Apple M4 Pro

~1.23M evt/s

0.81 / 0.89 µs

Varied SQL

x86 Xeon 8481C

~640k evt/s

1.54 / 1.69 µs

Varied SQL

Apple M4 Pro

~1.33M evt/s

0.75 / 0.81 µs

x86: GCP c3-standard-8 (Xeon 8481C @ 2.70 GHz, 8 vCPU), measured June 2026.
M4: Mac mini M4 Pro (12 cores, 24 GB), measured 2026-06-08.

Both measurements use release 0.8.5 (musl+mimalloc on x86, system allocator native on M4). The pipeline has changed since. With the native artifacts the M4 Pro sustains about 2.1× one 8481C vCPU (2.14× repeated, 2.08× varied). p50 / p99 are per-event latency over 10 iterations. Daemon memory: ~17 MB RSS at rest, peaking ~190 MB under a sustained ~1.0M evt/s ingestion load (down from 237 MB on 0.6.1, under the 250 MB ceiling). Those are lab figures with an empty findings ring, and they are not what you size a pod with: the memory note below has the production numbers. Rust edition 2024, rustc 1.96.0. Reproduce with perf-sentinel bench --help.

Bench methodology (clock scope, datasets)

File reads, JSON parsing and ingestion all happen before the clock starts, and input batches are cloned up front. The pipeline is single-threaded (no rayon), so core count does not change throughput. Both datasets are 44,043 synthetic events built by duplicating the demo fixture. One repeats the same pattern and the other uses random SQL per query. This isolates pipeline throughput but does not reflect real-production diversity.

Same-chip allocator breakdown

The x86 musl artifact links mimalloc while the macOS arm64 artifact uses the system allocator, so cross-platform binaries differ by allocator as well as ISA. On the same M4 Pro the musl+mimalloc build (the linux/arm64 artifact in a Docker container) reaches ~1.39M / ~1.51M evt/s versus ~1.23M / ~1.33M native, about 13% faster, confirming the allocator as the main cause of higher Docker throughput. At equal build (both musl+mimalloc), the M4 Pro is ~2.4× the x86 8481C (2.41× repeated, 2.36× varied).

Memory: lab idle vs a production daemon

bench prints rss_peak_bytes, but that is dominated by the pre-cloned input batches (10 iterations × 44,043 events), not the daemon footprint, and it is not comparable across operating systems (current RSS from /proc on Linux, peak RSS via getrusage on macOS). Profiled separately on the same M4 Pro in a Docker Desktop linux/arm64 VM, the daemon idles at ~17 MB in musl+mimalloc (~10 MB native, mimalloc trades a little RSS for allocator speed) and peaks ~190 MB under a sustained ~1.0M evt/s load, down from 237 MB on 0.6.1, under the 250 MB ceiling. A production daemon is a different shape. On a Kubernetes StatefulSet running 0.19.0 with max_retained_findings = 100000, process_resident_memory_bytes starts near 88 MB, climbs while the ring fills and then holds: over 7 days, a 980 MB median and a 1.2 GB p90 that is the day-to-day ceiling, with one excursion to 1.8 GB that decayed back over the following day, no restart and no OOM under a 2 GiB limit with a 1 GiB request. Ingestion throughput did not drive it (30 evt/s median, 520 evt/s p95, 2.1k evt/s peak). The retained findings did, at roughly 3 to 6 KB of RSS each depending on how much evidence a finding carries. So size the pod from max_retained_findings, not from the idle figure: a few hundred MB at the 10000 default, 1 to 2 GB at 100000, and 0 hands the whole store back when the query API is off.

How it compares

Perf Sentinel’s niche: lightweight, protocol-agnostic, CI-native and carbon-aware, not a replacement for a full observability suite.

Capability

Hypersistence

Datadog

New Relic

Sentry

Digma

Pyroscope

OTJAE

Perf Sentinel

N+1 SQL detection

JPA only, test-time

Yes (DBM)

Yes

Yes (OOTB)

Yes, IDE (JVM/.NET)

Yes, protocol-level

N+1 HTTP detection

Yes

Partial

Yes

Polyglot support

Java only

Per-language agents

Most languages (SDK)

JVM + .NET

eBPF + SDKs

JVM only

Any OTel runtime

Cross-service correlation

Yes

Limited

Trace-to-profile

Intra-JVM only

Via trace ID

Per-span carbon/energy

Yes (CCF)

Yes (SCI, directional)

GreenOps score (IIS, waste)

Built-in

Runtime footprint

Library

Host agent ~130 MB + tracers

In-process agent, RSS unpublished

SDK + backend

Local backend

Alloy or SDK, scales with load

JVM agent

One daemon, 17 MB idle

Native CI/CD quality gate

Manual test asserts

Alerts, no gate

Yes (exit 1)

License

Commercial

Proprietary SaaS

FSL → Apache-2

Freemium

AGPL-3.0

Apache-2.0

AGPL-3.0

The Datadog figure is the one Datadog publishes for its Agent 7.34 on a c5.xlarge instance, excluding the tracers loaded in each service. New Relic publishes no RSS figure for its APM agents, and Grafana sizes Alloy by profile volume. The Perf Sentinel figure likewise excludes the OTel instrumentation it reads. Real overhead depends on instrumentation scope.

Acknowledging known findings

Drop a .perf-sentinel-acknowledgments.toml at your repo root to suppress findings the team has accepted: each gets a SHA-256 signature, and they are filtered from analyze / report / inspect / diff and don’t count toward the quality gate. Runtime acks against a live daemon are exposed via the ack CLI, the live HTML dashboard, the TUI, and the findings dashboard, whose Ack column opens the ack page of a PerfSentinelHub that relays to the daemon.

Acknowledgment workflow →

Deployment

Pick the lightest deployment that fits where your traces live. The central collector is a single stateful daemon, so horizontal replicas need trace-id-aware load balancing and do not share correlation state. Upstream trace sampling (head-based vs tail-based) and the daemon’s own sampling_rate undercount the repetition-based detectors, and under sustained overload the daemon sheds whole analysis batches rather than blocking ingestion. Every shed batch is counted in the metrics.

The deployments on the synoptic map →

Standalone

Local analysis, TUI and single-file HTML reports. Nothing deployed on the app side.

CI batch

perf-sentinel analyze --ci on captured traces, exit 1 over a threshold. The simplest gate, nothing long-running.

Sidecar daemon

One watch daemon per service, ingesting that service’s OTLP traces right beside it for isolated debugging.

Central collector

An OpenTelemetry Collector routes to a single long-running watch daemon, with Prometheus /metrics and the query API. The daemon keeps its window of traces in memory: a clean shutdown finishes the analysis, a hard kill loses it.

Hub-backed fleet

Daemons push their findings to PerfSentinelHub, a separate service that keeps a queryable history across services and the date each finding was first seen. The central collector above holds a bounded ring and shares no state between replicas, so it cannot provide that history. A scheduled job can also publish the periodic reports there: the Hub checks them, renders their PDF and shares them under an expiring link. A daemon that does not push is polled instead, and the Hub counts the findings its ring evicted before a poll could read them. Its recap screen ranks, by day, week or month, what to fix first and what moved, in an engineering and a management view. With Hub:Recaps:Subscriptions set, each closed period is frozen into a report sent by signed webhook and e-mail. Share cards need engine 0.39.0 or later, Hub:Cards:Organisation set by an admin and the cards.write permission (role member): the recap screen then shows “Share a result” under the reports, and anyone checks a card at perf-sentinel.dev/verifier or with perf-sentinel verify-card.

Local dev

CI/CD

Staging

Production

GreenOps (cross-cutting)

End-to-end

Live operator monitor over a running daemon, for DevOps / SRE, four Tab-cycled tabs (Advisor hints, Energy/carbon mix, Trends charts, Scrapers health) via perf-sentinel query --daemon <URL> monitor:

Two example Grafana dashboards ship in the repository. Neither ships in the Helm chart. You import them into the Grafana you already run. The overview reads the Prometheus /metrics counters, the findings one reads the query API through the Infinity plugin, and both carry a Compatibility badge that says when the daemon feeding them is older than their panels need. A collapsed History (Hub) row reads PerfSentinelHub instead, over the time picker’s range, where the daemon’s ring is bounded by volume and cannot look back:

Full integration guide →

Privacy & security

Perf Sentinel runs inside your network and makes no outbound call you did not configure.

✓

On-prem processing

Traces are analyzed in place. No silent outbound calls, no usage telemetry baked in. Raw span content lives in memory only (30s TTL, 10k-trace LRU), and the daemon never writes it to disk. What it emits carries the normalized template, with SQL literals and numeric or UUID path segments replaced by placeholders, and a few documented exceptions.

✓

Localhost by default

The binary listens on 127.0.0.1, the Helm chart on 0.0.0.0 inside the pod. TLS, CORS and the ack API key are all opt-in.

✓

Locked-down surface

Read-only GET endpoints and the OTLP ingestion listeners can be restricted, and the no-auth threat model is documented. The endpoints trust their senders, so keep ingestion on a trusted network and put a reverse proxy or network policy in front before exposing anything beyond localhost.

Limitations & threat model →

Supply chain & releases

Every GitHub Action is pinned to a 40-character commit SHA, the production image is FROM scratch, Cargo.lock is committed and audited daily by cargo audit, and workflow token permissions default to read-only. Release tags are signed, and binaries carry SLSA build provenance attested through Sigstore.

Releases follow a documented procedure gated by a mandatory simulation-lab run, with the Helm chart versioned in lockstep. Release binaries carry SLSA build provenance (Build Level 2, Sigstore + Rekor) and embedded cargo-auditable data (cargo audit bin), and each release ships an SPDX SBOM attested under the SPDX predicate. Dependabot opens weekly grouped PRs.

Supply chain & provenance →