// documentation

Get perf-sentinel running

A single self-hosted binary that reads the OpenTelemetry traces your services already emit, flags wasteful I/O, and prices it in energy and carbon. From first run to CI gate.

Quickstart

Five steps, from the built-in demo to a live daemon. Nothing to install on your application side, but your spans must already carry the query text (db.statement) and the target URL (http.url): spans without those attributes are dropped silently, so audit your own tracing first with perf-sentinel inspect.

bash

# 1. Try the built-in demo (no app-side setup)
perf-sentinel demo                   # colored terminal report
perf-sentinel demo --tui             # interactive TUI report
perf-sentinel demo --html demo.html  # HTML dashboard

# 2. Capture traces from your test suite, no Collector needed
perf-sentinel capture --output traces.json -- mvn verify   # or pytest, npm test, go test ./...

# 3. Analyze a captured trace file
perf-sentinel analyze --input traces.json

# 4. Use it as a CI quality gate (exit 1 over threshold)
perf-sentinel analyze --input traces.json --ci --config .perf-sentinel.toml

# 5. Stream traces from your apps (daemon mode)
perf-sentinel watch

demo --html is a full showcase: every dashboard tab is populated (Overview, Findings with inline Explain, Carbon, pg_stat, mysql_stat, Diff and synthesized cross-trace correlations).

On dd-trace? Bridge it through the Collector datadogreceiver, then either feed its file exporter OTLP JSON straight to analyze --input for batch (auto-detected), or stream to the watch daemon over OTLP. See Coming from Datadog.

Installation

Pick crates.io, a prebuilt binary, or the container image. Linux binaries target musl, so they are static and run on any distribution, including FROM scratch images.

bash

# from crates.io
cargo install perf-sentinel --locked

# or a prebuilt binary (Linux amd64/arm64, macOS arm64, Windows amd64)
chmod +x perf-sentinel-linux-amd64
sudo mv perf-sentinel-linux-amd64 /usr/local/bin/perf-sentinel

# or via Docker
docker run --rm -p 4317:4317 -p 4318:4318 \
  ghcr.io/robintra/perf-sentinel:latest watch --listen-address 0.0.0.0

A Helm chart is published for Kubernetes deployments. Container images are available on ghcr.io and Docker Hub.

Configuration

Drop a .perf-sentinel.toml at the repository root. Thresholds drive the CI gate; detection tunes how anti-patterns are flagged.

.perf-sentinel.toml

[thresholds]
n_plus_one_sql_critical_max = 0   # zero-tolerance for N+1 SQL
io_waste_ratio_max = 0.30         # at most 30% avoidable I/O

[detection]
n_plus_one_min_occurrences = 5
slow_query_threshold_ms = 500

thresholds.n_plus_one_sql_critical_max

Max critical N+1 SQL findings before the gate fails. 0 = zero tolerance.

thresholds.io_waste_ratio_max

Maximum share of avoidable I/O tolerated, between 0 and 1.

detection.n_plus_one_min_occurrences

Repetitions of a query template within a trace before it counts as N+1.

detection.slow_query_threshold_ms

Duration above which a SQL query, HTTP call or broker publish is flagged as slow.

CLI reference

One binary, a handful of subcommands. Run perf-sentinel <cmd> --help for the full flag list.

demo Built-in synthetic dataset, no capture required.

$ perf-sentinel demo --html demo.html

analyze Analyze captured traces. --ci to gate, --tui to drill down, --format json|sarif.

$ perf-sentinel analyze --input traces.json --ci

watch OTLP daemon: gRPC :4317 / HTTP :4318 ingestion, Prometheus /metrics, /health, query API.

$ perf-sentinel watch --listen-address 0.0.0.0

report Single-file offline HTML dashboard, light/dark, with CSV export.

$ perf-sentinel report --input traces.json --output report.html

// more subcommands

capture Receive OTLP into a trace file, no Collector needed.

diff Compare two analyses to catch regressions in CI.

explain Drill into one trace (--trace-id), annotated span tree.

inspect Standalone four-panel TUI.

pg-stat pg_stat_statements hotspots, ranked by total time.

mysql-stat performance_schema digests, ranked by total time.

calibrate Per-service energy factors from measured power.

disclose Verifiable periodic disclosure, signable via Sigstore.

tempo Fetch traces straight from a Grafana Tempo backend.

jaeger-query Fetch from Jaeger or Victoria Traces.

query Query the live daemon (HTTP API and TUI).

ack Manage acknowledged, known findings.

One-liner cheat sheet for the rest of the surface

perf-sentinel explain --input traces.json --trace-id abc123        # tree view of one trace
perf-sentinel inspect --input traces.json                          # interactive TUI
perf-sentinel diff --before base.json --after head.json            # PR regression diff
perf-sentinel pg-stat --input pg_stat.csv --traces traces.json     # PostgreSQL hotspots
perf-sentinel mysql-stat --input mysql_perf.csv --traces traces.json  # MySQL hotspots
perf-sentinel tempo --endpoint http://tempo:3200 --trace-id <id>   # pull from Grafana Tempo
perf-sentinel jaeger-query --endpoint http://jaeger:16686 --service order-svc
perf-sentinel calibrate --traces traces.json --measured-energy rapl.csv
perf-sentinel completions zsh > ~/.zfunc/_perf-sentinel            # shell completions
perf-sentinel man > perf-sentinel.1                                # man page
perf-sentinel query findings --service order-svc                   # talk to a running daemon

The whole CLI surface in one map (18 subcommands, one pipeline, 11 outputs)

perf-sentinel CLI command map: 18 subcommands, one shared pipeline, 11 output sinks

Input & output formats

Traces come in from files or live OTLP; results go out as terminal text, machine-readable JSON/SARIF, an HTML dashboard, or live daemon streams. Output is deterministic: identical input yields byte-identical JSON and SARIF, findings sorted on a stable key rather than HashMap order, so a CI quality gate never flickers.

Inputs

Trace files

Native perf-sentinel JSON, OTLP JSON, Jaeger JSON, Zipkin v2. Auto-detected from the first bytes, no --format flag.

Live OTLP

gRPC :4317 and HTTP :4318, ingested by the watch daemon from your OTel Collector or SDK. On dd-trace? Bridge it through the Collector datadogreceiver, see Coming from Datadog.

Grafana Tempo

Pull traces directly from a Tempo backend with perf-sentinel tempo.

Jaeger query API

Pull from a Jaeger or Victoria Traces backend with perf-sentinel jaeger-query.

pg_stat_statements

Rank PostgreSQL hotspots from the catalog view with perf-sentinel pg-stat.

performance_schema

Rank MySQL hotspots from the digest table with perf-sentinel mysql-stat.

Outputs

text (default)

Colored terminal output, grouped by severity.

json

Structured report with a documented schema.

sarif (v2.1.0)

GitHub/GitLab code scanning with inline PR annotations.

HTML dashboard

Single-file offline report with CSV export.

Interactive TUI

Analyze, Inspect and Explain in one keyboard drill-down.

Live daemon

NDJSON findings, Prometheus /metrics, /health probe, query API.

Periodic disclosure

Hash-verifiable perf-sentinel-report/v1.0 JSON, Sigstore-signable.

GreenOps metrics

Every finding carries a GreenOps reading. The figures are directional and meant to rank and trend, not to certify (for now).

I/O intensity score (IIS) I/O ops ÷ invocations

An endpoint's total I/O operations divided by its number of invocations. The higher it is, the more I/O each call costs.

I/O waste ratio avoidable ÷ total

Avoidable operations over total operations: the share you can directly recover by fixing N+1s and redundant calls.

co2.total (E × I) + M

Reported as the Software Carbon Intensity v1.0 numerator (ISO/IEC 21031:2024): energy used (E) times grid carbon intensity (I), plus embodied hardware emissions (M), summed over the analyzed traces.

Bands (stable across versions)

healthy moderate high critical

io_intensity_band and io_waste_ratio_band enum values are stable across versions; the underlying numeric thresholds may evolve.

Which energy source on which infrastructure

No infrastructure prerequisite: the I/O proxy model and the embedded grid tables produce estimates from the first trace, and measured sources refine them where the infrastructure allows. When several sources cover one service, the most precise wins automatically, per service.

Your infrastructure

Energy refinement

Precision

Any, zero setup

I/O proxy model (default)

directional, ~2x bracket

Cloud VMs (AWS, GCP, Azure)

Cloud SPECpower (CPU% + instance type)

~±30%

Kubernetes, cloud or on-prem

Kepler (eBPF per container)

good, best on RAPL nodes

Bare metal x86 (incl. AWS *.metal, OVH, Hetzner, Scaleway)

Alumet (recommended) or Scaphandre (RAPL)

highest tier

Physical servers with a BMC

Redfish (wall-plug power per chassis)

node-level, periphery included

Anywhere, on top of any row above

Electricity Maps (real-time grid intensity)

refines the I axis, not E

The carbon side prices the detected I/O with the rigor of a specialized software and compute emissions calculator: activity-based methodology, region-hourly grid intensity (Electricity Maps, ENTSO-E, RTE, National Grid ESO, EIA, and more), bottom-up embodied carbon (Boavizta and HotCarbon 2024) and Sigstore-signed, hash-verifiable disclosures.

It is suitable as a primary data source for a horizontal carbon accounting platform, or as an internal controlling tool for software-emissions KPIs and RGESN conformance.

It is not yet third-party verified for standalone CSRD or GHG Protocol Scope 2/3 inventory reporting, which requires audit by a qualified body and integration with non-IT scopes. CO₂ figures carry a ~2× uncertainty bracket in the default proxy mode, tighter with any measured-energy source (Alumet or Scaphandre RAPL, Kepler eBPF, Redfish BMC, or cloud SPECpower with calibration). Sources and bounds: estimate accuracy and the methodology.

Energy & carbon, in plain language →

Concrete pairings: pass the I/O counts and per-region energy estimates to Watershed, Sweep, Greenly or Persefoni as activity data, or use perf-sentinel directly to demonstrate RGESN (Référentiel Général d’Écoconception de Services Numériques, ARCEP/Ademe/DINUM 2024) software-optimization conformance, where N+1 detection, redundant calls, caching and fanout reduction map onto the corresponding criteria.

For organisations who still want a non-regulatory periodic efficiency disclosure (quarterly or yearly JSON, optional Sigstore signature, intentionally kept off the main quickstart path), the optional perf-sentinel disclose workflow is documented in the disclosure guide.

Performance

perf-sentinel bench times the analysis pipeline only (normalize → correlate → detect → score), single-threaded, on synthetic datasets: the pure pipeline cost, not end-to-end throughput nor a daemon-under-load benchmark.

Dataset (44,043 synthetic events)

Platform

Pipeline throughput

p50 / p99 per event

Repeated pattern

x86 Xeon 8481C

~576k evt/s

1.72 / 1.88 µs

Repeated pattern

Apple M4 Pro

~1.23M evt/s

0.81 / 0.89 µs

Varied SQL

x86 Xeon 8481C

~640k evt/s

1.54 / 1.69 µs

Varied SQL

Apple M4 Pro

~1.33M evt/s

0.75 / 0.81 µs

x86: GCP c3-standard-8 (Xeon 8481C @ 2.70 GHz, 8 vCPU), measured June 2026.
M4: Mac mini M4 Pro (12 cores, 24 GB), measured 2026-06-08.

Both release 0.8.5 (musl+mimalloc on x86, system allocator native on M4). With the native artifacts the M4 Pro sustains about 2.1× one 8481C vCPU (2.14× repeated, 2.08× varied). p50 / p99 are per-event latency over 10 iterations. Daemon memory: ~17 MB RSS at rest, peaking ~190 MB under a sustained ~1.0M evt/s ingestion load (down from 237 MB on 0.6.1, under the 250 MB ceiling). Rust edition 2024, rustc 1.96.0. Reproduce with perf-sentinel bench --help.

Bench methodology (clock scope, datasets)

File reads, JSON parsing and ingestion all happen before the clock starts, and input batches are cloned up front. The pipeline is single-threaded (no rayon), so core count does not change throughput. Both datasets are 44,043 synthetic events built by duplicating the demo fixture, one repeats the same pattern, the other uses random SQL per query. This isolates pipeline throughput but does not reflect real-production diversity.

Same-chip allocator breakdown

The x86 musl artifact links mimalloc while the macOS arm64 artifact uses the system allocator, so cross-platform binaries differ by allocator as well as ISA. On the same M4 Pro the musl+mimalloc build (the linux/arm64 artifact in a Docker container) reaches ~1.39M / ~1.51M evt/s versus ~1.23M / ~1.33M native, about 13% faster, confirming the allocator as the main cause of higher Docker throughput. At equal build (both musl+mimalloc), the M4 Pro is ~2.4× the x86 8481C (2.41× repeated, 2.36× varied).

Memory: bench rss_peak vs daemon footprint

bench prints rss_peak_bytes, but that is dominated by the pre-cloned input batches (10 iterations × 44,043 events), not the daemon footprint, and it is not comparable across operating systems (current RSS from /proc on Linux, peak RSS via getrusage on macOS). Profiled separately on the same M4 Pro in a Docker Desktop linux/arm64 VM, the daemon idles at ~17 MB in musl+mimalloc (~10 MB native, mimalloc trades a little RSS for allocator speed) and peaks ~190 MB under a sustained ~1.0M evt/s load, down from 237 MB on 0.6.1, under the 250 MB ceiling.

How it compares

perf-sentinel’s niche: lightweight, protocol-agnostic, CI-native and carbon-aware, not a replacement for a full observability suite.

Capability

Hypersistence

Datadog

New Relic

Sentry

Digma

Pyroscope

OTJAE

perf-sentinel

N+1 SQL detection

JPA only, test-time

Yes (DBM)

Yes

Yes (OOTB)

Yes, IDE (JVM/.NET)

Yes, protocol-level

N+1 HTTP detection

Yes

Partial

Yes

Polyglot support

Java only

Per-language agents

Most languages (SDK)

JVM + .NET

eBPF + SDKs

JVM only

Any OTel runtime

Cross-service correlation

Yes

Limited

Trace-to-profile

Intra-JVM only

Via trace ID

Per-span carbon/energy

Yes (CCF)

Yes (SCI, directional)

GreenOps score (IIS, waste)

Built-in

Runtime footprint

Library

Agent ~100–150 MB

SDK + backend

Local backend

Agent ~50–100 MB

JVM agent

Standalone <20 MB

Native CI/CD quality gate

Manual test asserts

Alerts, no gate

Yes (exit 1)

License

Commercial

Proprietary SaaS

FSL → Apache-2

Freemium

AGPL-3.0

Apache-2.0

AGPL-3.0

Commercial-APM agent footprints are order-of-magnitude estimates from public deployments; real overhead depends on instrumentation scope.

Acknowledging known findings

Drop a .perf-sentinel-acknowledgments.toml at your repo root to suppress findings the team has accepted: each gets a SHA-256 signature, and they are filtered from analyze / report / inspect / diff and don’t count toward the quality gate. Runtime acks against a live daemon are exposed via the ack CLI, the live HTML dashboard and the TUI.

Acknowledgment workflow →

Deployment

One binary, two modes, four environments, three deployment models. Pick the lightest that fits where your traces live. The central collector is a single stateful daemon, so horizontal replicas need trace-id-aware load balancing and do not share correlation state. Upstream trace sampling (head-based vs tail-based) and the daemon’s own sampling_rate undercount the repetition-based detectors, and under sustained overload the daemon sheds whole analysis batches rather than blocking ingestion, every shed counted in the metrics, never a silent drop.

Standalone

Local analysis, TUI and single-file HTML reports. Nothing deployed on the app side.

CI batch

perf-sentinel analyze --ci on captured traces, exit 1 over a threshold. The simplest gate, nothing long-running.

Sidecar daemon

One watch daemon per service, ingesting that service’s OTLP traces right beside it for isolated debugging.

Central collector

An OpenTelemetry Collector routes to a single long-running watch daemon, with Prometheus /metrics and the query API.

Local dev

CI/CD

Staging

Production

GreenOps (cross-cutting)

End-to-end

Live operator monitor over a running daemon, for DevOps / SRE, four Tab-cycled tabs (Advisor hints, Energy/carbon mix, Trends charts, Scrapers health) via perf-sentinel query --daemon <URL> monitor:

Full integration guide →

Privacy & security

perf-sentinel is built to sit inside your perimeter and stay quiet.

✓

On-prem processing

Traces are analyzed in place. No silent outbound calls, no usage telemetry baked in. Raw span content lives in memory only (30s TTL, 10k-trace LRU) and is never written to disk; everything emitted carries the normalized template only, with SQL literals and URL values replaced by placeholders.

✓

Localhost by default

The daemon listens on 127.0.0.1. TLS, CORS and the ack API key are all opt-in.

✓

Locked-down surface

Read-only GET endpoints and the OTLP ingestion listeners can be restricted; the no-auth threat model is documented. The endpoints trust their senders, so keep ingestion on a trusted network and put a reverse proxy or network policy in front before exposing anything beyond localhost.

Limitations & threat model →

Supply chain & releases

Every GitHub Action is pinned to a 40-character commit SHA, the production image is FROM scratch, Cargo.lock is committed and audited daily by cargo audit, and workflow token permissions default to read-only. Releases are signed with Sigstore and carry SLSA build provenance.

Releases follow a documented procedure gated by a mandatory simulation-lab run, with the Helm chart versioned in lockstep. Release binaries carry SLSA Build L3 provenance (Sigstore + Rekor) and embedded cargo-auditable data (cargo audit bin), and each release ships an SPDX SBOM attested under the SPDX predicate. Dependabot opens weekly grouped PRs.

Supply chain & provenance →