Configuration reference

perf-sentinel is configured via a .perf-sentinel.toml file. All fields are optional and have sensible defaults.

<img alt="CLI commands overview" src="https://raw.githubusercontent.com/robintra/perf-sentinel/main/docs/diagrams/svg/cli-commands.svg">

Contents

• Configuration fragments: deterministic multi-file loading.
• Subcommands: which subcommands read .perf-sentinel.toml.
• Sections: full per-section reference ([thresholds], [detection], [green], [daemon], [reporting]).
• Minimal configuration: the smallest useful .perf-sentinel.toml.
• Full configuration example: every section populated with example values.
• Migration from 0.5.x: the 8 legacy top-level keys removed in 0.6.0 and how to migrate.
• Environment variables: which env vars override config-file values.

Configuration fragments

perf-sentinel loads TOML documents from the directory .perf-sentinel.d/ beside the main config, then loads .perf-sentinel.toml last. This also applies when --config path/to/custom.toml is used: fragments come from path/to/.perf-sentinel.d/ and custom.toml remains the final override. The main file is optional only when --config is not supplied. Defaults are used only when neither the implicit main file nor any fragment exists. An unreadable file or an individual TOML parse error stops the command with exit code 75. After all overrides are applied, the merged configuration must pass typed deserialization and validation or the command also stops.

Fragment names must follow NN-lowercase-name.toml, where NN is a unique two-digit priority from 00 to 99. Files load in ascending priority order. Duplicate priorities, uppercase names and ambiguous separators are rejected. Non-TOML files in the directory are ignored.

When both values are tables, they merge recursively. Any other later value replaces the earlier value at the same key. The final merged document must still match the typed configuration schema. Since 0.12.0 that schema is strict: a key or a table name no section declares fails the load instead of being ignored, so a misspelled knob stops the command rather than silently leaving the default in place. The examples use these reserved bands:

The ready-to-copy fragments in examples/ preserve their priority in their filename. Keep those names when copying them into .perf-sentinel.d/: 30-green-alumet.toml, 31-green-cloud.toml, 32-green-scaphandre.toml, 33-green-kepler.toml, 34-green-redfish.toml and 40-green-electricity-maps.toml. 60-daemon-docker.toml is a standalone main config for the collector and sharded Compose topologies. Mount it as .perf-sentinel.toml, then place only the optional GreenOps fragments in the sibling .perf-sentinel.d/ directory.

Subcommands

Sections

[thresholds]

Quality gate thresholds. The quality gate fails if any rule is violated.

[detection]

Detection algorithm parameters.

sanitizer_aware_classification

OpenTelemetry agents and database drivers ship with SQL statement sanitization ON by default to keep PII out of trace attributes. The placeholder style depends on the stack: JDBC agents produce bare ?, PostgreSQL native drivers (pgx, asyncpg, sqlx) produce $1/$2 (normalized to $?), Python DB-API drivers produce %s, .NET drivers produce @p0/@Name, and Oracle/SQLAlchemy produce :name. In all cases the spans reach perf-sentinel with the same template and no extractable parameters, so the standard distinct-params rule rejects the group and the redundant detector picks it up as redundant_sql instead of n_plus_one_sql. This setting controls the heuristic that recovers the correct classification:

• "auto" (default): emit n_plus_one_sql when either the ORM scope signal (Spring Data, Hibernate, EF Core, SQLAlchemy, ActiveRecord, GORM, Prisma, Diesel, Laravel/Eloquent, Doctrine, ...) or the per-span timing variance is high enough to indicate distinct row lookups. Otherwise leave the group to the redundant detector. Best recall on production Spring Data, EF Core and similar ORM stacks.
• "strict": reclassify only when a primary signal (ORM scope marker, high occurrence >= 3 x n_plus_one_min_occurrences, or sequential siblings) fires conjointly with a corroborating signal (high timing variance or high occurrence). Preserves redundant_sql precision on moderate-count cached identical queries (legacy polling loops, unmemoized config lookups, typically 5-10 calls per request). Above the high-occurrence bar (default 15), any sanitized group fires regardless of ORM scope, sequential siblings, or variance, under the looks_sanitized guard. Use this when actionable redundant_sql findings are valuable signal that should not be silently absorbed into n_plus_one_sql. The simulation lab runs all of its stacks this way, because under auto an ORM scope marker alone reclassifies a cache-warmed repeat of the same query as an N+1. The change of verdict only applies to moderate counts, since the high-occurrence bar above fires under strict as well.
• "always": reclassify any sanitized group with at least n_plus_one_min_occurrences spans as n_plus_one_sql. Aggressive, may flip a real single-param redundancy.
• "never": disable the heuristic entirely and fall back to the strict distinct_params check.

Findings reclassified by the heuristic (whether under "auto", "strict", or "always") carry classification_method = "sanitizer_heuristic" in their JSON representation so operators can spot where it is firing. Findings produced by the standard rule omit the field.

sanitizer_aware_min_cv

The timing-variance signal behind the modes above compares the coefficient of variation of the group's per-span durations to this threshold. Row lookups against different keys spread their durations across cache hits and misses, repeats of one cached query cluster. The default of 0.5 favors reporting an N+1 over missing one, since a wrong call only swaps redundant_sql for n_plus_one_sql at the same avoidable-I/O weight.

Raise it on a runtime whose scheduling jitter spreads even cached repeats: PHP-FPM workers, CPU-throttled containers, shared CI runners. The simulation lab measured a CV of about 0.75 on ten identical Doctrine lookups served from cache under load, enough to cross the default and turn a redundant_sql finding into n_plus_one_sql with a remediation hint (leftJoin, with()) that does not apply to a repeat. At 1.0 the group keeps its redundant_sql verdict.

The same threshold feeds the HTTP heuristic, which decides whether a repeated outbound call with few distinct parameters reads as n_plus_one_http or redundant_http, so raising it moves both verdicts. What still reports a real N+1 whatever the variance depends on the mode and the path:

• SQL under "auto": the ORM scope marker alone, so raising the threshold changes nothing on an ORM-instrumented group.
• SQL under "strict": the high-occurrence bar (3 x n_plus_one_min_occurrences).
• HTTP: the direct rule, since distinct path or query parameters classify the group before the heuristic runs.
• "always" ignores the variance entirely, "never" never consults it.

One value serves the whole configuration: a daemon in front of several runtimes picks the noisiest one's threshold and accepts the loss on moderate-count groups elsewhere.

The value is recorded in detection_config of every report. A report written before the knob existed reads back as 0.5, the value its run hard-wired.

[green]

See also. The Energy and SCI primer in the methodology doc defines SCI v1.0 (E + I + M terms), RAPL, Scaphandre, SPECpower, Boavizta and the Electricity Maps API used by the config sections below. Read it once if any term feels unfamiliar.

GreenOps scoring configuration aligned with SCI v1.0 (operational + embodied terms, confidence intervals, multi-region).

[green.service_regions]

Per-service region overrides used when OTel cloud.region is absent from spans (e.g. Jaeger / Zipkin ingestion). Maps service name → region key.

[green]
default_region = "eu-west-3"
embodied_carbon_per_request_gco2 = 0.001

[green.service_regions]
"order-svc" = "us-east-1"
"chat-svc"  = "ap-southeast-1"

Region resolution chain

For each span, the carbon scoring stage resolves the effective region in this order (first match wins):

1. event.cloud_region: from the OTel cloud.region resource attribute (or span attribute as fallback). Most authoritative.
2. [green.service_regions][event.service]: per-service config override.
3. [green] default_region: global fallback.

I/O ops with no resolvable region land in a synthetic "unknown" bucket (zero operational CO₂; the row appears in regions[] for visibility). Embodied carbon is still emitted because hardware manufacturing emissions are region-independent. The region cardinality is capped at 256 distinct buckets; excess values fold into the unknown bucket to prevent memory exhaustion from misconfigured ingestion.

Output shape

When green scoring is enabled and at least one event is analyzed, the JSON report's green_summary includes:

• co2: structured { total, avoidable, operational_gco2, embodied_gco2 } object. Both total and avoidable are { low, mid, high, model, methodology } with 2× multiplicative uncertainty (low = mid/2, high = mid×2). The methodology tag distinguishes total ("sci_v1_numerator+transport": (E × I) + M + T summed over traces, including when T is zero) from avoidable ("sci_v1_operational_ratio": region-blind global ratio, excludes embodied and transport). Legacy reports can carry "sci_v1_numerator". model values, most precise wins: "electricity_maps_api" > "scaphandre_rapl" > "kepler_ebpf" > "redfish_bmc" > "cloud_specpower" > "io_proxy_v3" > "io_proxy_v2" > "io_proxy_v1". When calibration factors are active on proxy models, +cal is appended (e.g. "io_proxy_v2+cal"). The +cal suffix never applies to a measured tag.
• regions[]: per-region breakdown with { region, grid_intensity_gco2_kwh, pue, io_ops, co2_gco2, intensity_source }, sorted by co2_gco2 descending (highest-impact regions first) with alphabetical tiebreak. intensity_source is "annual", "hourly", "monthly_hourly" or "real_time" (Electricity Maps API) depending on which carbon intensity source was used for the region.

Carbon intensity data is embedded in the binary (no network egress). See 05 · GreenOps & carbon for the complete formula and methodology and Limitations for the directional / non-regulatory disclaimer.

User-supplied hourly profiles

Set [green] hourly_profiles_file to a JSON file to provide your own hourly profiles. This is useful for datacenter operators with their own power purchase agreements (PPAs) or for overriding the embedded data with local measurements.

{
  "profiles": {
    "my-datacenter": {
      "type": "flat_year",
      "hours": [45.0, 44.0, 43.0, "... 24 values total ..."]
    },
    "eu-west-3": {
      "type": "monthly",
      "months": [
        [50.0, 49.0, "... 24 values for January ..."],
        ["... 11 more months ..."]
      ]
    }
  }
}

User-supplied profiles take precedence over embedded profiles for the same region key. Validation at config load: each flat_year must have exactly 24 values, each monthly must have exactly 12 arrays of 24 values. All values must be finite and non-negative. If the region key exists in the embedded carbon table, a warning is logged when the profile mean deviates more than 5% from the annual value, but the profile is still accepted.

Hourly profile region aliases

Country-code aliases and cloud-provider synonyms are resolved to the same hourly profile. For example, "fr", "francecentral" and "europe-west9" all map to the eu-west-3 (France) profile. Notable mappings:

• "us", "eastus" -> us-east-1 (US-East, the most common US deployment region)
• "westeurope", "nl", "nl-ams" -> eu-west-4 (Netherlands)
• "northeurope", "ie" -> eu-west-1 (Ireland)
• "uksouth", "gb", "uk", "uk1" -> eu-west-2 (UK)
• "westus2" -> us-west-2 (Oregon)
• "gra11", "gra", "sbg", "fr-par", "outscale-eu-west-2" -> eu-west-3 (France)
• "waw1", "pl-waw" -> europe-central2 (Poland)
• "bhs5", "bhs" -> ca-central-1 (Quebec)

OVHcloud, Scaleway and OUTSCALE keys. OVHcloud names one datacenter three ways depending on which API you ask (GRA11 for the OpenStack Public Cloud region, GRA for the zone code, gra for the S3 location string), and all three are keyed. OUTSCALE keys carry an outscale- prefix because OUTSCALE reuses AWS region identifiers for different places: its eu-west-2 is Paris where the AWS one is London. An OUTSCALE deployment declares default_region = "outscale-eu-west-2" rather than the bare identifier, otherwise it scores against the British grid.

The full alias table is in score/carbon_profiles.rs. If your region key is not aliased, the flat annual value from the primary carbon table is used.

Every energy and grid-intensity backend is daemon-only. [green.alumet], [green.scaphandre], [green.kepler], [green.redfish], [green.cloud], [green.broker_static] and [green.electricity_maps] are scraped by the watch daemon and by nothing else. A batch analyze or report run starts no scraper, so it scores with the I/O proxy estimate over embedded intensity data whatever these sections say, and it emits a warning naming the sections it had to ignore. Attributing power measured now to traces recorded earlier would produce a wrong figure rather than a missing one, which is why batch does not scrape.

[green.scaphandre] (optional, opt-in)

Opt-in integration with Scaphandre for per-process energy measurement on Linux hosts with Intel RAPL support. When configured, the watch daemon spawns a background task that scrapes the Scaphandre Prometheus endpoint every scrape_interval_secs and uses the measured power readings to replace the fixed ENERGY_PER_IO_OP_KWH constant for each mapped service.

Prefer [green.alumet] for new deployments. Both integrations read the same RAPL counters, but Alumet's sampling is measurably less error-prone, as characterized by its own authors in Dissecting the software-based measurement of CPU energy consumption (Raffin et al.), and it attributes per cgroup rather than per process. Scaphandre support is kept for existing deployments, and alumet_rapl outranks scaphandre_rapl whenever both feed the same service. See Limitations.

Each process_map entry is a table with two fields: exe_contains (required, substring matched against the Scaphandre exe label) and cmdline_contains (optional, substring matched against the cmdline label). The matcher requires both substrings to be present when cmdline_contains is set. Exactly one Scaphandre process must match per entry, otherwise the scoring stage skips that service for the tick and emits a warn log naming the ambiguity.

[green.scaphandre]
endpoint = "http://localhost:8080/metrics"
scrape_interval_secs = 5

[green.scaphandre.process_map."order-svc"]
exe_contains = "bin/java"
cmdline_contains = "order-svc.jar"

[green.scaphandre.process_map."chat-svc"]
exe_contains = "bin/java"
cmdline_contains = "chat-svc.jar"

[green.scaphandre.process_map."native-svc"]
exe_contains = "/opt/native-svc/bin/native-svc"

Why both exe_contains and cmdline_contains. Scaphandre emits exe as an absolute path of the runtime (/usr/lib/jvm/.../bin/java, /usr/share/dotnet/dotnet). Several co-located services sharing a runtime (multiple JVMs, multiple .NET assemblies) collide on exe, and only cmdline discriminates them. Real Scaphandre also concatenates argv without separators: java -jar /tmp/order-svc.jar is emitted as cmdline="java-jar/tmp/order-svc.jar". Configure cmdline_contains with a substring that appears in this concatenated form (e.g. the jar/dll filename), NOT with a POSIX command line containing spaces.

Ignored in analyze batch mode. Only the watch daemon spawns the scraper. The analyze command always uses the proxy model regardless of this section.

Fallback behaviour. When the endpoint is unreachable, a service is not present in process_map or a service had zero ops in the current scrape window, the scoring stage falls back to the proxy model for those spans. The first failure logs at warn level; subsequent failures log at debug to avoid spam. The perf_sentinel_scaphandre_last_scrape_age_seconds Prometheus gauge lets operators detect a hung scraper.

Precision bounds (important). Scaphandre improves the per-service energy coefficient but does NOT give per-finding attribution. RAPL is process-level, not span-level: two findings in the same process during the same scrape window share the same coefficient. See Limitations for the full discussion.

[green.kepler] (optional, opt-in)

Opt-in integration with Kepler (CNCF sandbox) for per-container or per-process energy measurement via eBPF. Unlike Scaphandre, Kepler works on ARM64 (Graviton, Ampere, Apple Silicon, Cobalt 100) with degraded precision but a real signal. When configured, the watch daemon scrapes Kepler's Prometheus /metrics endpoint, computes a per-service joules delta vs the previous scrape, and publishes a measured per-op coefficient tagged kepler_ebpf.

[green.kepler]
endpoint = "http://kepler.kube-system.svc.cluster.local:9102/metrics"
scrape_interval_secs = 5
metric_kind = "container"

[green.kepler.service_mappings]
"order-svc" = "order-svc-deployment"
"chat-svc" = "chat"

Ignored in analyze batch mode. Like Scaphandre, only watch spawns the scraper.

Counters sharing a label value are summed. One container name repeated across pods (or one comm shared by several processes) yields several cumulative series under one mapping value. Their counters are summed before the per-window delta is computed, so the coefficient covers all of them together.

Precedence vs Scaphandre. Scaphandre RAPL outranks Kepler eBPF on x86_64 with RAPL access. The Kepler integration shines on ARM64 where Scaphandre is unavailable. See Limitations for the ARM eBPF accuracy caveats (Kepler upstream issue #1556).

Production deployment shape. Kepler typically runs as a Kubernetes DaemonSet, one pod per node. The current scraper performs a direct GET and the response must expose the Kepler series themselves. A Prometheus server's own /metrics endpoint exposes Prometheus internals, not the series it scraped. For a multi-node cluster, run one perf-sentinel per node or provide a federation/proxy endpoint that directly exposes the aggregated Kepler series in Prometheus exposition format. Native PromQL query mode is reserved for a follow-up release.

[green.alumet] (optional, opt-in)

Opt-in integration with Alumet (INRIA/LIG, EUPL-1.2) for measured energy. Alumet is a modular measurement framework: a source plugin (rapl, nvidia-nvml, ...) produces readings, optional transform plugins attribute them to workloads, and an output plugin exposes them. perf-sentinel scrapes the prometheus-exporter output. When configured, the watch daemon publishes a measured per-op coefficient tagged alumet_rapl, which outranks every other measured source, including Scaphandre.

Why metric_name and label_key have no default. Alumet's exporter prepends prefix and appends suffix (default _alumet) to every metric name, and the per-service series is produced by an energy-attribution formula whose name you choose. No default could be right for every deployment, and a wrong guess would scrape nothing. Read the names off your own endpoint:

curl -s http://localhost:9091/metrics | grep -i energy

Several rows per service are summed. Alumet's label_key is routinely shared: one pod carries a row per RAPL domain (package + dram), and label_key = "domain" on a dual-socket host carries one domain="package" row per socket. Every row sharing a label value is summed, which is the physically correct read since energy is additive (NaN and negative rows are skipped). Two consequences to configure for. Pick label_key so that the rows sharing a value are the ones you want added together. And make sure those rows do not overlap: RAPL domains nest (psys contains package, package contains pp0/pp1), so a formula that emits both a parent and its child domain for one label double-counts the shared share. package plus dram sums correctly, psys plus package does not.

Why energy_interval_secs exists. This is the one field to get right. Alumet's exporter publishes every measurement as a Prometheus gauge holding the last flushed value, and rapl_consumed_energy is a CounterDiff: the joules burned during one source poll_interval, not a cumulative counter and not a power reading. perf-sentinel divides by this interval to recover watts. The interval appears nowhere on the wire, so it must be declared here, and it must match the Alumet side. A mismatch rescales energy and carbon linearly and silently: declaring 1.0 while Alumet polls at 5s overstates energy 5x, with no warning. See Limitations. The daemon echoes the value it is using in the Alumet scraper started log line.

Matching Alumet config. Per-service attribution needs three Alumet plugins working together, rapl alone only measures the whole machine and procfs only identifies processes by PID:

# alumet-config.toml
[plugins.rapl]
poll_interval = "1s"          # <- perf-sentinel's energy_interval_secs must equal this

[plugins.k8s]
# pod discovery, provides the `name` / `namespace` attributes

[plugins.energy-attribution.formulas.attributed_energy_cpu]
expr = "cpu_energy * cpu_usage / 100.0"
ref = "cpu_energy"

[plugins.energy-attribution.formulas.attributed_energy_cpu.per_resource]
cpu_energy = { metric = "rapl_consumed_energy", resource_kind = "local_machine", domain = "package_total" }

[plugins.energy-attribution.formulas.attributed_energy_cpu.per_consumer]
cpu_usage = { metric = "cpu_percent", kind = "total" }

[plugins.prometheus-exporter]
port = 9091
suffix = "_alumet"            # <- why the metric name below ends in _alumet

The matching perf-sentinel side:

[green.alumet]
endpoint = "http://localhost:9091/metrics"
scrape_interval_secs = 5
metric_name = "attributed_energy_cpu_alumet"
label_key = "name"
energy_interval_secs = 1.0

[green.alumet.service_mappings]
"order-svc" = "order-svc-pod"
"chat-svc" = "chat-svc-pod"

Ignored in analyze batch mode. Like every measured-energy backend, only watch spawns the scraper.

Precedence. alumet_rapl leads the measured chain, ahead of scaphandre_rapl. Both read RAPL, but Alumet's sampling is measurably less error-prone and it attributes per cgroup rather than per process. Running both on the same service is supported, Alumet wins.

Upstream packaging gotchas. The upstream .deb ships /etc/alumet/alumet-config.toml (with sections for csv, procfs, perf, ...) and its alumet-agent wrapper points ALUMET_CONFIG at it unless the variable is already set. Enabling prometheus-exporter with --plugins works even though that file has no prometheus-exporter section. The agent fills the absent section from the plugin's defaults (prefix "", suffix "_alumet", port 9091). For a controlled config instead of the shipped one, point ALUMET_CONFIG at a fresh path so the agent regenerates defaults for your plugin set, or run config regen. In containers, the packaged binary carries file capabilities (cap_perfmon, cap_sys_nice, cap_sys_ptrace), so a plain docker run fails with EPERM unless those are granted with --cap-add.

The scraper alone does not put alumet_rapl in the report. Setting [green.alumet] starts the scraper (visible on /api/energy), but the measured coefficient only reaches green_summary when green scoring resolves a region for the spans ([green] default_region, [green.service_regions], or a cloud.region span attribute). Without one, per_service_energy_model keeps showing the proxy tag, which is easy to misread as a broken Alumet integration.

Alumet is pre-1.0 (v0.9.5 at time of writing). Metric names and plugin config may change between releases. If a scrape stops matching after an Alumet upgrade, the daemon warns with no samples matched the configured metric after three consecutive ticks.

[green.alumet.database] (optional)

Declares one database workload measured by Alumet. A database emits no spans, so it can never appear in service_mappings (zero ops, the per-op coefficient path skips it). Instead, its energy over each scoring window is multiplied by the SQL-only waste ratio (avoidable_sql_io_ops / total_sql_io_ops) and reported as green_summary.database_waste, a standalone figure excluded from energy_kwh, co2 and the public disclosure. See Methodology for the formula and Limitations for why it is a lower bound.

[green.alumet.database]
label_value = "postgres-pod"   # value carried by label_key for the DB cgroup, verbatim
region = "eu-west-3"           # optional, enables the gCO2 conversion (declared, not inferred)

label_value is required, matched exactly like a service_mappings value. region is optional and uses the same region ids as [green.service_regions]: without it the waste is reported in kWh only. One database per config, declare the cgroup that serves your SQL traffic.

[green.alumet.broker] (optional)

The messaging twin of the section above. Declares one message broker measured by Alumet: a broker emits no spans of its own either, so its window energy is multiplied by the messaging-only waste ratio (avoidable_messaging_io_ops / total_messaging_io_ops) and reported as green_summary.messaging_waste.

[green.alumet.broker]
label_value = "kafka-pod"      # value carried by label_key for the broker cgroup, verbatim
region = "eu-west-3"           # optional, enables the gCO2 conversion

The same cgroup cannot feed two figures: a label_value that also appears in service_mappings, or that matches the database declaration, is rejected at config load. Requires an agent on the broker host, so it does not apply to a managed broker. For those, see [green.broker_static].

[green.broker_static] (optional, opt-in)

Declares a provisioned broker cluster, with no agent and no metric. This is the only path that works for a managed broker (Confluent Cloud, MSK, SQS, managed Pulsar), where there is no host to instrument.

[green.broker_static]
nodes = 3                      # provisioned broker nodes, required
instance_type = "m5.2xlarge"   # looked up in the embedded SPECpower table, required
provider = "aws"               # optional: aws, gcp, azure, scaleway or generic (default)
region = "eu-west-3"           # optional, enables the gCO2 conversion

The energy is nodes × max_watts × window duration, following E(n) = n × P_max: provisioned nodes times their power ceiling. Three properties to accept before relying on it:

• It bounds compute, not wall power. max_watts is the power at 100 % CPU, drawn from a SPECpower table that covers CPU and baseboard and excludes storage, network and PSU overhead. Those dominate on a broker, so this is a ceiling on the declared vCPUs and not on what the cluster actually pulls from the wall. A storage-bound Kafka node can draw more than this figure reports.
• It counts provisioned infrastructure, not consumed. A three-node cluster is immobilized whether it runs at 10 % or 60 %. In the other direction, a stretch with no traffic bills at most one hour, so a mostly-idle cluster is under-counted.
• It does not react to application changes. Batching your publishes will not move this number, because nothing about it depends on traffic. If you want a figure that responds to remediation, you need the measured path.

An unknown instance_type warns and falls back to a provider default rather than failing: the figure stays coarser, and the warning says so. An unrecognised provider is rejected outright, because it would silently resolve to the generic on-prem watts. Daemon only, like every measured path, since a window duration is needed. When both this section and [green.alumet.broker] are configured, the measurement wins. Precedence follows the broker's own Alumet series, not the scrape endpoint: a scrape that answers without carrying the declared label_value measures nothing, so the declaration takes over rather than being suppressed by a working endpoint. Energy the series already banked is delivered first, and the delta that lands when it recovers is dropped once, since it reaches back over wall clock the declaration billed.

[green.redfish] (optional, opt-in)

Opt-in integration with the Redfish BMC standard for bare-metal wall-plug power readings. Unlike Scaphandre and Kepler (which measure CPU + DRAM only), Redfish reads the actual power supply output via the BMC, so periphery (NIC, drives, fans, PSU overhead) is included. Bare-metal only, no cloud VMs.

Each endpoint table has two fields: url (string, full Redfish URL including path) and schema (string, either "legacy_power" or "environment_metrics"). The schema selects the canonical JSON pointer the parser uses, no operator-typed pointer involved:

[green.redfish]
scrape_interval_secs = 60

[green.redfish.endpoints."chassis-legacy-1"]
url = "https://bmc-rack-01.dc.example/redfish/v1/Chassis/1/Power"
schema = "legacy_power"

[green.redfish.endpoints."chassis-modern-1"]
url = "https://bmc-rack-02.dc.example/redfish/v1/Chassis/1/EnvironmentMetrics"
schema = "environment_metrics"

[green.redfish.service_mappings]
"order-svc"  = "chassis-legacy-1"
"chat-svc"   = "chassis-legacy-1"
"ledger-svc" = "chassis-modern-1"

Which schema to choose. /Power (legacy_power) was deprecated by DMTF Release 2020.4 but is still mandatory on BMC firmware as of 2026, every shipping vendor exposes it. /EnvironmentMetrics (environment_metrics) is the modern replacement that carries PowerWatts.Reading directly, present alongside /Power during the transition. Pick legacy_power unless your BMC documentation explicitly recommends EnvironmentMetrics. A mixed fleet is declared by giving each chassis the schema its firmware serves.

Ignored in analyze batch mode. Like Scaphandre and Kepler, only watch integrates Redfish.

Node-level coefficient. Every service mapped to the same chassis receives the same coefficient. Two services on one chassis will never get distinct measured per-op values via Redfish. See Limitations for the full discussion of this trade-off and the vendor-specific JSON response variance.

[green.cloud] (optional, opt-in)

Cloud-native energy estimation via CPU utilization + SPECpower interpolation. When configured, the watch daemon scrapes CPU% from a Prometheus/VictoriaMetrics endpoint and uses an embedded lookup table (idle/max watts per cloud instance type) to estimate per-service energy consumption. Supports AWS, GCP, Azure and on-premise hardware with manual watts override.

Per-service entries in [green.cloud.services] support two forms:

Cloud instance (table lookup):

[green.cloud]
prometheus_endpoint = "http://prometheus:9090"
scrape_interval_secs = 15
default_provider = "aws"

[green.cloud.services]
"account-svc" = { provider = "aws", instance_type = "m7i.4xlarge" }       # Sapphire Rapids
"api-asia" = { provider = "gcp", instance_type = "c4d-standard-8" }       # AMD Turin
"analytics" = { provider = "azure", instance_type = "Standard_D8s_v6" }   # Emerald Rapids
"ml-bench" = { provider = "aws", instance_type = "m8g.4xlarge" }          # Graviton 4

The full list of covered types, with their idle and max wattage, is Instance types. Modern instance families covered include AWS m7i/c7i/r7i, m7a/c7a, m6a/c6a, m7g/c7g, m8g/c8g; GCP c3, c3d, c4, c4d, n2d, t2a; Azure Standard_Dv6, Standard_Dadsv6, Standard_Dpsv6 (Cobalt 100), Standard_Ev6. One CPU-named bare-metal entry covers Sierra Forest (xeon-6780e, system-level watts assuming full chip ownership).

Manual watts (on-premise or custom hardware):

[green.cloud.services]
"my-service" = { idle_watts = 45, max_watts = 120 }

Ignored in analyze batch mode. Only the watch daemon spawns the Prometheus scraper.

Fallback behaviour. If the Prometheus endpoint is unreachable, the daemon falls back to the proxy model for all cloud-configured services. Unknown instance types fall back to a provider-level default.

Precision bounds. The SPECpower interpolation model has approximately +/-30% accuracy, better than the proxy model but less precise than Scaphandre RAPL. See Limitations for details.

[green.electricity_maps] (optional, opt-in)

Real-time carbon intensity from the Electricity Maps API. Daemon-only.

The region_map sub-table maps cloud regions to Electricity Maps zone codes:

[green.electricity_maps]
# Use PERF_SENTINEL_EMAPS_TOKEN env var instead of api_key in config
poll_interval_secs = 300

[green.electricity_maps.region_map]
"eu-west-3" = "FR"
"us-east-1" = "US-NY"
"ap-northeast-1" = "JP-TK"

Staleness: if the last successful poll is older than 3x poll_interval_secs, the scraper falls back to embedded hourly profiles.

Rate limits: the Electricity Maps free tier allows approximately 30 requests per month per zone. For free tier users, set poll_interval_secs = 3600 or higher. The default of 300s is intended for paid plans.

API version: the default endpoint targets v4 since perf-sentinel 0.5.11. v3 remains accepted (the response schema is identical on carbon-intensity/latest), but a deprecation warning is logged once at daemon startup. To silence the warning, set endpoint = "https://api.electricitymaps.com/v4" explicitly. To keep v3 deliberately (for example to A/B-validate against v4), leave endpoint = "https://api.electricitymaps.com/v3" and acknowledge the warning.

Unknown values for emission_factor_type and temporal_granularity: these two knobs use a fail-graceful parser. A typo or unsupported value (e.g. temporal_granularity = "5min" instead of "5_minutes") does not reject the config at load time. The value is sanitized, a tracing::warn! is emitted, and the daemon falls back to the default. Watch the daemon logs at startup if you suspect a typo, the warn line will name the offending field and value.

Visibility in reports (since perf-sentinel 0.5.12): the active scoring configuration (API version, emission factor type, temporal granularity) is surfaced in three places so Scope 2 reporters can audit which carbon model produced the numbers without reading the operator's TOML.

• The JSON report always carries green_summary.scoring_config while GreenOps scoring is enabled: it records the applied coefficients and an electricity_maps flag. The API-specific fields are meaningful only when that flag is true.
• The HTML dashboard renders a chip bandeau above the green-regions table. Default values (v4, lifecycle, hourly) are neutral chips, opt-in values (direct, 5_minutes, 15_minutes) are accent chips, the legacy v3 endpoint shows as a warning chip mirroring the deprecation warning. Native browser tooltips explain each value.
• The terminal print_green_summary output prepends a one-liner Carbon scoring: Electricity Maps v4, lifecycle, hourly before the per-region breakdown.

The bandeau and the terminal line are hidden when [green.electricity_maps] is not configured.

[green] calibration_file (optional)

Path to a calibration TOML file generated by perf-sentinel calibrate. When present, per-service calibration factors are loaded at config time and multiply the proxy model energy per op. Does not affect Scaphandre or cloud SPECpower measured energy.

[green]
calibration_file = ".perf-sentinel-calibration.toml"

perf-sentinel calibrate input size limits. Both inputs are capped to protect against unbounded memory use: the --traces file is capped at 1 GiB (the fixed batch cap since 0.8.7, same as analyze) and the --measured-energy CSV is capped at 64 MiB. Calibrate exits with a clear error if either file exceeds its limit. 64 MiB is generous for thousands of RAPL samples per minute, if you need more, file an issue describing the workload.

perf-sentinel tempo (no config section)

The tempo subcommand runs in batch mode (not daemon), fetches traces from a Grafana Tempo HTTP API and pipes them through the standard analysis pipeline. Its own settings are CLI flags only, there is no [tempo] section: --endpoint is required, --max-traces defaults to 100 and is bounded to 1..=10000 (the client's own read ceiling, not Tempo's), alongside --trace-id, --service, --lookback, --from/--to, --sort and --auth-header. Run perf-sentinel tempo --help for the current list. A [tempo] table written into the config file fails the load since 0.12.0, as any unknown top-level table does. The --config file still applies for everything else, thresholds and detection in particular, since the fetched traces go through the same pipeline.

[daemon]

Streaming mode (perf-sentinel watch) settings.

Comfort zones and startup warnings

Daemon limits accept any value inside their hard bounds (rejected at config load), but perf-sentinel watch emits a one-shot WARN log at startup when a value falls outside the recommended comfort zone. The warning is informational: the daemon still runs. Use it as a sanity check that an unusual value was deliberate.

Comfort zones judge the static value at startup. At runtime the daemon complements them with a settings advisor: when lifetime counters show a knob undersized for the observed load (queue sheds, ingest rejects, near-full trace window...), /api/export/report emits tuning entries in Report.warning_details naming the knob, its current value, and the suggested adjustment. See Metrics section "Warning kinds: transient vs sticky" for the rule table.

[daemon.correlation] (optional)

Cross-trace temporal correlation in daemon mode. When enabled, the daemon detects recurring co-occurrences between findings from different services or traces (e.g. "every time the N+1 in order-svc fires, pool saturation appears in payment-svc within 2 seconds").

[daemon.correlation]
enabled = true
window_minutes = 10
lag_threshold_ms = 5000
min_co_occurrences = 5
min_confidence = 0.7

A 24 h window suits patterns that recur over a day. It costs the same pairing memory as the default; only the counters span longer:

[daemon.correlation]
enabled = true
window_minutes = 1440
lag_threshold_ms = 2000
min_co_occurrences = 10
min_confidence = 0.7
max_tracked_pairs = 50000

Correlations are exposed via GET /api/correlations (when api_enabled = true) and snapshotted under correlations in GET /api/export/report. The daemon's stdout stream never carries them.

[daemon.ack] (optional, since 0.5.20)

Daemon-side runtime ack store. Complements the CI TOML acks (see Acknowledgments) with a JSONL append-only file mutated through the HTTP API endpoints POST / DELETE /api/findings/{signature}/ack.

[daemon.ack]
enabled = true
storage_path = "/var/lib/perf-sentinel/acks.jsonl"
# api_key = "<rotate-this>"
toml_path = "/etc/perf-sentinel/acknowledgments.toml"

The JSONL file is replayed and atomically rewritten (via tmp + rename) at every daemon restart, so repeated ack / unack cycles cannot accumulate beyond their net active state. On Unix, the file is created with mode 0600 (owner read-write only).

[daemon.hub_export] (optional)

Bounded, asynchronous export of live findings to PerfSentinelHub. The daemon coalesces repeated findings by signature and keeps only their latest value, so a hot issue does not create one network request per detection.

[daemon.hub_export]
enabled = true
endpoint = "https://hub.example.com/api/import/findings"
source_id = "production-a"
api_key_file = "/run/secrets/perf-sentinel-hub-api-key"
batch_size = 100
flush_interval_secs = 5
max_pending = 10000

The pending structure is a latest-value table, not an unbounded queue. A signature is sent immediately when first discovered or when its severity worsens, then refreshed at most hourly while it recurs. Both the pending table and the recent-success cache are capped at max_pending; evicting from the recent-success cache is not a loss and is not counted, while a pending-table eviction, an oversized finding and a batch the Hub rejects with an unretryable 4xx all increment perf_sentinel_hub_export_dropped_total. Failed requests retain their coalesced batch and retry with exponential backoff and jitter; a daemon restart clears both caches. Requests and JSON bodies are bounded to 100 findings and 2 MiB. Use HTTPS outside a trusted private network.

On a graceful shutdown (SIGTERM, helm upgrade, a rolling restart) the exporter flushes what it still holds before the process exits, for up to 10 seconds. That budget is deliberate: an unreachable Hub must not hold the daemon past the orchestrator's grace period, where the next signal is SIGKILL and nothing is flushed at all. When the budget expires, the findings still pending are dropped and a WARN names how many. Size terminationGracePeriodSeconds above that budget so the drain has room to run, otherwise the pod is killed mid-flush.

Mount the API key as a Secret-backed file. Do not put it in a ConfigMap or in .perf-sentinel.toml. With the Helm chart, use extraVolumes and extraVolumeMounts to expose the key at the configured api_key_file path.

[daemon.incidents] (optional, since 0.20.0)

Inbound incident webhooks, so an alert can hand the daemon the moment a service crashed or saturated and the daemon can freeze the findings of the window before the ring evicts them. See Query API for the endpoints. Daemon-only, off by default, and not to be confused with [daemon] memory_high_water_pct, which is the daemon's own cgroup.

[daemon.incidents]
enabled = true
# api_key = "<rotate-this>"      # or PERF_SENTINEL_INCIDENTS_API_KEY
lookback_ms = 300000

The window a posted incident freezes is [at_ms - lookback_ms, at_ms + 2 * trace_ttl_ms], resolved with both bounds, so seen_count and first_seen_ms on the frozen findings describe the window rather than the whole retained history. See Query API for why the window closes after the incident and for the settle pass that fills its tail.

[daemon.cors] (optional, since 0.5.23)

Cross-origin resource sharing for the daemon's /api/* query endpoints. Disabled by default (no Access-Control-Allow-Origin header is emitted, the loopback-only posture is preserved). Enable when a browser client needs to call the daemon, typically the HTML report in live mode (perf-sentinel report --daemon-url <URL>, see HTML report).

Scope: the CORS layer is wired only on the /api/* query API sub-router. The OTLP ingest path (/v1/traces), Prometheus exposition (/metrics), and liveness probe (/health) are NOT exposed cross-origin even under wildcard mode. Browser pages cannot post traces, scrape /metrics, or hit /health regardless of allowed_origins. This containment is intentional, browser clients have no legitimate use for those surfaces.

Read-endpoint exposure: every /api/* GET endpoint (/api/findings, /api/acks, /api/status, /api/correlations, /api/explain/*, /api/export/report) is unauthenticated by design, in line with the loopback-only posture pre-0.5.23. Once you whitelist an origin, any browser tab on that origin can read every finding signature, ack metadata, and trace export the daemon holds. Only whitelist origins you trust to view all daemon-resident data. Mixing untrusted origins with wildcard mode (["*", "https://x"]) is rejected at config load.

Wildcard example (development):

[daemon.cors]
allowed_origins = ["*"]

Production example (whitelist):

[daemon.cors]
allowed_origins = [
    "https://reports.example.com",
    "https://gitlab.example.com",
]

Methods allowed: GET, POST, DELETE, OPTIONS. Headers allowed: Content-Type, X-API-Key. (X-User-Id is not advertised because the daemon does not enforce it server-side; the by field on an ack POST body is operator-attested only.) Preflight Access-Control-Max-Age: 120 seconds. Long enough to amortize the OPTIONS roundtrip across a typical interaction, short enough that a tightened whitelist takes effect on the next browser preflight without a forced refresh.

The CORS layer does not set Access-Control-Allow-Credentials: true, which is incompatible with ["*"] and unnecessary because the daemon auths via the X-API-Key header rather than cookies. Browsers running on a non-whitelisted origin receive responses without the Access-Control-Allow-Origin header and the request is blocked client-side without a daemon-side rejection.

Origins that fail to parse as a valid HTTP header value (typically a copy-paste with embedded control characters) are dropped at startup with a warn! log and the rest of the list is honored. If every entry is invalid, the layer is disabled entirely. If daemon_api_enabled = false, the CORS layer is skipped (the /api/* sub-router is not mounted in the first place) and a warn! notes the unused config.

Since 0.5.27, combining allowed_origins = ["*"] with [daemon.ack] api_key also emits a startup warn!. Wildcard CORS plus an X-API-Key auth lets any browser origin replay a captured key through the daemon, even though no cookie or Allow-Credentials mode is in play. Whitelist explicit origins for production deployments where the API key is set.

[reporting]

Public-disclosure settings consumed by disclose, hash-bake and verify-hash. The whole section is optional. An absent section means the operator never asked for a periodic disclosure. Full walkthrough in Reporting, field reference in Schema.

disclose reads none of these: it takes --intent, --confidentiality, --period-type, --org-config and --output from the command line. What this section still does is gate the daemon at startup through intent and org_config_path.

The [reporting.sigstore] sub-section holds the Sigstore endpoints, Rekor being the transparency log and Fulcio the certificate authority. Both are reserved: they are parsed and then unused. verify-hash delegates signature checking to the cosign binary and invokes cosign verify-blob without --rekor-url or --fulcio-url, so cosign follows its own configuration and setting either key here has no effect. Point a private Sigstore instance at cosign itself until this is wired up.

Minimal configuration

An empty file or no file at all uses all defaults. A minimal configuration for CI might only set thresholds:

[thresholds]
n_plus_one_sql_critical_max = 0
io_waste_ratio_max = 0.25

Full configuration example

[thresholds]
n_plus_one_sql_critical_max = 0
n_plus_one_http_warning_max = 3
n_plus_one_messaging_warning_max = 3
io_waste_ratio_max = 0.30
# min_usable_span_ratio = 0.9   # unset = disabled, see the thresholds table

[detection]
n_plus_one_min_occurrences = 5
window_duration_ms = 500
slow_query_threshold_ms = 500
slow_query_min_occurrences = 3
slow_query_window_minutes = 15
max_fanout = 20
chatty_service_min_calls = 15
pool_saturation_concurrent_threshold = 10
serialized_min_sequential = 3
# Recovery heuristic for already-parameterized SQL: "auto", "strict",
# "always", "never". The variance bar below is what separates a real N+1
# from a cached repeat. Raise it on a jittery runtime such as PHP-FPM,
# where repeats of one cached query spread past 0.5 and read as N+1.
sanitizer_aware_classification = "auto"
sanitizer_aware_min_cv = 0.5

[green]
enabled = true
default_region = "eu-west-3"

[daemon]
listen_address = "127.0.0.1"
listen_port_http = 4318
listen_port_grpc = 4317
json_socket = "/tmp/perf-sentinel.sock"
max_active_traces = 10000
trace_ttl_ms = 30000
sampling_rate = 1.0
max_events_per_trace = 1000
max_payload_size = 16777216
# Optional: enable TLS on both gRPC and HTTP listeners.
# Both fields must be set together (or both absent for plain TCP).
# tls_cert_path = "/etc/tls/server-cert.pem"
# tls_key_path = "/etc/tls/server-key.pem"
api_enabled = true
max_retained_findings = 10000
max_retained_traces = 50
# Optional: tune the bounded queues (defaults shown). Raise under bursty
# load to reduce ingestion backpressure / analysis shedding.
ingest_queue_capacity = 1024
analysis_queue_capacity = 1024
# Optional: `service` label on findings and slow-span histograms (bounded
# by the daemon's cardinality caps). false restores the pre-0.18 shape.
per_service_labels = true
# Optional: `grouping` label next to `service` on the same series plus the
# per-service I/O counters (capped on admitted (service, grouping) pairs).
# false restores the 0.18 shape.
per_grouping_labels = true

# Optional: reject OTLP ingest when cgroup memory crosses this percent of
# the container limit, bounding RSS against OOM. 0 disables (default).
# Linux/cgroup-v2 only. Set 80-85 to leave headroom above steady state.
memory_high_water_pct = 0

# Optional: cross-trace correlation (daemon mode only)
# [daemon.correlation]
# enabled = true
# window_minutes = 10
# lag_threshold_ms = 5000

Migration from 0.5.x

Eight legacy top-level keys were deprecated in 0.5.26 and removed in 0.6.0. A 0.5.x config that still uses any of them now fails at load time with a migration message rather than silently falling back to the default. Update to the sectioned form below before upgrading.

Migration example. Before (0.5.x):

n_plus_one_threshold = 5
listen_port = 4318
max_payload_size = 2097152

After (0.6.0+):

[detection]
n_plus_one_min_occurrences = 5

[daemon]
listen_port_http = 4318
max_payload_size = 2097152

Loading a 0.5.x file on 0.6.0 returns a ConfigError::Validation whose message names both the removed key and its replacement, so a single tail of the error stream tells you exactly what to edit.

Environment variables

Configuration files must never contain secrets. For sensitive values (API keys, tokens), use environment variables in your deployment tooling. perf-sentinel reads a fixed list of them, each one overriding the matching config field when set and trimmed of a trailing newline, so a Kubernetes Secret can feed it directly:

The overrides apply on every run, batch commands and a run without any config file included, so a job that inherits the daemon's Secret must carry valid values too (the incidents key only counts once [daemon.incidents] enabled is set). A variable set to an empty string counts as set: an empty key is rejected at config load rather than silently ignored, so a Secret mounted empty fails every command that loads it, the daemon included, instead of opening the route.

Acknowledgments file

.perf-sentinel-acknowledgments.toml is a separate file from .perf-sentinel.toml. It lives at the root of the application repo and lists findings the team has accepted as known. Acknowledged findings are filtered from the CLI output (analyze, report, inspect, diff) and excluded from the quality gate.

Loading rules:

• The default path is ./.perf-sentinel-acknowledgments.toml in the current working directory. Override with --acknowledgments <path>.
• If the file does not exist, the run is a no-op (no error, no output noise).
• --no-acknowledgments skips the file entirely (audit view).
• A typo in signature, a missing required field, or a malformed expires_at fails the run loud rather than silently widening the matched set.

Minimal entry:

[[acknowledged]]
signature = "redundant_sql:order-service:POST__api_orders:cafebabecafebabecafebabecafebabe"
acknowledged_by = "alice@example.com"
acknowledged_at = "2026-05-02"
reason = "Cache invalidation pattern, intentional. See ADR-0042."

The expires_at = "YYYY-MM-DD" field is optional. Omitting it makes the ack permanent. Setting it lets you require a periodic re-evaluation: when the date passes, the ack stops applying and the finding reappears in the next CI run.

There is no glob or wildcard support, each entry is matched against an exact signature. Signatures are emitted on every finding in the JSON output, copy-paste them into the file rather than recomputing the SHA-256 prefix by hand.

For the full workflow and FAQ, see Acknowledgments.