Daemon query API

Deployment notes

• The query API shares the same HTTP port as OTLP HTTP ingestion ([daemon] listen_port_http, default 4318), the /metrics Prometheus scrape endpoint and the GET /health liveness probe. One port, four surfaces.
• The query API can be disabled at startup by setting [daemon] api_enabled = false. Useful when the daemon runs in a hostile multi-tenant host and you only want OTLP ingestion. In that mode, /metrics and /health stay exposed, they are infrastructure surfaces, not part of the query API.
• For Kubernetes or load-balancer probes, prefer GET /health over GET /api/status: /health is always on, holds no locks and stays responsive under any ingestion load.
• The findings ring buffer (a fixed-size circular store that evicts oldest entries when full) is bounded by [daemon] max_retained_findings (default 10000). Older findings are evicted FIFO.

Restricting writes in production (reverse proxy)

A common production requirement is to let any developer read findings while reserving the write paths (acknowledge and revoke) and the official report export to architects or DevOps. This stops a finding from being acked without sign-off from the people accountable for the production posture.

The daemon does not carry an identity provider or a role model. The optional [daemon.ack] api_key (see POST /api/findings/{signature}/ack) is a single shared secret: it gates writes coarsely, but it cannot tell one user from another and cannot express "this group may, that group may not". For per-identity authorization, put a reverse proxy in front of the daemon. The proxy authenticates every caller against your SSO, then authorizes by HTTP method and path. The daemon stays a pure analysis engine, which matches its design (no implicit network surface, no embedded IAM).

The rule the proxy enforces:

/api/export/report sits in the privileged column because it materializes the full report snapshot that feeds the official HTML dashboard. Producing an official report is itself a privileged action, see Reporting for the CI-side counterpart (who may run disclose --intent official).

oauth2-proxy + nginx

oauth2-proxy handles the OIDC authentication and surfaces the authenticated identity as response headers. Its /oauth2/auth endpoint also enforces group membership per request through the allowed_groups query parameter, so the authorization decision is made by oauth2-proxy, not by fragile nginx if logic. nginx routes privileged paths to a group-checked auth subrequest and everything else to a plain one.

oauth2-proxy.cfg (auth-only mode, nginx does the proxying):

provider          = "oidc"
oidc_issuer_url   = "https://sso.example.com/realms/prod"
client_id         = "perf-sentinel"
client_secret     = "${OAUTH2_PROXY_CLIENT_SECRET}"   # from your secret manager, never committed
cookie_secret     = "${OAUTH2_PROXY_COOKIE_SECRET}"   # 32-byte base64
email_domains     = ["example.com"]
upstreams         = ["static://202"]   # auth-only: return 202 on success, nginx proxies the daemon
reverse_proxy     = true
set_xauthrequest  = true               # emit X-Auth-Request-User / -Email / -Groups
oidc_groups_claim = "groups"           # so the group claim reaches nginx
scope             = "openid email groups"

nginx.conf (relevant server block):

upstream perf_sentinel { server 127.0.0.1:4318; }   # daemon, loopback-only
upstream oauth2_proxy  { server 127.0.0.1:4180; }

server {
    listen 443 ssl;
    server_name perf-sentinel.internal;
    # ssl_certificate / ssl_certificate_key ...

    # oauth2-proxy sign-in and callback routes.
    location /oauth2/ {
        proxy_pass        http://oauth2_proxy;
        proxy_set_header  Host                     $host;
        proxy_set_header  X-Real-IP                $remote_addr;
        proxy_set_header  X-Forwarded-Proto        $scheme;
        proxy_set_header  X-Auth-Request-Redirect  $request_uri;
    }

    # Plain authentication: any valid SSO session.
    location = /oauth2/auth {
        internal;
        proxy_pass               http://oauth2_proxy;
        proxy_pass_request_body  off;
        proxy_set_header         Content-Length "";
        proxy_set_header         X-Original-URI $request_uri;
    }

    # Group-checked authentication: oauth2-proxy returns 403 when the
    # caller is not in the group, which auth_request propagates as 403.
    location = /oauth2/auth-admin {
        internal;
        proxy_pass               http://oauth2_proxy/oauth2/auth?allowed_groups=perf-sentinel-admins;
        proxy_pass_request_body  off;
        proxy_set_header         Content-Length "";
        proxy_set_header         X-Original-URI $request_uri;
    }

    # Privileged routes: ack create/revoke and the official report export.
    # A regex location wins over the /api/ prefix, so these never fall
    # through to the open rule below.
    location ~ ^/api/(findings/[^/]+/ack|export/report)$ {
        auth_request /oauth2/auth-admin;
        error_page 401 = /oauth2/sign_in;
        auth_request_set $auth_user $upstream_http_x_auth_request_user;
        proxy_set_header X-User-Id $auth_user;   # overwrites any client-supplied value
        proxy_pass       http://perf_sentinel;
        proxy_set_header Host $host;
    }

    # Everything else under /api/: read access for any authenticated user.
    location /api/ {
        auth_request /oauth2/auth;
        error_page 401 = /oauth2/sign_in;
        auth_request_set $auth_user $upstream_http_x_auth_request_user;
        proxy_set_header X-User-Id $auth_user;
        proxy_pass       http://perf_sentinel;
        proxy_set_header Host $host;
    }
}

Why this is safe

• Bind the daemon to loopback ([daemon] listen_address = "127.0.0.1") or an internal interface the proxy alone can reach. The proxy is the only front door.
• Keep [daemon.ack] api_key set as a second factor. If someone reaches the daemon port directly, bypassing the proxy, they still cannot write without the key.
• The daemon trusts X-User-Id for the audit by field. The nginx block sets it from the authenticated subrequest ($auth_user) and so overwrites any value a client supplies, which closes the spoofing gap. The authenticated identity then lands in the JSONL ack store, giving you an audit trail of who acked what.
• perf-sentinel-admins is illustrative. Use whatever group your IdP exposes in the groups claim.

Endpoints

GET /api/status

Returns a compact liveness object. Use this as a readiness probe or as the cheapest way to verify the daemon is up.

Query parameters: none.

Response shape:

The three gauge/capacity pairs back the Headroom chart of perf-sentinel query monitor's Trends tab: each pair reads as "how close is this runtime gauge to its configured cap". The settings advisor starts hinting at 90% of max_active_traces. The fields are additive; clients written against older daemons keep parsing.

Example:

curl -sS http://127.0.0.1:4318/api/status

{
  "version": "0.8.8",
  "uptime_seconds": 48,
  "active_traces": 12,
  "max_active_traces": 10000,
  "analysis_queue_depth": 0,
  "analysis_queue_capacity": 1024,
  "stored_findings": 5,
  "max_retained_findings": 10000
}

GET /api/config

The daemon's effective [daemon] configuration, read-only (since 0.8.8). Backs the Config tab of perf-sentinel query monitor. Built as an explicit allowlist, never a blanket serialization of the internal config, so no secret is exposed: TLS cert/key paths and the ack API key are summarized to booleans (tls_configured, ack_api_key_set) and never echoed. The values are frozen at daemon startup.

Query parameters: none.

Response shape: an object with the [daemon] scalars (listen_addr, listen_port, listen_port_grpc, json_socket, max_active_traces, trace_ttl_ms, sampling_rate, max_events_per_trace, max_payload_size, environment, max_retained_findings, ingest_queue_capacity, analysis_queue_capacity, api_enabled), the summarized sub-systems (tls_configured, ack_enabled, ack_api_key_set, cors_allowed_origins, archive_configured), and the correlation block (correlation_enabled, correlation_window_ms, correlation_lag_threshold_ms, correlation_min_co_occurrences, correlation_min_confidence, correlation_max_tracked_pairs).

Example:

curl -sS http://127.0.0.1:4318/api/config

{
  "listen_addr": "127.0.0.1",
  "listen_port": 4318,
  "max_active_traces": 10000,
  "trace_ttl_ms": 30000,
  "sampling_rate": 1.0,
  "environment": "staging",
  "api_enabled": true,
  "tls_configured": false,
  "ack_enabled": true,
  "ack_api_key_set": false,
  "cors_allowed_origins": [],
  "archive_configured": false,
  "correlation_enabled": false,
  "correlation_max_tracked_pairs": 10000
}

(Fields elided above for brevity; the live response carries the full set listed under Response shape.)

GET /api/energy

Live health of the five energy/intensity backends (since 0.8.8): the four scraped measured-energy sources (Scaphandre, Kepler, Redfish, cloud SPECpower) and the Electricity Maps real-time intensity API. Backs the Scrapers tab of perf-sentinel query monitor. The effective mix itself (which source won the precedence chain per service, grid intensity per region) lives on /api/export/report under green_summary; this endpoint only answers "is each backend configured, fresh, and succeeding".

Query parameters: none.

Response shape: an object with a backends array of five entries in a fixed order (scaphandre, kepler, redfish, cloud_energy, electricity_maps), each:

The optional fields are omitted rather than zeroed for unconfigured backends: the underlying Prometheus gauges are pre-registered at 0, and a literal 0 would read as a fresh scrape. electricity_maps carries no freshness gauge by design; its liveness shows as intensity_source = "real_time" entries on the report's region breakdown.

Two age-reading caveats. A configured backend still reads last_scrape_age_seconds = 0.0 during its first scrape interval after daemon start, before anything has actually been scraped: read it together with scrapes_ok = 0 to tell "not scraped yet" from "fresh". And for cloud_energy the age tracks the reachability of the configured Prometheus endpoint, not per-service coverage: a tick counts as successful as soon as one service yields a reading.

Example:

curl -sS http://127.0.0.1:4318/api/energy

{
  "backends": [
    {
      "backend": "scaphandre",
      "configured": true,
      "last_scrape_age_seconds": 3.0,
      "scrapes_ok": 120,
      "scrapes_failed": 2
    },
    { "backend": "kepler", "configured": false },
    { "backend": "redfish", "configured": false },
    { "backend": "cloud_energy", "configured": false },
    { "backend": "electricity_maps", "configured": true }
  ]
}

GET /api/findings

Returns a JSON array of recent findings, newest first. Each element wraps the finding itself plus a daemon-side ingestion timestamp.

Query parameters:

Unknown parameters are ignored. Malformed values (e.g. limit=abc) return HTTP 400 with an axum-generated error body.

Response shape: array of StoredFinding. Each StoredFinding has:

• finding: the detected finding. See Finding schema below.
• stored_at_ms: integer Unix timestamp in milliseconds, recorded when the daemon inserted this finding into the ring buffer.

Example:

curl -sS "http://127.0.0.1:4318/api/findings?severity=warning&limit=2"

[
  {
    "finding": {
      "type": "n_plus_one_sql",
      "severity": "warning",
      "trace_id": "trace-n1-sql",
      "service": "order-svc",
      "source_endpoint": "POST /api/orders/42/submit",
      "pattern": {
        "template": "SELECT * FROM order_item WHERE order_id = ?",
        "occurrences": 6,
        "window_ms": 250,
        "distinct_params": 6
      },
      "suggestion": "Use WHERE ... IN (?) to batch 6 queries into one",
      "first_timestamp": "2025-07-10T14:32:01.000Z",
      "last_timestamp": "2025-07-10T14:32:01.250Z",
      "green_impact": {
        "estimated_extra_io_ops": 5,
        "io_intensity_score": 6.0,
        "io_intensity_band": "high"
      },
      "confidence": "daemon_staging"
    },
    "stored_at_ms": 1776350162450
  },
  {
    "finding": {
      "type": "n_plus_one_http",
      "severity": "warning",
      "trace_id": "trace-n1-http",
      "service": "order-svc",
      "source_endpoint": "POST /api/orders/42/submit",
      "pattern": {
        "template": "GET /api/users/{id}",
        "occurrences": 6,
        "window_ms": 200,
        "distinct_params": 6
      },
      "suggestion": "Use batch endpoint with ?ids=... to batch 6 calls into one",
      "first_timestamp": "2025-07-10T14:32:01.000Z",
      "last_timestamp": "2025-07-10T14:32:01.200Z",
      "green_impact": {
        "estimated_extra_io_ops": 5,
        "io_intensity_score": 6.0,
        "io_intensity_band": "high"
      },
      "confidence": "daemon_staging"
    },
    "stored_at_ms": 1776350162450
  }
]

Finding schema

The finding object exposed by /api/findings and /api/findings/{trace_id} is identical to the JSON emitted by perf-sentinel analyze --format json. Stable fields as of v0.4.1:

GET /api/findings/{trace_id}

Returns all findings whose trace_id matches the path segment, as a JSON array. Same element shape as /api/findings. Hard cap of 1000 entries applies (pathological traces with hundreds of N+1 clusters).

Path parameter: trace_id (string, exact match). The path segment is URL-decoded by axum before comparison.

Response shape: same Vec<StoredFinding> as /api/findings. An empty array [] is returned when the trace ID is unknown (the endpoint does not return 404).

Example:

curl -sS "http://127.0.0.1:4318/api/findings/trace-n1-sql"

[
  {
    "finding": {
      "type": "n_plus_one_sql",
      "severity": "warning",
      "trace_id": "trace-n1-sql",
      "service": "order-svc",
      "source_endpoint": "POST /api/orders/42/submit",
      "pattern": {
        "template": "SELECT * FROM order_item WHERE order_id = ?",
        "occurrences": 6,
        "window_ms": 250,
        "distinct_params": 6
      },
      "suggestion": "Use WHERE ... IN (?) to batch 6 queries into one",
      "first_timestamp": "2025-07-10T14:32:01.000Z",
      "last_timestamp": "2025-07-10T14:32:01.250Z",
      "green_impact": {
        "estimated_extra_io_ops": 5,
        "io_intensity_score": 6.0,
        "io_intensity_band": "high"
      },
      "confidence": "daemon_staging"
    },
    "stored_at_ms": 1776350162450
  }
]

GET /api/explain/{trace_id}

Returns the span tree for a trace still held in the daemon correlation window (default TTL: 30 seconds after the last span of the trace arrived). Useful for debugging a live trace right after it is emitted.

Important: findings are retained in the ring buffer long after the trace itself evicts from the window. That means /api/findings/{trace_id} keeps working for hours after the trace is gone, but /api/explain/{trace_id} only works within the TTL window.

Path parameter: trace_id (string, exact match).

Response shape (trace in memory): object with a roots array. Each node describes a span with:

Response shape (trace unknown or evicted): an object with a single error field.

Examples:

# Trace still in memory
curl -sS "http://127.0.0.1:4318/api/explain/trace-n1-sql"

{
  "roots": [
    {
      "children": [],
      "duration_us": 800,
      "findings": [
        {
          "occurrences": 6,
          "severity": "warning",
          "suggestion": "Use WHERE ... IN (?) to batch 6 queries into one",
          "type": "n_plus_one_sql"
        }
      ],
      "operation": "SELECT",
      "parent_span_id": null,
      "service": "order-svc",
      "span_id": "span-1",
      "template": "SELECT * FROM order_item WHERE order_id = ?",
      "timestamp": "2025-07-10T14:32:01.000Z"
    }
  ]
}

# Trace not in memory (evicted or never seen)
curl -sS "http://127.0.0.1:4318/api/explain/trace-does-not-exist"

{
  "error": "trace not found in daemon memory"
}

GET /api/correlations

Returns active cross-trace temporal correlations, sorted by confidence descending. Empty array when [daemon.correlation] enabled = false (default). Capped at 1000 entries.

Query parameters: none.

Response shape: array of CrossTraceCorrelation. Each entry has:

Example:

curl -sS "http://127.0.0.1:4318/api/correlations"

[
  {
    "source": {
      "finding_type": "redundant_sql",
      "service": "cache-svc",
      "template": "SELECT * FROM settings WHERE key = ?"
    },
    "target": {
      "finding_type": "n_plus_one_sql",
      "service": "order-svc",
      "template": "SELECT * FROM order_item WHERE order_id = ?"
    },
    "co_occurrence_count": 2,
    "source_total_occurrences": 1,
    "confidence": 2.0,
    "median_lag_ms": 0.0,
    "first_seen": "2026-04-16T14:36:02.450Z",
    "last_seen": "2026-04-16T14:36:02.450Z"
  }
]

GET /api/export/report

Snapshot the daemon's current in-memory state as a Report JSON, identical in shape to perf-sentinel analyze --format json. This closes the loop between the live daemon and the post-mortem perf-sentinel report HTML dashboard: the HTML report can ingest a daemon snapshot over HTTP via standard shell composition.

The analysis section reflects daemon-lifetime counters (cumulative since daemon start). The green_summary field is refreshed by the event loop after each batch (regions, top offenders, avoidable I/O ratio, CO2 numbers, scoring config), so the snapshot carries a live CO2 picture. The chip banner and the GreenOps tab in the HTML dashboard surface naturally on Electricity-Maps-configured daemons. The quality gate is not recomputed on the snapshot path. See 05 · GreenOps & carbon for the full audit-trail story.

# Materialize a live daemon snapshot as an HTML dashboard
curl -s http://daemon.internal:4318/api/export/report \
    | perf-sentinel report --input - --output report.html

{
  "by": "alice@example.com",
  "reason": "deferred to next quarter, see TICKET-1234",
  "expires_at": "2026-08-01T00:00:00Z"
}

SIG="n_plus_one_sql:order-svc:_api_v1_orders:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
curl -fsS -X POST "http://127.0.0.1:4318/api/findings/${SIG}/ack" \
  -H "Content-Type: application/json" \
  -H "X-User-Id: alice@example.com" \
  -d '{"reason":"deferred to next quarter","expires_at":"2026-08-01T00:00:00Z"}'
# 201 Created

[
  {
    "action": "ack",
    "signature": "n_plus_one_sql:order-svc:_api_v1_orders:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "by": "alice@example.com",
    "reason": "deferred to next quarter",
    "at": "2026-05-04T13:30:00Z",
    "expires_at": "2026-08-01T00:00:00Z"
  }
]

groups:
  - name: perf-sentinel
    rules:
      - alert: PerfSentinelCriticalFinding
        expr: perf_sentinel_findings_total{severity="critical"} > 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "perf-sentinel detected a critical performance anti-pattern"
          description: |
            Critical finding count is {{ $value }}.
            Query `/api/findings?severity=critical` on the daemon for details.

URL:     http://perf-sentinel.internal:4318/api/findings
Method:  GET
Params:  service=order-svc
         limit=20
Fields:  $.finding.type,
         $.finding.severity,
         $.finding.pattern.template,
         $.finding.pattern.occurrences,
         $.finding.source_endpoint,
         $.stored_at_ms

#!/usr/bin/env bash
set -euo pipefail

DAEMON="${DAEMON:-http://127.0.0.1:4318}"
response=$(curl -sSf --max-time 3 "${DAEMON}/api/status")
uptime=$(echo "$response" | jq -r '.uptime_seconds')
traces=$(echo "$response" | jq -r '.active_traces')
findings=$(echo "$response" | jq -r '.stored_findings')

if [ "$uptime" -gt 300 ] && [ "$traces" -eq 0 ] && [ "$findings" -eq 0 ]; then
  echo "perf-sentinel daemon has been idle for ${uptime}s with no traces or findings"
  echo "Check ingestion path: OTLP endpoint, collector config, Java agent env vars"
  exit 1
fi