Detection algorithms

Detection is the fourth pipeline stage. It analyzes correlated traces to identify seven types of anti-patterns: N+1 queries, redundant calls, slow operations, excessive fanout, chatty services, connection pool saturation and serialized-but-parallelizable calls.

Shared pattern: borrowed HashMap keys

All three detectors group spans by a composite key. A key insight is that the spans live in the Trace struct, which outlives the detector function. This means we can borrow from the spans instead of cloning:

// N+1: group by (event_type, template)
HashMap<(&EventType, &str), Vec<usize>>

// Redundant: group by (event_type, template, params)
HashMap<(&EventType, &str, &[String]), Vec<usize>>

// Slow: group by (event_type, template)
HashMap<(&EventType, &str), Vec<usize>>

The values are Vec<usize>: indices into trace.spans rather than cloned spans. This keeps the HashMap small and avoids copying the event data.

For a trace with 50 spans, each having a 40-character template string, borrowed keys save 50 × 40 = 2,000 bytes of String allocations per grouping pass.

N+1 detection

Algorithm

1. Skip SQL spans whose template is a session command (normalize::sql::is_session_command)
2. Group spans by (&EventType, &str template)
3. Skip groups with fewer than threshold occurrences (default 5)
4. Count distinct parameter sets via HashSet<&[String]>
5. Skip groups with fewer than threshold distinct params (same params = redundant, not N+1)
6. Compute time window between earliest and latest timestamp
7. Skip groups where the window exceeds window_limit_ms (default 500ms)
8. Assign severity: Critical if >= 10 occurrences, Warning otherwise

Distinct params via borrowed slices

let distinct_params: HashSet<&[String]> = indices
    .iter()
    .map(|&i| trace.spans[i].params.as_slice())
    .collect();

Using &[String] as a HashSet key is a critical design choice:

• No allocation: borrows the existing Vec as a slice reference
• No collision bug: directly compares the full Vec content, unlike a join(",") approach where ["a,b"] and ["a", "b"] would produce the same joined string

Rust's standard library implements Hash and Eq for &[T] when T: Hash + Eq, making this zero-cost.

Iterator-based window computation

pub fn compute_window_and_bounds_iter<'a>(
    mut iter: impl Iterator<Item = &'a str>,
) -> (u64, &'a str, &'a str) {
    let Some(first) = iter.next() else {
        return (0, "", "");
    };
    let mut min_ts = first;
    let mut max_ts = first;
    let mut has_second = false;
    for ts in iter {
        has_second = true;
        if ts < min_ts { min_ts = ts; }
        if ts > max_ts { max_ts = ts; }
    }
    // ...
}

Why iterator instead of &[&str]? The caller would need to collect timestamps into a Vec first:

// Old (allocates):
let timestamps: Vec<&str> = indices.iter().map(|&i| ...).collect();
let (w, min, max) = compute_window_and_bounds(&timestamps);

// New (zero allocation):
let (w, min, max) = compute_window_and_bounds_iter(
    indices.iter().map(|&i| trace.spans[i].event.timestamp.as_str())
);

The iterator-based version eliminates one Vec<&str> allocation per detection group. With 3 detectors × multiple groups per trace × thousands of traces, this adds up.

The has_second boolean replaces a count variable that was only used to check count < 2. This avoids incrementing a counter on every iteration.

ISO 8601 timestamp parser

fn parse_timestamp_ms(ts: &str) -> Option<u64> {
    let time_part = ts.split('T').nth(1)?;
    let time_part = time_part.trim_end_matches('Z');
    let mut colon_parts = time_part.split(':');
    let hours: u64 = colon_parts.next()?.parse().ok()?;
    let minutes: u64 = colon_parts.next()?.parse().ok()?;
    let sec_str = colon_parts.next()?;
    // ... parse seconds and fractional part
}

Why not chrono? chrono adds ~150KB to the binary and parses ~200ns per timestamp. This hand-rolled parser handles the fixed format (YYYY-MM-DDTHH:MM:SS.mmmZ) in ~5ns by splitting on known delimiters and using iterator .next() calls instead of collecting into Vecs.

The parser uses iterators throughout (split(':') -> .next(), split('.') -> .next()) to avoid allocating intermediate Vec<&str> collections.

The parser computes milliseconds since Unix epoch by parsing both the date (YYYY-MM-DD) and time components. The date-to-days conversion uses the Howard Hinnant algorithm (public domain), which requires no external dependencies.

Lexicographic timestamp comparison

Min/max timestamps are found via string comparison: if ts < min_ts { min_ts = ts; }. This works because ISO 8601 timestamps with fixed-width fields (2025-07-10T14:32:01.123Z) sort chronologically when compared lexicographically. This is guaranteed by the ISO 8601 standard, Section 5.3.3.

Sanitizer-aware classification

OpenTelemetry agents and database drivers collapse SQL literals to placeholder tokens before the statement reaches perf-sentinel. The placeholder style depends on the stack: JDBC agents produce bare ?, PostgreSQL native drivers (pgx, asyncpg, sqlx, node-pg) emit $1/$2 (which normalize_sql rewrites to $? with empty params since v0.7.7), Python DB-API drivers emit %s, .NET drivers emit @p0/@Name, and Oracle/SQLAlchemy emit :name. In all cases the sanitized statement reaches perf-sentinel with the placeholder already in place and an empty params vector. The standard distinct_params >= threshold check sees one distinct empty params slice and never fires, the redundant detector then groups all the spans together and misclassifies them as redundant_sql.

The heuristic in crates/sentinel-core/src/detect/sanitizer_aware.rs recovers the correct classification via four signals, evaluated in order:

1. looks_sanitized: every span has a recognized placeholder in its template (?, $?, %s, @alpha, :alpha) and an empty params vector. See template_has_placeholder in sanitizer_aware.rs for the full list. Required to activate the heuristic at all.
2. has_orm_scope: at least one OpenTelemetry instrumentation scope on the spans matches a known ORM marker (Hibernate, Spring Data, EF Core, SQLAlchemy, ActiveRecord, GORM, Prisma, Diesel, Laravel/Eloquent, Doctrine, etc.). Markers are matched with a word-boundary check (preceded and followed by a non-alphanumeric byte), so jpa only fires on spring-data-jpa and friends, never on myappjpastats. A positive match is treated as strong evidence of N+1.
3. timing_variance_suggests_n_plus_one: when the scope signal is absent, fall back to the coefficient of variation of duration_us. True N+1 hits different rows with different cache states, so the spread is wider, cached redundant calls cluster tightly. Threshold 0.5 is empirical.
4. sequential_siblings_indexed (Strict mode only): every span shares one non-empty parent_span_id and the group chains prev.end_us <= next.start_us after sort by start time. Bounds are computed in microseconds to avoid the silent truncation of sub-millisecond durations. Substitutes for has_orm_scope on bare-driver stacks (Vert.x reactive PG, pgx, asyncpg, sqlx, Prisma queryRaw) that never emit an ORM scope.
5. high_occurrence (Strict mode, all branches): a high occurrence count (>= 3 x n_plus_one_threshold, default 15) serves as both a primary signal and a corroborator. Under the looks_sanitized guard (params empty, template has ?), 15+ identical sanitized templates in one trace is structurally n+1 regardless of ORM scope, sequential siblings, or timing variance. Legacy polling loops below the threshold (typical 5-10 calls per request) stay classified as redundant_sql.

The four emission modes (Auto, Strict, Always, Never) are documented in Configuration § "sanitizer_aware_classification" with their precision/recall trade-offs.

The HTML detail makes that decision auditable without changing it: direct N+1 findings are labeled direct, recovered groups are labeled heuristic, and the view shows the observation window, distinct-parameter count, available p50/p99/CV timing statistics, and one timestamped duration/status row per exact offending span in the representative trace. For a cross-trace finding, the summary still reports the full occurrence count and the view states how many of those occurrences the representative trace proves. Raw parameters and targets remain omitted, and unrelated spans stay grouped. Older reports without detection settings cannot rebuild exact span ids, so matching spans remain grouped instead of being presented as individual proof.

Known limit

looks_sanitized cannot tell a sanitized literal ? apart from a PostgreSQL JSONB existence operator (data ? 'key') when the latter happens to appear in a query with no other literals. The harm direction is asymmetric: a misclassified JSONB group flips from redundant_sql to n_plus_one_sql, both of which contribute equally to GreenOps avoidable_io_ops, only the suggestion text differs.

HTTP extension (0.7.8+)

The same dispatch also covers HTTP outbound groups via classify_http_group_indexed. HTTP has no looks_sanitized analogue (the normalizer always collapses path IDs to {id}/{uuid}, params are never erased to empty the way a SQL sanitizer erases them), and no ORM scope concept. The HTTP path therefore relies on a narrower signal set:

• Auto/Always: timing variance alone (CV >= 0.5).
• Strict: a primary signal (HTTP placeholder in the template, high occurrence, or sequential siblings) corroborated by timing variance. Unlike the SQL path, high occurrence alone is not sufficient corroboration for HTTP, because without the looks_sanitized gate a busy polling loop or CDN-warmed repeated call would otherwise be promoted to n_plus_one_http.

Known limit: query-string redaction

N+1 HTTP detection requires the varying parameter to be visible in the span. An N+1 loop that varies a path segment is detected (distinct extracted params, or the {id} placeholder anchors the Strict primary). An N+1 loop that varies a query parameter is invisible when the instrumentation redacts the query string before export. OpenTelemetry .NET System.Net.Http redacts to ?* by default, so every call carries a byte-identical url.full, distinct_params collapses to 1, and the group is correctly classified as redundant_http. The distinguishing parameter was destroyed upstream, so no trace consumer can recover it. See Limitations § "HTTP query-string redaction and N+1 visibility" for the operator-facing workarounds.

Redundant detection

Borrowed slice keys

HashMap<(&EventType, &str, &[String]), Vec<usize>>

The three-part key includes the full params slice, ensuring that two spans with the same template but different params are in different groups. This is the correct behavior: redundant detection flags exact duplicates (same template AND same params).

The use of &[String] instead of joining params into a single string prevents a subtle collision bug: ["a,b"] (one param containing a comma) and ["a", "b"] (two params) would produce the same joined key "a,b" but are semantically different parameter sets.

Session commands

Like N+1 detection, grouping skips SQL spans whose template is a session command. A pooled driver emits one per connection checkout, so a request borrowing N connections would report N exact duplicates that no cache can deduplicate. These statements stay in total_io_ops but leave avoidable_io_ops, which is derived from findings: io_waste_ratio therefore falls, and an io_waste_ratio_max calibrated on an earlier version becomes looser.

Severity

• Info (< 5 occurrences): common for config lookups, health checks
• Warning (>= 5 occurrences): likely a loop bug or missing cache

The threshold of 2 (minimum to flag) catches any exact duplicate. Unlike N+1 which requires 5+ occurrences, even 2 identical queries in one request is suspicious and worth flagging at Info level.

ORM bind parameters

ORMs that use named bind parameters (Entity Framework Core with @__param_0, Hibernate with ?1) produce SQL spans where actual parameter values are not visible in db.statement/db.query.text. In this case, N+1 patterns (same query with different values) appear as redundant queries (same template, same visible params), because perf-sentinel cannot distinguish the bound values. Both findings correctly identify the repeated query pattern. ORMs that inline literal values (SeaORM raw statements, JDBC without prepared statements) allow accurate N+1 vs redundant classification.

Sanitizer-aware classification (0.5.7+)

The same shape appears whenever the OpenTelemetry agent runs its SQL statement sanitizer (default ON), since literals are collapsed to ? before the span reaches perf-sentinel. The standard distinct-params rule sees one bucket of empty params and rejects the group, so the redundant detector misclassifies the N+1 as redundant_sql and the operator gets the wrong remediation.

The 0.5.7 sanitizer-aware heuristic recovers the correct classification by running a second pass over the same (event_type, template) groups that the first pass rejected. It activates only when every span in the group has an empty params vector and a recognized placeholder in its template (the on-wire signature of a sanitized N+1). Since v0.7.7 the template_has_placeholder check recognizes five styles: bare ? (JDBC), $? (PostgreSQL native, normalized from $1/$2), %s (Python DB-API), @alpha (.NET, excluding @@ system variables), :alpha (Oracle/SQLAlchemy, excluding :: casts). Truly literal-free queries like SELECT NOW() have no placeholder in the template and do not activate the heuristic. It then evaluates two independent signals:

1. Instrumentation scope marker (high confidence). Per-span instrumentation_scopes chains are searched, case-insensitively, for any of the known ORM substrings: spring-data, hibernate, jpa, micronaut-data, jdbi, r2dbc, entityframeworkcore, entity-framework, sqlalchemy, django, active-record/activerecord, gorm, sequelize, prisma, typeorm, mongoose, sea-orm, diesel. Bare SQL drivers like sqlx (Go/Rust), pgx, asyncpg and the Vert.x reactive PG client are intentionally excluded: their n+1 patterns are handled by the sequential-siblings signal instead. A match flips the verdict to LikelyNPlusOne.
2. Timing-variance fallback (medium confidence). When no ORM marker is present, the heuristic computes the coefficient of variation (std-dev / mean) of duration_us. True N+1 lookups hit different rows with different cache states, so durations spread out (CV typically 0.4 to 1.0), cached redundant calls cluster tightly (CV near 0). The threshold of 0.5 is empirical and is the only knob in the heuristic. At least 3 spans are required for a stable variance estimate.

The configurable [detection] sanitizer_aware_classification mode gates emission across four points on a recall-vs-precision dial: auto (default) emits when either signal fires, strict (0.5.8+) requires a primary signal (ORM scope OR sequential siblings) plus a corroborating signal (variance OR, on the ORM branch, high occurrence count), always reclassifies every sanitized group regardless of signal, and never disables the second pass entirely. Findings emitted by the heuristic carry classification_method = SanitizerHeuristic so consumers can distinguish them from direct classifications. The mode picks where to sit on the trade-off:

• auto favors recall: catches all ORM-induced N+1 because the ORM scope alone fires the verdict, at the cost of absorbing legitimate redundant_sql findings on Spring Data / EF Core stacks (a findById(sameId) called in a loop served from row cache flips to n_plus_one_sql).
• strict favors precision: preserves redundant_sql findings on moderate-count cached identical queries (below the 3 x threshold bar) because the timing-variance signal stays low. Above the bar (default 15 occurrences), any sanitized group fires regardless of ORM scope, sequential siblings, or variance. Recommended when actionable redundant_sql findings are valuable signal in your environment.

Known limits: a real single-param redundancy whose literal happens to be collapsed by the sanitizer (e.g. SELECT * FROM config WHERE key = ? queried 10 times for the same key) cannot be distinguished from an N+1 without scope or variance signal. In auto mode it flips to n_plus_one_sql whenever an ORM scope is present (harm-reducing direction: batch fetch is a strict superset of "cache one value"). In strict mode it stays redundant_sql because the timing variance is low. In always mode it always flips. In never mode the heuristic is bypassed entirely.

The timing-variance signal (timing_variance_suggests_n_plus_one, coefficient of variation > 0.5) carries an asymmetric-harm tuning: a false positive merely swaps redundant_sql for n_plus_one_sql (same avoidable_io_ops weight, only the suggestion text differs), while a false negative leaves a real N+1 silent, so the threshold favors false positives. Under strict the signal becomes load-bearing as the sole corroborator on the ORM branch below the high-occurrence bar, and it has a warm-cache blind spot: a real ORM-induced N+1 against a fully warm row cache (e.g. 100 primary-key lookups with every row in shared_buffers) can cluster within roughly 10% (CV around 0.1) and stay silent. The threshold is [detection] sanitizer_aware_min_cv, default 0.5 across modes. The simulation lab supplied the empirical case the default was waiting for: under strict, ten identical Doctrine lookups served from cache on a PHP-FPM worker measured a CV near 0.75 once the runner was loaded, crossing the bar and turning a redundant_sql finding into n_plus_one_sql. Raising the knob to 1.0 restores the redundant verdict there while the high-occurrence bar keeps real N+1 reported.

Slow detection

Saturation arithmetic

let threshold_us = threshold_ms.saturating_mul(1000);
// ...
if max_duration_us > threshold_us.saturating_mul(5) {
    Severity::Critical
}

saturating_mul returns u64::MAX on overflow instead of wrapping to zero. This prevents a malicious or misconfigured threshold_ms = u64::MAX from disabling severity thresholds.

Not part of waste ratio

Slow findings have green_impact.estimated_extra_io_ops = 0. They are necessary operations that happen to be slow: they need optimization (indexing, caching), not elimination. Including them in the waste ratio would conflate "avoidable I/O" (N+1, redundant) with "slow I/O" (needs a different fix).

Detection orchestration

pub fn detect(traces: &[Trace], config: &DetectConfig) -> Vec<Finding> {
    let mut findings = Vec::new();
    for trace in traces {
        findings.extend(detect_n_plus_one(trace, ...));
        findings.extend(detect_redundant(trace));
        findings.extend(detect_slow(trace, ...));
    }
    findings
}

The detectors run sequentially on each trace. While they could theoretically share a single grouping pass, the key types differ ((&EventType, &str) vs (&EventType, &str, &[String])) and the separate implementations are clearer and independently testable. With typical trace sizes of 10-50 spans, multiple O(n) passes are negligible.

Fanout detection

Algorithm

1. Group spans by parent_span_id
2. Skip groups where the parent has max_fanout or fewer children (default 20)
3. For each parent exceeding the threshold, emit an ExcessiveFanout finding
4. Severity: Warning if > max_fanout, Critical if > 3x max_fanout

The fanout detector uses a HashMap<&str, usize> span index for O(1) parent lookup and compute_window_and_bounds to compute the chronological span of child timestamps in a single pass.

Not part of waste ratio

Like slow findings, fanout findings have green_impact.estimated_extra_io_ops = 0. Excessive fanout is a structural concern (too many child operations per parent) that needs architectural optimization, not I/O elimination. Both the dedup loop and the green_impact enrichment use FindingType::is_avoidable_io() to make this determination, ensuring a single source of truth.

Cross-trace slow percentiles

detect_slow_cross_trace collects slow spans across all traces of one batch (the whole input for analyze, one eviction batch in the daemon) and computes p50/p95/p99 percentiles per normalized template. This complements the per-trace slow detection by identifying templates that are consistently slow across multiple requests.

• Only spans exceeding the threshold are collected (pre-filter for performance)
• Only templates appearing in at least 2 distinct traces are reported (single-trace cases are handled by per-trace detection)
• Percentile computation uses the nearest-rank method via div_ceil

The daemon also counts slow spans across batches, see the next section.

Cross-batch slow window (daemon)

The daemon analyzes eviction batches of about trace_ttl_ms / 2, so a template slow once every few minutes never gathers slow_query_min_occurrences spans in one batch. daemon/slow_window.rs keeps, on the analysis worker, a window of slow episodes per (event type, service, grouping, normalized template) key. [detection] slow_query_window_minutes (default 15, 0 disables, range 0-60) sets the window. analyze and the other batch commands never build it.

• Episodes. Slow spans of one key within max(60 s, 1.5 x trace_ttl_ms) of an episode's first span count as one episode, which keeps the slowest span. An isolated slow span, or a short lock whose victims evict within that span of time, stays one episode and only reaches the duration histogram.
• Suppression. A slow span whose (type, template, grouping) already produced a slow finding in the same batch is not counted. The entry of its key loses its episodes and enters the cooldown, as if it had reported.
• Emission. A key reports when a batch opens a fresh episode and the window holds at least slow_query_min_occurrences episodes from at least 2 distinct traces. Time is analysis time, not span timestamps.
• Cooldown. After a report the key clears its episodes and stays silent for one window. A persisting problem reports again at its first new episode after the cooldown, once the window holds enough episodes.
• Shape. The finding is built by the same function as a batch cross-trace slow finding: same type, severity rule, suggestion wording and signature, so it folds with them in the findings store. pattern.occurrences and the percentiles count episodes, one span (the slowest) per episode.
• Key cap. At most 1024 keys are tracked. Slow spans of a new key past the cap are refused and counted in perf_sentinel_slow_window_keys_refused_total, with a warning logged once per process.
• Representative trace. The finding's trace_id is the trace of the fresh episode's slowest span, which belongs to the current batch and is retained for /api/explain. The signature does not depend on it. The other episodes come from earlier batches, whose traces the store may no longer hold.
• Long locks. Slowness lasting more than two episodes, for example a row lock held for several minutes on a low-traffic template, still reports: durations alone cannot tell a lock from a chronic problem. Set the window to 0 to turn the feature off.

Chatty service detection

Algorithm

1. Filter spans to HTTP outbound only (type: http_out)
2. Count total HTTP outbound spans in the trace
3. If count < chatty_service_min_calls (default 15), skip
4. Collect the top called normalized endpoints for the suggestion message
5. Assign severity: Warning if > threshold, Critical if > 3x threshold

Input:  trace with N spans
Output: 0 or 1 ChattyService finding

filter spans where type == http_out
if count(http_spans) < chatty_service_min_calls:
    return []

group http_spans by normalized template
sort groups by count descending
top_endpoints = first 5 groups

severity = Critical if count >= 3 * threshold else Warning
emit finding with top_endpoints in suggestion

Complexity: O(n) to filter and count, O(k log k) to sort groups where k is the number of distinct templates. Since k << n in practice, this is effectively O(n).

Difference from fanout

Excessive fanout detects a single parent with too many direct children. Chatty service detects an entire trace with too many outbound HTTP calls, independently of the parent-child structure. A trace can trigger both when a single parent generates all the calls or only chatty service when the calls are spread across multiple parents.

Not part of waste ratio

Chatty service findings have green_impact.estimated_extra_io_ops = 0. The detector flags an architectural concern (too many inter-service calls per request), not a batching opportunity. The calls may all be necessary; the problem is that the service boundary is too fine-grained. FindingType::is_avoidable_io() returns false for ChattyService.

Connection pool saturation detection

Algorithm

1. Filter spans to SQL only (type: sql)
2. Group SQL spans by service name
3. For each service group, compute peak concurrency via sweep-line
4. If peak concurrency < pool_saturation_concurrent_threshold (default 10), skip
5. Severity is always Warning, whatever the peak: unlike fanout and chatty service, this detector has no Critical tier

Input:  trace with N spans, grouped by service
Output: 0 or more PoolSaturation findings (one per service)

for each service in sql_spans_by_service:
    events = []
    for span in service_spans:
        start = parse_timestamp(span.timestamp)
        end = start + span.duration_us
        events.push((start, +1))
        events.push((end, -1))

    sort events by timestamp, with -1 before +1 on ties
    current = 0
    peak = 0
    for (ts, delta) in events:
        current += delta
        peak = max(peak, current)

    if peak >= pool_saturation_concurrent_threshold:
        emit finding

Complexity: O(n log n) for the sort step, O(n) for the sweep. Total: O(n log n) per service group.

Sweep-line tie-breaking

When a span ends and another begins at the exact same microsecond, the algorithm processes the end event (-1) before the start event (+1). This avoids inflating peak concurrency when spans are merely adjacent rather than overlapping.

Not part of waste ratio

Pool saturation findings have green_impact.estimated_extra_io_ops = 0. High concurrency is not avoidable I/O. It signals potential contention on the database connection pool, which is a tuning or architectural concern. FindingType::is_avoidable_io() returns false for PoolSaturation.

Serialized calls detection

Algorithm

1. Drop SQL siblings whose template is a session command: a COMMIT cannot move off the chain, and counting it inflates both the link count and the sequential total
2. Group sibling spans by parent_span_id
3. For each parent group, sort children by end time (ascending)
4. Find the longest non-overlapping subsequence via dynamic programming (Weighted Interval Scheduling with unit weights)
5. If the optimal sequence has >= serialized_min_sequential (default 3) spans with distinct templates, emit a finding
6. Severity: always Info (heuristic, inherent false positive risk)

Input:  trace with N spans, grouped by parent_span_id
Output: 0 or more SerializedCalls findings

for each parent_id in spans_by_parent:
    children = spans with this parent_id
    if len(children) < serialized_min_sequential:
        skip

    sort children by end_time ascending
    
    // Predecessor computation: for each span i, binary search for p(i),
    // the rightmost span j (j < i) whose end_time <= span i's start_time.
    // O(log n) per span.
    
    // DP recurrence:
    //   dp[i] = max(dp[i-1], dp[p(i)] + 1)
    // where dp[i] = longest non-overlapping subsequence in children[0..=i]
    
    // Backtrack from dp[n-1] to reconstruct the selected spans.
    // Guard: predecessor must be strictly less than current index
    // to guarantee termination on degenerate input (zero-duration spans).
    
    if len(selected) >= serialized_min_sequential
       AND distinct_templates(selected) > 1:
        emit finding for selected sequence

Complexity: O(n log n) for sorting + O(n log n) for all binary searches + O(n) for the DP fill and backtrack = O(n log n) total per parent group. This is the same asymptotic cost as the simpler greedy approach, but the DP guarantees finding the longest possible non-overlapping sequence. For example, given spans A:[0-200ms], B:[100-150ms], C:[160-300ms], D:[310-400ms], a greedy approach sorted by start time would select {A, D} (length 2), while the DP correctly identifies {B, C, D} (length 3).

The binary search uses partition_point directly on the sorted slice, avoiding a separate predecessor array allocation.

Why info only

The detector cannot observe data dependencies between calls. Two sequential calls to different services may be intentionally ordered (e.g. create a record, then notify a dependent service). The info severity signals an investigation opportunity, not a confirmed defect.

Template filtering

The detector skips sequences where all spans share the same normalized template. That pattern is N+1 (same operation repeated with different params), not serialization. By requiring different templates, the detector targets the "fetch user, then fetch orders, then fetch preferences" pattern where the calls are independent and could run concurrently.

Time savings estimate

The finding includes the potential time savings: total_sequential_duration - max_individual_duration. If 3 sequential calls each take 100ms, parallelizing them could reduce latency from 300ms to 100ms, saving 200ms. This is a best-case estimate that assumes no shared resource contention.

Not part of waste ratio

Serialized call findings have green_impact.estimated_extra_io_ops = 0. Parallelizing calls does not reduce the total number of I/O operations. It reduces latency, not I/O volume. FindingType::is_avoidable_io() returns false for SerializedCalls.

Detection orchestration (updated)

pub fn detect(traces: &[Trace], config: &DetectConfig) -> Vec<Finding> {
    let mut findings = Vec::new();
    for trace in traces {
        findings.append(&mut detect_n_plus_one(trace, ...));
        findings.append(&mut detect_redundant(trace));
        findings.append(&mut detect_slow(trace, ...));
        findings.append(&mut detect_fanout(trace, config.max_fanout));
        findings.append(&mut detect_chatty(trace, config.chatty_service_min_calls));
        findings.append(&mut detect_pool_saturation(trace, config.pool_saturation_concurrent_threshold));
        findings.append(&mut detect_serialized(trace, config.serialized_min_sequential));
    }
    findings
}

The seven detectors run sequentially on each trace. append(&mut ...) is used instead of extend() to move the backing allocation in O(1) without iterator overhead. Cross-trace slow percentile analysis runs separately in pipeline.rs after per-trace detection and before scoring.

Cross-trace temporal correlation (daemon mode)

In daemon mode (perf-sentinel watch), perf-sentinel sees findings from all traces over time. The CrossTraceCorrelator detects recurring temporal co-occurrences between findings from different services: "every time the N+1 in order-svc fires, pool saturation appears in payment-svc within 2 seconds."

Two clocks

Every finding carries two times. Its event time is first_timestamp, the start of its first offending span, parsed with time::parse_iso8601_utc_to_ms; a missing or non-UTC value falls back to the ingest time. Its ingest time is the now_ms of the analysis tick that produced it. Pairing, orientation and lag use event time, so two findings pair when their own spans started within lag_threshold_ms of each other, whichever ticks analysed them. Retention, eviction and the reporting window use ingest time, so replayed or skewed traffic still ages out.

Internal state

pub struct CrossTraceCorrelator {
    occurrences: VecDeque<FindingOccurrence>,
    pair_counts: HashMap<PairKey, PairState>,
    endpoints: HashMap<Arc<CorrelationEndpoint>, HalfWindowCount>,
    now_idx: u64,
    pruned_idx: u64,
    config: CorrelationConfig,
}

• occurrences: the pairing horizon, a VecDeque in ingest order. Each entry holds the interned endpoint, event_ms, ingest_ms, the grid index at ingest, a capped trace id and counted_targets, the targets this occurrence already counted for as a source. An entry leaves once ingest_ms + lag_threshold_ms + ingest_skew_ms < now_ms. The horizon depends on the lag and the skew only, never on window_ms, so a 24 h window holds the same deque as a 10 min one. The skew is 2 x trace_ttl_ms, so the deque and the scan each finding makes over it scale with the TTL (about a minute of findings at the default 30 s).
• endpoints: the endpoint registry. Each distinct CorrelationEndpoint (finding type, service, template, grouping) is stored once behind an Arc, and the deque, the pair keys and counted_targets share that allocation: a long SQL template is kept once however many findings carry it. The value is the endpoint's occurrence count, the confidence denominator.
• pair_counts: keyed by PairKey (source, target), two interned Arcs. Each PairState holds the co-occurrence count, a bounded lag reservoir, a total_observations counter, a SplitMix64 PRNG state, first_seen_ms/last_seen_ms on the ingest clock and the source and target trace ids of the latest match.

Global half-window grid

Both counters, the pair's co-occurrences and the endpoint's occurrences, are a HalfWindowCount { idx, cur, prev } on one grid shared by the whole correlator: idx = now_ms / (window_ms / 2). A count reads prev + cur in its own bucket, cur one step later and 0 beyond, so it covers between half a window and one window, never more. Numerator and denominator sit on the same buckets and cover the same span: a co-occurrence is credited to the bucket where its source occurrence was counted, not to the bucket of the later ingest. A credit one bucket behind the counter's goes to prev, an older one is dropped. Reads are lazy: nothing rotates counters on each tick, and a counter nobody touches reads 0 one window after its last increment. now_idx never decreases, so a wall clock stepping back does not reset counts.

Ingest skew

ingest_skew_ms is the extra ingest-time reach that lets findings analysed in different ticks meet in the horizon. It is not a TOML key: setup_correlator derives it as 2 x trace_ttl_ms. A trace flushed by LRU pressure reaches analysis at once, while a TTL flush waits the TTL plus up to one eviction tick (half a TTL); the rest of the budget covers exporter and collector batching. The struct default (60 s) matches the default 30 s TTL.

The ingest() algorithm

ingest() is called from process_traces after findings are produced and confidence is stamped, with the batch and the tick's now_ms:

1. Advance the grid. now_idx = max(now_idx, now_ms / half_window).
2. Evict the horizon. Pop occurrences from the front while they are past lag_threshold_ms + ingest_skew_ms of ingest time. Eviction touches no counter.
3. Prune stale pairs. One HashMap::retain pass drops pairs whose last_seen_ms is older than window_ms.
4. Prune the registry. Once per grid step, drop endpoints whose count reads 0 and that no pair or horizon occurrence still pins (Arc::strong_count == 1).
5. Pair each finding. Intern its endpoint and count it on the grid, then scan the whole horizon. An occurrence pairs when its event time is within lag_threshold_ms, it is a different endpoint from a different service, and both share the same grouping key and value. The earlier event is the source and the later one the target; on equal event times the earlier arrival stays the source. The lag is the event-time delta. The finding is then pushed onto the deque, so findings of one batch pair with each other too.
6. Count once per source occurrence. Every match refreshes last_seen_ms and the sample trace id (the target's). The co-occurrence count, credited at the source occurrence's grid index, and the lag reservoir only move when the source occurrence has not yet counted for that target endpoint, tracked on the source's counted_targets. One source occurrence followed by three targets counts once, and the result does not depend on the order in which the four findings arrived.
7. Enforce the pair cap. A new pair is refused while the map is at max_tracked_pairs (default 10,000). When a batch saw refusals, the map is cut to 90% of the cap in one pass: pairs are ranked by (windowed co-occurrence count, last_seen_ms) ascending, so the lowest counts go first and, among equal counts, the stalest. The threshold comes from select_nth_unstable on the rank tuples, so only the removed keys are cloned.

The return value is the number of pairs lost to the cap in that batch (refusals plus evictions), fed to perf_sentinel_correlator_pairs_evicted_total.

The active_correlations() filter

For each pair, with every count read at now_idx:

• co_occurrence_count is the pair's windowed count. Pairs below min_co_occurrences (default 5) are skipped.
• source_total_occurrences is the source endpoint's windowed count. A source missing from the registry or reading 0 has nothing to measure the pair against, and the pair is skipped.
• confidence = co_occurrence_count / source_total_occurrences. Each co-occurrence sits in its source occurrence's bucket and counts once per source occurrence, so the ratio stays in [0, 1]; the clamp to 1 is defensive only. Pairs below min_confidence (default 0.7) are skipped.

median_lag_ms is the median of the reservoir, an event-time lag. first_seen and last_seen are on the ingest clock.

Reservoir sampling for lag values

A hot pair firing thousands of times within the window would otherwise grow lags_ms without bound. To keep memory per pair flat, record_lag uses Algorithm R reservoir sampling capped at MAX_LAG_SAMPLES = 64 (512 bytes per pair):

• While the reservoir has space, append unconditionally.
• Once full, draw r uniformly in [0, total_observations) via SplitMix64. If r < MAX_LAG_SAMPLES, replace lags_ms[r]. Conditional on r < k, r is itself uniform in [0, k), so the slot pick is unbiased without a second PRNG draw.

The PRNG is a SplitMix64 state per PairState, seeded at construction from now_ms ^ (hash_endpoint(source) << 17) ^ hash_endpoint(target). hash_endpoint is a deterministic FNV-1a over the endpoint's finding_type, service and template strings (NOT the DefaultHasher, which uses a per-process RandomState and would make the correlator non-deterministic across runs). Two daemon runs replaying the same trace file produce identical reservoir samples and therefore identical median lags.

Median lag calculation

The median() helper sorts a clone of the lag values and returns the middle element (odd length) or the midpoint of the two middle elements (even length). Sorting is bounded by MAX_LAG_SAMPLES thanks to the reservoir, so the median computation is O(k log k) with k = 64 regardless of how often the pair fires.

Memory management

• Horizon deque: about (lag_threshold_ms + ingest_skew_ms) x findings per second entries of roughly 100 bytes, whatever window_ms is.
• Endpoint registry: one entry per distinct endpoint seen in the window, template included, pruned once per grid step. Uncapped: it grows with the number of distinct endpoints, so templates that normalize poorly stay for the whole window.
• Pairs: at most max_tracked_pairs, each well under 1 KB with the 64-sample reservoir.
• CPU: one horizon scan per incoming finding, no per-tick pass over the pairs.

Integration point

The correlator is created by setup_correlator when [daemon.correlation] enabled is true (default false), with ingest_skew_ms derived from trace_ttl_ms. It is wrapped in Arc<Mutex<CrossTraceCorrelator>> and passed to process_traces. After findings are produced and pushed to the FindingsStore, the correlator's ingest() method is called with the findings and the current timestamp.

Batch mode exclusion

The correlator is not used in batch mode (perf-sentinel analyze). Cross-trace correlation requires a stream of findings over time to detect recurring patterns. A single batch run typically processes a fixed set of traces without the temporal dimension needed for meaningful correlation.

Actionable fixes (framework-aware suggestions)

Starting in v0.4.2, a suggested_fix: Option<SuggestedFix> field on Finding carries a framework-specific remediation that goes beyond the generic suggestion string. This field is populated by detect::suggestions::enrich after the per-trace detectors return, inside detect(), and on each cross-trace slow finding as build_cross_trace_finding builds it, which covers the batch detect_slow_cross_trace pass and the daemon's cross-batch slow window.

Coverage grew in seven steps:

• v1: Java/JPA only.
• v2: Quarkus reactive and non-reactive, WebFlux, Helidon SE/MP, EF Core, Diesel and SeaORM.
• v3: the seven anti-patterns that previously returned suggested_fix = None (redundant_http, slow_sql, slow_http, excessive_fanout, chatty_service, pool_saturation, serialized_calls), plus Python (Django ORM, SQLAlchemy) with scope detection via the opentelemetry.instrumentation.* prefix.
• v4: Go (GORM) and Node.js/TypeScript (Prisma) with scope detection via the @opentelemetry/instrumentation-* prefix and language detection via .go, .js, .ts file extensions.
• v5: Ruby (ActiveRecord) with scope detection via the OpenTelemetry::Instrumentation:: vendor prefix and language detection via the .rb extension.
• v6: PHP (Laravel/Eloquent, Symfony/Doctrine) with scope detection via the native io.opentelemetry.contrib.php.* scopes and language detection via the .php extension. The io.opentelemetry.contrib.php.doctrine scope is DB-specific, so it tags only DB findings, but io.opentelemetry.contrib.php.laravel is app-wide (it hooks the HTTP kernel, console, queue and Eloquent model), so it rides every Laravel finding. PhpLaravelEloquent therefore carries fixes for all 10 SQL and HTTP anti-patterns while PhpDoctrine carries only the SQL ones. Only this path is framework-aware: dd-trace-php bridged through the Collector datadogreceiver exposes no PHP code attributes (scope is a fixed Datadog), so those findings fall to PhpGeneric or stay unenriched.
• v7: the two messaging types (n_plus_one_messaging, slow_messaging), keyed by broker technology instead of framework. The remediation for a publish anti-pattern lives in the broker client's batching API (linger.ms, SendMessageBatch, a transacted JMS session), which the application framework does not name, so a second table MESSAGING_FIXES is keyed (FindingType, MessagingSystem). Detection reads the first token of the finding's pattern template, which carries the span's messaging.system value verbatim (see the normalization note in 02 · Normalization): no scope or code-attribute heuristics involved, and no span access. Kafka, RabbitMQ, SQS (aws_sqs plus the sqs shorthand), Pulsar, NATS and JMS are covered, activemq maps to the JMS advice since that is the client API involved. An unlisted system (rocketmq, servicebus, ...) keeps the generic suggestion.

New entries lean on the per-language *Generic tag when the recommendation is framework-agnostic, and reuse a framework-specific tag when the ecosystem ships a canonical primitive worth pointing at. The current state covers Java, C# (.NET 8 to 10), Python, Rust, Go, Node.js, Ruby and PHP across all 10 SQL and HTTP anti-patterns, each with a generic per-language fallback, plus the two messaging anti-patterns across six broker technologies.

The SuggestedFix struct

pub struct SuggestedFix {
    pub pattern: String,          // "n_plus_one_sql" mirrors parent finding.type
    pub framework: String,        // "java_jpa" or "java_generic"
    pub recommendation: String,   // short, imperative sentence
    pub reference_url: Option<String>,
}

Serialized in JSON as a nested object under finding.suggested_fix, skipped when absent. Emitted in SARIF under result.fixes[0].description.text (description-only form of the SARIF 2.1.0 fix object). The CLI renders it as a nested Suggested fix: line right after the generic Suggestion: line.

Framework detector

The detector is a pure function over fields already present on Finding (instrumentation_scopes, code_location, service), all populated at detection time from the span's OTel attributes. No span-level access, no extra allocations. It inspects five signals in order, most reliable first:

1. Instrumentation scope chain, captured at OTLP ingest from the originating span and its ancestors (e.g. io.opentelemetry.spring-data-3.0). Most reliable: the scope name is emitted by the agent regardless of how the user names their classes, so it survives user-code naming quirks. Vendor-specific scopes (io.quarkus.*, Microsoft.EntityFrameworkCore, the Ruby gem OpenTelemetry::Instrumentation::ActiveRecord, the PHP scopes io.opentelemetry.contrib.php.doctrine and io.opentelemetry.contrib.php.laravel) are checked before the standard io.opentelemetry.* / opentelemetry.instrumentation.* / @opentelemetry/instrumentation-* convention scopes. Go and Node are deliberately absent from the convention scope rules: their instrumentations use ecosystem-native scope names (gorm.io/plugin/opentelemetry, @prisma/instrumentation), and the - segment boundary used for Java version suffixes would false-positive on npm package names (pg vs instrumentation-pg-pool).
2. Language from ecosystem-native scope prefix. When the scope-chain check misses, the prefix still reveals the language (github.com/ = Go module path, @opentelemetry/instrumentation- or @prisma/ = npm, Microsoft.EntityFrameworkCore / OpenTelemetry.Instrumentation.* = NuGet, OpenTelemetry::Instrumentation:: = Ruby gem, io.opentelemetry.contrib.php. = PHP, then any other io.opentelemetry. scope = Java agent, such as io.opentelemetry.jdbc or io.opentelemetry.apache-httpclient-5.0). The PHP prefix is checked first so PHP keeps its scopes, and Python's opentelemetry.instrumentation. is not claimed. That language's namespace rules then run on code_location when present (a Spring Data *Repository namespace yields java_jpa), then the service-name rules of step 5 when they name a framework of that language (helidon-mp-orders yields java_helidon_mp), and the language generic applies otherwise, so even a span without code.filepath or code.namespace gets a language-appropriate suggestion.
3. code_location namespace with filepath-derived language (.java → Java, .cs → C#, .rs → Rust, .py → Python, .go → Go, .js/.ts → Node, .rb → Ruby, .php → PHP). Walks that language's rules in declared order; falls back to the language generic when no rule matches. PHP namespaces use \ separators, recognized by the same segment-boundary matcher as . and ::, and the ingest-time namespace derivation splits code.function.name on \ when it has no dot.
4. code_location namespace alone when filepath is absent: tries every language's rules in order and returns the first hit. No generic fallback in this path because the language cannot be known.
5. Service name as a last resort, only for framework names distinctive enough to avoid false positives in arbitrary service names (e.g. helidon in helidon-se-svc). Lowest confidence, only reached when all OTel signals are absent.

The structural findings (serialized_calls, excessive_fanout, chatty_service, pool_saturation) have no single originating span, so each carries the instrumentation_scopes and code_location of one representative call it already references: the first call of the serialized sequence, the first child of the fan-out, the first outbound HTTP call of the chatty trace, the first SQL span of the saturated service. The detector reads them like any other finding. Every surface that shows code_location (CLI Source:, HTML report source, SARIF locations[]) then points at that representative call, as it points at the first span of the group for an N+1 finding.

The namespace match is segment-boundary-aware on both sides: the hint must start at the namespace root or immediately after a separator and must end at the namespace end or immediately before another separator. Boundary characters are . (Java, C#) and :: (Rust). Examples:

• diesel:: matches diesel::query_dsl::FilterDsl and crate::diesel::reexport but not crate::mydiesel::query (leading boundary protects user code that embeds the hint).
• io.helidon matches io.helidon.webserver.Routing but not io.helidongrpc.Foo (trailing boundary protects against user packages whose first segment merely starts with the hint).
• Microsoft.EntityFrameworkCore matches Microsoft.EntityFrameworkCore.Query but not Microsoft.EntityFrameworkCoreCache.Provider.

Per-language rules

Order matters within a language: the first matching framework wins. JPA hints intentionally trail Quarkus reactive hints because org.hibernate.reactive contains org.hibernate.

Each hint is one of two kinds. Substring matches a boundary-delimited package segment (every rule below except where noted). LastSegmentEndsWith matches only the suffix of the namespace's last segment, for user-code conventions like Spring Data repositories where the framework package never appears in code.namespace (e.g. com.example.OrderRepository).

Java (JAVA_RULES):

JavaQuarkusReactive enumerates its reactive sub-packages explicitly. The catch-all io.quarkus belongs to JavaQuarkus (non-reactive), so any reactive Quarkus namespace must be matched by one of the more-specific reactive hints first. Helidon MP must come before Helidon SE because io.helidon.microprofile is a sub-package of io.helidon.

Note on Helidon MP and JPA: Helidon MP entities are JPA-managed under Hibernate. A typical OTel JDBC span on Helidon MP code carries code.namespace = jakarta.persistence.* or org.hibernate.*, which routes to JavaJpa (not JavaHelidonMp). The JavaHelidonMp rule fires when the span comes from Helidon MP plumbing itself (REST resources, CDI containers, MicroProfile Rest Client). For database findings on Helidon MP apps, the JavaJpa recommendation applies.

C# (CSHARP_RULES):

Rust (RUST_RULES):

Python (PYTHON_RULES):

Go (GO_RULES):

Node.js (JS_RULES):

Ruby (RUBY_RULES):

RUBY_RULES is empty: Ruby has no reliable namespace convention in code.namespace, so RubyActiveRecord is reached through the OpenTelemetry::Instrumentation::ActiveRecord vendor scope, and any other Ruby OTel scope (the pg/mysql2 drivers, Rack) or a .rb filepath routes to RubyGeneric.

PHP (PHP_RULES):

PHP frameworks are reached primarily through the vendor scopes io.opentelemetry.contrib.php.doctrine and io.opentelemetry.contrib.php.laravel. The namespace hints are the secondary signal: the Laravel SQL leaf span is PDO-scoped (code.function.name = "PDO::query") and exposes no app namespace, but Doctrine's own SQL span carries a Doctrine\DBAL\... namespace. Any other PHP OTel scope (pdo, mongodb, curl, guzzle) or a .php filepath routes to PhpGeneric.

Go and Node frameworks are reached through the namespace hints above and the language-from-scope-prefix fallback, never through SCOPE_RULES: their instrumentations emit ecosystem-native scope names (gorm.io/plugin/opentelemetry, @prisma/instrumentation) that the convention prefixes do not match. See the framework detector section above.

Mapping table

Two LazyLock<HashMap<_, SuggestedFix>> statics, and lookup_fix routes on the finding type before reading any framework signal. The protocol anti-patterns use FIXES, keyed (FindingType, Framework): a generic per-language fallback plus framework-specific entries. The two messaging anti-patterns use MESSAGING_FIXES, keyed (FindingType, MessagingSystem), so a messaging finding never reaches the framework table and vice versa. A FIXES lookup that misses the detected framework retries with that framework's language generic (JavaJpa to JavaGeneric, CsharpEfCore to CsharpGeneric), so a detected framework never yields less than the language advice. A miss on the generic too, or on MESSAGING_FIXES, leaves suggested_fix as None. Coverage is deliberately not a full language x pattern matrix. In particular, n_plus_one_sql and redundant_sql route mostly through framework-specific entries (a generic SQL N+1 fallback ships only for Java, Go, Node, Ruby and PHP), so a generic lookup for those patterns returns None for several languages. Representative anchors:

Extension path for contributors

To add a new framework:

1. Extend the private Framework enum in detect/suggestions/mod.rs.
2. Pick a language and append a (Framework, &[hint]) entry to that language's rule slice. Place more-specific frameworks before less-specific ones.
3. Add entries to the FIXES static for each (FindingType, Framework) pair you want to map.
4. Add unit tests under the tests module in the same file.

To add a new language:

1. Extend the Language enum and its rules() / generic() methods.
2. Add the file extension match in language_from_filepath.
3. Define a new *_RULES slice and a generic fallback variant on Framework.

No wiring changes elsewhere: the detect() orchestrator already calls suggestions::enrich at the end of the per-trace detection pass, build_cross_trace_finding calls it on each cross-trace slow finding, and the CLI / JSON / SARIF rendering already handle an optional suggested_fix.

Finding signatures and acknowledgments

acknowledgments.rs is the batch/CI half of the ack workflow. It loads .perf-sentinel-acknowledgments.toml, computes a signature per finding, moves acked findings into report.acknowledged_findings, and re-evaluates the quality gate on what survives. The daemon runtime store (daemon/ack.rs) shares the signature format and is unioned with the TOML at query time, with the TOML winning: it is the immutable baseline that went through PR review.

The signature is the load-bearing part, because it is what an operator's won't fix decision is pinned to. Its shape is <finding_type>:<service>:<sanitized_endpoint>:<sha256-prefix-of-template>.

• Why a hash only on the template. The (finding_type, service, sanitized_endpoint) triple is already in the signature, so the hash only disambiguates templates within one triple, a tiny population. Its 32 hex characters (128 bits) are therefore not collision resistance for its own sake, they are defense in depth against an ack silently masking a different finding after a SQL refactor or a service rename.
• Why / and space become _. So : stays a single unambiguous separator that operators can cut -d: in a shell pipeline.
• Why BiDi and invisible characters are stripped from service and source_endpoint (Trojan Source, CVE-2021-42574): two signatures that render identically must not map to distinct ack entries, or an ack becomes unverifiable by reading it.

Stability is a contract, not an implementation detail. Any change to the format, the sanitization, or the hash width silently invalidates every ack file in the wild, and the failure is silent in the worst direction: findings the operator accepted reappear, or worse, a stale ack keeps matching something else. A dedicated test suite pins the format for this reason. Treat a signature change as a breaking change requiring a re-ack, and say so in the changelog.

Re-evaluating the gate is the point. Filtering findings without re-running quality_gate would leave analyze --ci failing on findings the operator explicitly accepted, which is the entire semantics of "won't fix". The re-evaluation runs even when nothing matched, so the gate field is always consistent with the final findings slice rather than with a pre-filter snapshot. apply also clears acknowledged_findings first, so feeding a Report back through it (a baseline JSON round-trip) cannot accumulate stale pairs.

Expiry is deliberately fail-open on the finding, fail-loud on the file. An ack whose expires_at has passed is inactive and its finding comes back. A malformed date, on the other hand, aborts the run: a typo must not silently widen the acked set.