perf-sentinel CI guide

CI-side integration: run perf-sentinel in batch mode against a trace fixture produced by your integration test stage, and surface the findings on every pull request. For topology overviews see Integration, for application-side instrumentation see Instrumentation.

Contents

• CI mode (batch analysis): the underlying CLI invocation and exit-code semantics behind every recipe below.
• CI integration recipes: copy-pasteable templates for GitHub Actions, GitLab CI and Jenkins, plus the quality-gate philosophy and the interactive HTML report path for each provider.
• PR regression detection (diff subcommand): compare a PR trace set against a baseline trace set to flag regressions.

CI mode (batch analysis)

For CI pipelines, use batch mode instead of daemon mode:

perf-sentinel analyze --ci --input traces.json

Producing traces.json

Batch mode needs a trace file, and how a test suite hands one over depends entirely on the language. Only C++ and PHP implement an OTLP exporter that writes to a path you choose, and a forked Maven test JVM cannot even yield its stdout, which Surefire uses as its command channel. The portable answer is to let the application export over the network, as it does in production, and to listen:

perf-sentinel capture --output traces.json -- ./scripts/run-integration-tests.sh
perf-sentinel analyze --ci --input traces.json

The test suite only needs the standard OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317. Prefix the test step you already have rather than adding a stage next to it, or the integration suite runs twice. When the step cannot be prefixed, run capture in the background and stop it with a signal once the tests finish. See CLI reference for both shapes and Instrumentation for the per-language setup, including the Collector alternative.

Exit code is non-zero if the quality gate (a configurable set of pass/fail thresholds, the same idea as a SonarQube quality gate or a coverage gate) fails. Configure thresholds in .perf-sentinel.toml:

[thresholds]
n_plus_one_sql_critical_max = 0
n_plus_one_http_warning_max = 3
n_plus_one_messaging_warning_max = 3
io_waste_ratio_max = 0.30

Exit codes

The batch subcommands (analyze, report, diff, tempo, jaeger-query, pg-stat, mysql-stat, calibrate, explain, bench, demo) share a stable exit-code contract since 0.9.17:

• 0: success. Under --ci, this also means the quality gate passed.
• 1: quality gate FAILED. Only emitted by analyze --ci (or tempo --ci / jaeger-query --ci, which share the same gate path via emit_report_and_gate) when a threshold in [thresholds] was exceeded. The analysis itself succeeded, this is a genuine regression. A gate breach takes precedence over a simultaneous report-write failure, so a real regression on a broken pipe or a full disk still exits 1, never the tolerable 75. Every other batch command has no --ci flag and no quality gate at all, none of them ever emit 1.
• 2: a CLI usage error. Emitted both by clap for parse-level mistakes (a missing required flag, e.g. mysql-stat with no --input) and by perf-sentinel's own post-parse validation for unsupported flag combinations clap cannot express (e.g. report --pg-stat-top without --pg-stat, or bench --iterations 0). A usage error is a permanent invocation mistake and always blocks, deliberately kept out of the tolerable 75 bucket.
• 75: tooling/internal error (aligned with EX_TEMPFAIL, sysexits.h, the sentinel value the GitLab CI template already uses at the shell level). Covers every runtime failure that reaches perf-sentinel's own code and is neither a usage error nor a quality-gate breach: a missing or unreadable --input/--config/acknowledgments/baseline file, malformed trace/config/acknowledgments data, a tempo/jaeger-query fetch failure, an explain trace-not-found, or a failure writing the SARIF/JSON/HTML output. Never emitted for a threshold breach, and never means the analysis ran and disagreed with your config.

The two failure codes above the clap floor are deliberately distinct so a CI pipeline can branch on the exact code instead of inferring the cause from file existence or step outcome, see Tooling failures vs quality-gate breaches below for how each of the three official templates uses this. Before 0.9.17, tooling failures exited 1 too. Pipelines that only check for a non-zero exit code are unaffected.

CI integration recipes

Ready-to-copy templates for the three major CI providers live in docs/ci-templates/. Pick the one that matches your provider, drop it into your repository, adapt the three variables called out in the template's leading comment block (version pin, trace path, config path) and you are done.

The "What it surfaces" column below references three CI-side formats: SARIF (Static Analysis Results Interchange Format, the OASIS-standard JSON schema GitHub and GitLab use for inline PR annotations, spec), GitHub Code Scanning (the surface where GitHub renders SARIF findings on PRs, formerly the "Security" tab), and Warnings Next Generation (a Jenkins plugin that aggregates static-analysis findings across plugins into a unified issue tree and a trend chart, project).

Quality-gate philosophy

All three templates run perf-sentinel analyze --ci as the gating step. The --ci flag exits with code 1 when any threshold in [thresholds] is breached. The templates translate that exit code differently based on the trigger:

This split avoids the common failure mode where PR-gates that also enforce on trunk leave main red, the team works around it, and the tool gets disabled.

The recommended setup produces the report once per job, without --ci (SARIF + JSON, always available for reviewer inspection), then decides pass/fail separately. Jenkins and GitLab CI do that by re-running perf-sentinel analyze --ci a second time and reading its exit code. GitHub Actions instead reads quality_gate.passed straight from the JSON report already on disk, since the gate result is computed on every run regardless of --ci, only the exit code differs. Either way, the gate decision only ever runs once the report-only pass has already succeeded.

Per-provider PR-vs-trunk wiring:

• GitHub Actions: PR step runs when github.event_name == 'pull_request' and calls exit 1 on breach, trunk step emits a ::warning:: annotation without failing.
• GitLab CI: allow_failure: true on the $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH rule. The job still returns exit 1 on breach, the pipeline badge stays green, the job shows a yellow warning icon.
• Jenkins: when { expression { env.CHANGE_ID != null } } on the Quality gate (PR only) stage. CHANGE_ID is populated by MultiBranch Pipeline only on PRs, so branch builds skip the stage. The Warnings NG qualityGates follows the same guard so the post block does not reintroduce blocking on trunk.

Tooling failures vs quality-gate breaches

A --ci exit code of 1 is ambiguous on its own: it can mean a genuine threshold breach, or it can mean perf-sentinel never actually ran (a blocked download, a corrupted release, a crash on malformed traces). Treating both the same way is worse than it sounds: a flaky network blip on a Friday afternoon should not block every PR in the repo until someone notices and re-runs CI. All three templates isolate the two failure modes so only a genuine breach can turn a PR red:

• GitHub Actions: the download step tolerates failure (continue-on-error: true), but the checksum-verification step right after it does not, a tampered or corrupted release must always fail the job, never get folded into the tooling-tolerant bucket. The report-only analyze step also carries continue-on-error: true. Every downstream step (SARIF upload, PR comment, the two gate steps) checks steps.analyze.outcome == 'success' rather than file existence: shell > redirection creates its target file before the command runs, so a crashed analyze would still leave an empty findings.sarif behind and defeat a hashFiles() check. The analyze step also writes through a .tmp path and renames on success, a second, independent guard against that same trap. A final Report tooling failure step emits a ::warning:: when analyze did not succeed, so a tooling problem stays visible instead of being silently swallowed.
• GitLab CI: every download command explicitly exits 75 (EX_TEMPFAIL, sysexits.h) instead of propagating whatever exit code the underlying tool produced. allow_failure: exit_codes: [75] on the merge-request rule excludes only that specific code from blocking the merge. Checksum verification and the Code Quality jq conversion are deliberately excluded from that exit-75 convention: a checksum mismatch means a tampered release, and a jq failure means a bug in the conversion filter, neither is a tooling blip that should be tolerated. The final --ci re-run keeps its own exit code (normally 1 on a real breach), which still blocks as before.
• Jenkins: the download half of the Install perf-sentinel stage is wrapped in catchError(buildResult: 'UNSTABLE', stageResult: 'UNSTABLE'). Checksum verification and install run unwrapped right after it, so a bad checksum always fails the build. The perf-sentinel analyze stage writes its SARIF/JSON through a .tmp path and renames on success for the same reason as GitHub Actions above, otherwise fileExists() could not tell a crash from a real report. The Quality gate (PR only) stage adds a fileExists('perf-sentinel-results.sarif') condition alongside the existing CHANGE_ID check, so it only runs a real threshold check once the report-only stage has actually produced a SARIF.

In all three cases, a tooling failure now surfaces as a visible warning or an unstable/yellow build, clearly distinct from the red build a real breach produces, and it never blocks a merge or a push to trunk on its own. A checksum or conversion-logic failure, in contrast, always blocks, in every trigger context, because it is not the kind of failure this isolation is meant to tolerate.

Interactive report via GitHub Pages

The sticky PR comment (markdown block with finding counts and quality gate status) gives reviewers an at-a-glance view. For a deeper inspection (span tree with highlighted N+1s, framework-specific suggested fixes, pg_stat drill-down, full Diff against trunk), the GitHub Actions template optionally publishes a full HTML dashboard to GitHub Pages on every PR, linked from the sticky comment as:

📊 Interactive report (Diff view) → https://<owner>.github.io/<repo>/perf-sentinel-reports/pr-<N>/index.html#diff

Clicking the link opens the report on the Diff tab, which is the natural view for a reviewer: new findings introduced by the PR, resolved findings (regressions fixed), severity changes, and endpoint-level I/O metric deltas. The other tabs (Findings, Explain, pg_stat, Correlations, GreenOps) are one click away via the tab strip.

The reports are self-contained single-file HTML with deep-link hash routing, so sharing a specific finding is as simple as copying the URL from the address bar.

GitHub Pages tier requirement. On a personal GitHub Free account, Pages is only available for public repositories. Private repositories need GitHub Pro, Team, or Enterprise Cloud. See GitHub's plans for the current list. If you try to enable Pages on a private repo with a Free account, the branch push succeeds but Pages serves 404 permanently with no error in the Actions log. Either upgrade the account, make the repository public, or skip the Pages block and stay on the SARIF + markdown sticky comment mode.

Setup (opt-in, requires GitHub Pages on the repository):

1. Create an empty gh-pages branch in the repo (one-time, standard GitHub Pages bootstrap).
2. Enable GitHub Pages in Settings -> Pages, source = gh-pages branch, folder = / (root).
3. Copy the companion baseline workflow from docs/ci-templates/github-actions-baseline.yml to .github/workflows/perf-sentinel-baseline.yml. It runs on every push to main and stores the baseline report under gh-pages/perf-sentinel-reports/baseline.json.
4. Copy the cleanup workflow from docs/ci-templates/github-actions-report-cleanup.yml to .github/workflows/perf-sentinel-report-cleanup.yml. It runs on PR close and removes the per-PR directory.
5. Uncomment the Download baseline from gh-pages, Generate interactive HTML report, Checkout gh-pages worktree and Publish report to gh-pages blocks in your main workflow (the header comment in docs/ci-templates/github-actions.yml locates them).
6. In that same main workflow, raise contents: read to contents: write in the permissions: block. The publish step pushes the HTML report to the gh-pages branch, which a read-only GITHUB_TOKEN cannot do (the push fails with a 403). The baseline and cleanup workflows already declare contents: write, so only the main workflow needs the change.

Once the three workflows are in place, every PR gets its own interactive report at a stable URL:

https://<owner>.github.io/<repo>/perf-sentinel-reports/pr-<N>/

The baseline is refreshed on every push to main, so the Diff tab always compares the PR's traces against the latest merged state.

Two properties follow from that flow. The first trunk run is the seeding event: until then the fetch 404s and the report renders without a Diff tab, and a brand-new scenario likewise reads as New until its first trunk run. And nothing is carried between two pushes of the same pull request: every run compares against the trunk baseline, so the Diff shows the whole delta the pull request introduces, not the delta of its last commit.

The baseline workflow ends on analyze --ci, after the publish. The two steps answer different questions:

• the Diff shows what a pull request changed. A finding on both sides sits in neither column, so no one is blamed for an inherited regression, and no one is alerted by it either.
• the gate checks whether a state is acceptable, on absolute thresholds, with no baseline.

Without a gate on the trunk, a merged regression is invisible in every Diff and surfaces as the failing gate of the next unrelated pull request that exercises the same endpoint. Gating the merged state moves that alarm to the merge that caused it. The publish still runs first: a red trunk must keep refreshing the baseline, or every following pull request loses its Diff tab on top of being blocked.

While the fix is in flight, acknowledge the finding with an expires_at, which unblocks the queue without touching a threshold. Set its service and source_endpoint too, so the trunk run reports the entry as removable once the fix lands. See Acknowledgments.

If GitHub Pages is not enabled, the template falls back to the markdown-only sticky comment. No behaviour change for existing adopters.

Fork PR limitations. The Post PR comment step is marked continue-on-error: true because fork PRs receive a read-only GITHUB_TOKEN regardless of the workflow's permissions: block. Without the tolerance, every fork PR would turn the CI red at the sticky-comment step even when the rest of the pipeline succeeded. With the tolerance in place, fork PRs still upload SARIF findings to the Security tab and the Checks UI shows the quality gate result, but no sticky comment appears on the PR conversation. Same-repo PRs (internal contributors, same org) keep the full experience, sticky comment included. Projects where the sticky comment on fork PRs is a hard requirement should migrate to the pull_request_target + workflow_run split documented by GitHub Security Lab. That pattern splits the pipeline into a read-only workflow that builds and uploads artifacts and a write-enabled workflow triggered by workflow_run that downloads those artifacts and posts the comment. It is not the default in this template because it doubles the YAML surface and needs careful artifact passing, not proportional for a getting-started template. The Publish report to gh-pages step is guarded the same way (it runs only when github.event.pull_request.head.repo.full_name == github.repository), so a fork PR never fails on a push the read-only token could not make.

Concurrency trade-off. The concurrency.group: gh-pages-deploy guard serializes runs of this workflow against the baseline and cleanup workflows, so three PRs closed in the same minute cannot race each other on gh-pages. Because the guard is declared at workflow scope, it also serializes runs that would not touch Pages (for example when the Pages blocks are commented out). Repositories with heavy PR throughput can split the Pages-related steps into a dedicated job and narrow the concurrency to that job only. Skipped here to keep the template compact.

Dependencies. The deploy uses plain git against the gh-pages branch, authenticated with the built-in GITHUB_TOKEN and a contents: write permission. The baseline and cleanup workflows declare it out of the box. The main workflow ships with contents: read and you raise it to write when enabling the publish blocks (step 6 above). No third-party deploy action is required, which keeps the template free of supply-chain surface for the upload path. Only actions/checkout (pinned) is reused across all three workflows.

Storage footprint. A report starts around 450 KB whatever it contains, since the embedded fonts and logos that make the file self-contained are carried even by a report with nothing to show. It grows from there with the number of findings, up to a 5 MiB ceiling where the sink starts trimming. Size a quota on that ceiling, not on the floor. With retention handled by the cleanup workflow, the gh-pages branch only carries reports for open PRs plus the single baseline.json. No unbounded growth.

Other providers. See "Interactive report via GitLab Pages" and "Interactive report via Jenkins HTML Publisher" below.

Interactive report via GitLab Pages

Equivalent to the GitHub Pages path above, adapted to GitLab's native deployment surface. Two template blocks are provided in docs/ci-templates/gitlab-ci.yml, pick the one matching your GitLab tier.

Tier note. The per-MR deployment mode (pages.path_prefix) is documented as Experiment, Tier: Premium or Ultimate, and is not available on gitlab.com Free. On Free, the MR deployment appears successful in the environments list but is not actually served. A Free-tier compatible fallback is provided alongside.

Pick either block, not both (they would fight over the root deployment).

Setup (opt-in, requires GitLab Pages enabled on the project):

1. Enable GitLab Pages under Settings -> Pages if not already on.
2. Uncomment exactly one block in docs/ci-templates/gitlab-ci.yml. Both run in the perf-sentinel stage and reuse PERF_SENTINEL_VERSION / PERF_SENTINEL_TRACES / PERF_SENTINEL_CONFIG already declared for the main job.
3. For perf-sentinel-pages, confirm GitLab 17.9 or later. Not required for perf-sentinel-pages-simple.

Behavior of perf-sentinel-pages (Premium or Ultimate). The job differentiates two trigger paths via its rules: block:

• On merge request ($CI_PIPELINE_SOURCE == "merge_request_event"), fetches the trunk baseline from the project Pages root (strips the MR prefix from CI_PAGES_URL via ${CI_PAGES_URL%/mr-[0-9]*}, silent 404 fallback when absent), produces public/index.html via perf-sentinel report --output public/index.html, deploys with path_prefix: "mr-${CI_MERGE_REQUEST_IID}" and pages.expire_in: 30 days. environment.url points to the active ${CI_PAGES_URL}, which GitLab resolves to the MR-scoped deployment URL at runtime.
• On push to the default branch, produces public/perf-sentinel-reports/baseline.json via perf-sentinel analyze --format json, deploys with an empty path_prefix so the file lands at the site root and future MR deployments can fetch it.

Behavior of perf-sentinel-pages-simple (Free). Runs only on the default branch. Writes both public/index.html (the interactive trunk snapshot) and public/perf-sentinel-reports/baseline.json in one pass, then deploys a single Pages site at the project root.

Retention. perf-sentinel-pages delegates retention to GitLab. Parallel deployments are deleted immediately when the MR is closed or merged. The pages.expire_in: 30 days on the template is a backstop for stale-open MRs (GitLab's default is 24 hours when unset, which we widen so a long-running MR keeps its live report). Setting expire_in: never disables time-based expiry entirely and relies on close/merge events only. Use never only if your team reliably closes or merges MRs, otherwise abandoned MRs accumulate until the quota cap kicks in. perf-sentinel-pages-simple has no retention concern, it keeps a single deployment that is overwritten on every default-branch push.

Quota. gitlab.com allows up to 100 additional parallel deployments on Premium and 500 on Ultimate, tracked per namespace on top of the main deployment. Self-managed instances expose the limit through admin configuration. perf-sentinel-pages-simple is a single deployment, not subject to this cap. For projects running near the cap on perf-sentinel-pages, expire_in can be lowered or MRs should be closed/merged promptly to release slots.

Storage footprint. A report starts around 450 KB of embedded fonts and logos and grows with the findings up to a 5 MiB trim ceiling, and a baseline JSON is 10 to 50 KB. With retention active on the Premium path, only open MRs plus the current baseline consume space. The Free path stores a single deployment.

Dependencies. No third-party GitLab CI component. The job uses curl to install the pinned perf-sentinel release binary and the built-in pages: keyword for deployment. No deploy token or runner token beyond the default CI_JOB_TOKEN is required.

Interactive report via Jenkins HTML Publisher

Equivalent to the GitHub and GitLab paths above, adapted to the HTML Publisher plugin that is pre-installed on most enterprise Jenkins. The plugin exposes the report at a stable URL ${BUILD_URL}perf-sentinel/ and adds a "perf-sentinel" link in the build sidebar, next to the Warnings NG report already configured by the template.

Opening that link drops the reviewer into the Findings tab (the default landing when no baseline is wired, see the Diff tab note below). The five other tabs (Explain, pg_stat, Correlations, GreenOps, and a greyed-out Diff tab) are one click away via the tab strip.

Jenkins pipeline requirements:

• Use a MultiBranch Pipeline with a branch-source plugin installed (GitHub Branch Source, Bitbucket Branch Source, GitLab Branch Source, or Gitea Branch Source). The env.CHANGE_ID check that gates the quality-gate stage on PR builds is only set by these plugins. Inside a classic single-branch Pipeline, CHANGE_ID is always null and the quality gate never blocks.
• Use a Linux agent (or a controller without agents on a Linux host). The template relies on sh, curl, sha256sum, chmod, none of which are available on Windows agents by default.

Setup (opt-in, requires the HTML Publisher plugin on the controller):

1. Confirm the HTML Publisher plugin (>= 1.10 for CSP compatibility) is installed. Manage Jenkins -> Plugins -> Installed plugins, search for "HTML Publisher". If missing, install and restart the controller. The Warnings Next Generation plugin used by the rest of the template needs to be at >= 9.11.0 for the SARIF tool.
2. Uncomment the Generate interactive HTML report stage in docs/ci-templates/jenkinsfile.groovy, placed right before the Quality gate (PR only) stage.
3. Uncomment the publishHTML([...]) block in the post { always } section of the same file. It is paired with the stage above so both need to be enabled together for the link to appear.

Once enabled, every build (branch or pull request) produces a report available at ${JENKINS_URL}/job/<job-name>/<build-number>/perf-sentinel/. The build sidebar carries a "perf-sentinel" link that always points to the newest build's report via alwaysLinkToLastBuild: true. The keepAll: true option retains per-build reports so historical builds remain browsable.

If the report renders unstyled with broken tab navigation, see Configuring Jenkins to render the interactive report below. Jenkins applies a strict default Content Security Policy that blocks inline CSS and JavaScript, which is the most common cause of an unstyled perf-sentinel sidebar page.

Configuring Jenkins to render the interactive report.

Jenkins applies a strict Content Security Policy by default to content served from build workspaces. The perf-sentinel HTML report packs CSS and JavaScript inline in a single self-contained file, which the default CSP blocks. Without relaxing the policy or using a Resource Root URL, clicking the ${BUILD_URL}perf-sentinel/ sidebar link shows an unstyled HTML page with broken tab navigation and no message in the build log.

Two options to fix, in order of preference:

Option A: configure a Resource Root URL (Jenkins 2.200+, recommended). Serves user-generated content from a separate domain so the main instance CSP no longer applies. Set the URL in Manage Jenkins > System > Resource Root URL. See the inline help for details. No template change required, all reports across all jobs benefit immediately.

Option B: relax the CSP (legacy, broader scope). Set the following Java system property on the Jenkins controller startup (or run it once via the Script Console for a session-scoped experiment):

System.setProperty(
    "hudson.model.DirectoryBrowserSupport.CSP",
    "sandbox allow-scripts; default-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline';"
)

Tradeoffs:

• Affects all HTML content served by all jobs on the instance, not just perf-sentinel reports.
• Adds 'unsafe-inline' for both styles and scripts. Acceptable on a Jenkins instance where you trust the jobs being run, risky on a multi-tenant instance with untrusted contributors.
• Reverts to default on Jenkins restart unless persisted via the startup options (JAVA_OPTS, jenkins.xml, or systemd unit).

Splitting the report into sibling CSS and JavaScript files would not help, and no release will do it for this reason. The default Jenkins policy blocks scripts twice over: it declares no script-src at all, so an external one falls back to default-src 'none', and its sandbox directive omits allow-scripts, which turns scripting off for the document whatever the source. Measured on a control page under that exact policy: neither an inline nor a sibling script runs, while a sibling stylesheet and a same-origin image do load.

Moving the content into the static DOM would not be enough either. Inline <style> is blocked the same way (style-src 'self' carries no 'unsafe-inline'), and the data: URI fonts and logos fall under default-src 'none'. A report that renders on this policy is a multi-file artifact with no script, no inline style and no embedded asset, which is a second output format rather than a fix to the current one. Options A and B stay the two answers.

What the report does carry since 0.9.25 is a notice at the top of the page, in plain unstyled text, saying that its content is built by script, that a restrictive policy is the usual reason it did not run, and pointing here. A script placed right after it removes it during parsing, so a normal load never shows it. A page that explains itself is not a rendered dashboard, it only replaces the blank page that sent the first reporter looking through build logs.

The constraint is specific to Jenkins. GitHub Pages and GitLab Pages serve the report with no policy of their own, and the two paths above render it as a browser opening the file locally would, tab navigation included. Nothing to configure on either.

Diff tab via the Copy Artifact plugin. Unlike GitHub Actions and GitLab CI where a companion baseline workflow refreshes baseline.json on every push to the default branch, Jenkins has no built-in artifact store to publish a trunk baseline to. The template's baseBranchJob() and fetchBaseline() helper functions (top of docs/ci-templates/jenkinsfile.groovy) use the Copy Artifact plugin instead, pulling perf-sentinel-report.json straight from a previous build rather than from a separate published artifact. Following the same "compare against what you are merging into" model as SonarQube's new-code period. On a PR build (env.CHANGE_TARGET set by MultiBranch Pipeline) the baseline is the last successful build of the target branch's job. Outside a PR (no CHANGE_TARGET to resolve, and this stage runs without a git checkout so the base cannot be inferred any other way) it falls back to this job's own last successful build. Both lookups are best-effort (optional: true): a job that has never built successfully, or a first-ever build with no history at all, simply renders without the Diff tab, same as leaving the enhancement disabled. Enable it by uncommenting the Generate interactive HTML report stage, the helper functions are already wired in.

No PR comment posting. Jenkins does not have a native pull-request comment mechanism equivalent to GitHub's sticky comment or GitLab's Code Quality widget. Reviewers who follow a Jenkins build consult the build page directly, same pattern as for Warnings NG findings. Teams who want a PR comment can wire the gh CLI or a forge-specific REST API from within the pipeline, but that requires managing a forge token in Jenkins credentials and is out of scope for this template.

Storage footprint is per-build and retained indefinitely (keepAll: true). A report starts around 450 KB of embedded fonts and logos and grows with the findings up to a 5 MiB trim ceiling. For long-lived Jenkins controllers with high build volume, pair publishHTML keepAll: true with the build discarder in the job configuration (e.g. keep last N builds) to cap the footprint.

Where SARIF surfaces in each provider

• GitHub Code Scanning lists each finding under the Security tab of the repository, with inline source annotations on the PR diff when the code_location field is present. Requires permissions.security-events: write on the workflow.
• GitLab Code Quality widget shows up on the merge request page, with severity colors derived from the perf-sentinel severity field (critical -> critical, warning -> major, info -> info).
• Jenkins Warnings Next Generation publishes a structured issue tree with a trend chart per build. The plugin natively understands SARIF v2.1.0 and supports its own qualityGates declaration as a defense in depth on top of the perf-sentinel --ci exit code.

PR regression detection (diff subcommand)

The diff subcommand compares two trace sets and emits a delta report listing new findings, resolved findings, severity changes and per-endpoint I/O op count deltas. The natural fit is a PR check that compares the PR branch's traces against the base branch's traces.

The comparison is a set difference over finding identities, not a comparison of test runs. Nothing tells it which scenarios each side executed, so what a column means depends on what the two suites happened to cover:

Three things follow:

• A finding on both sides sits in neither column, so a pull request is never blamed for an inherited regression.
• Coverage asymmetry only flatters. A scenario renamed, dropped or not run lands its findings in Resolved with no code change behind it.
• An acked finding is filtered from both sides, so its real fix resolves nothing here. The unmatched_acknowledgment warning is that signal, see Acknowledgments.

Upgrade note (0.9.22): finding identity is keyed on (type, service, source_endpoint, template), and source_endpoint now resolves entry points that previously reported unknown (see Acknowledgments). Whether that churns your first post-upgrade comparison depends on what the baseline is. A baseline that persists findings, such as the report --before baseline.json gh-pages flow below, shows each moved finding once as resolved and once as new, with no application change behind it: re-capture it against 0.9.22 first. A baseline that is a trace corpus fed to diff --before sees no churn at all, both sides are re-analyzed by the current binary.

# .github/workflows/perf-sentinel-diff.yml
name: perf-sentinel diff

on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  pull-requests: write

jobs:
  diff:
    runs-on: ubuntu-latest
    env:
      PERF_SENTINEL_VERSION: "0.11.1"
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
        with:
          fetch-depth: 0

      - name: Install perf-sentinel
        run: |
          set -euo pipefail
          BASE_URL="https://github.com/robintra/perf-sentinel/releases/download/v${PERF_SENTINEL_VERSION}"
          curl -sSLf -o perf-sentinel-linux-amd64 "${BASE_URL}/perf-sentinel-linux-amd64"
          curl -sSLf -o SHA256SUMS.txt            "${BASE_URL}/SHA256SUMS.txt"
          grep 'perf-sentinel-linux-amd64' SHA256SUMS.txt | sha256sum -c -
          mkdir -p "${GITHUB_WORKSPACE}/bin"
          install -m 0755 perf-sentinel-linux-amd64 "${GITHUB_WORKSPACE}/bin/perf-sentinel"
          echo "${GITHUB_WORKSPACE}/bin" >> "${GITHUB_PATH}"

      # Run integration tests on the PR branch and capture traces. Your
      # script owns the trace file, it takes the output path as an argument
      # here. See Instrumentation for how each language produces one,
      # Java in particular has no file exporter and needs a stdout capture.
      - name: Collect PR-branch traces
        run: ./scripts/run-integration-tests.sh pr-traces.json

      # Re-run on the base branch.
      - name: Collect base-branch traces
        run: |
          git checkout ${{ github.event.pull_request.base.sha }} -- .
          ./scripts/run-integration-tests.sh base-traces.json

      - name: Diff
        run: |
          perf-sentinel diff \
            --before base-traces.json \
            --after pr-traces.json \
            --config .perf-sentinel.toml \
            --format json \
            --output diff.json
          # SARIF for GitHub Code Scanning (only new findings).
          perf-sentinel diff \
            --before base-traces.json \
            --after pr-traces.json \
            --config .perf-sentinel.toml \
            --format sarif \
            --output diff.sarif

      - name: Upload SARIF
        if: hashFiles('diff.sarif') != ''
        uses: github/codeql-action/upload-sarif@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v4.35.2
        with:
          sarif_file: diff.sarif
          category: perf-sentinel-diff

      - name: Comment regression summary on PR
        run: |
          NEW=$(jq '.new_findings | length' diff.json)
          RESOLVED=$(jq '.resolved_findings | length' diff.json)
          REGRESSIONS=$(jq '[.severity_changes[] | select(.after_severity == "critical" or (.after_severity == "warning" and .before_severity == "info"))] | length' diff.json)
          {
            echo "## perf-sentinel diff vs base"
            echo
            echo "- $NEW new finding(s)"
            echo "- $RESOLVED resolved finding(s)"
            echo "- $REGRESSIONS severity regression(s)"
          } > pr-comment.md

      - uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
        with:
          header: perf-sentinel-diff
          path: pr-comment.md

      - name: Fail on regression
        run: |
          NEW=$(jq '.new_findings | length' diff.json)
          REGRESSIONS=$(jq '[.severity_changes[] | select(.after_severity == "critical")] | length' diff.json)
          if [ "$NEW" -gt 0 ] || [ "$REGRESSIONS" -gt 0 ]; then
            echo "::error::diff introduces $NEW new finding(s) and $REGRESSIONS critical regression(s)"
            exit 1
          fi

Tweak the threshold logic in the final step to match your team's policy. Some teams gate on any new finding, others tolerate Info-level new findings and only fail on Warning or Critical regressions.