GitHub Checks - MCPJam Inspector Documentation Index Fetch the complete documentation index at: /llms.txt Use this file to discover all available pages before exploring further. Skip to main content MCPJam Inspector home page Search... ⌘ K * Discord * GitHub * Website * MCPJam * Getting Started * Installation * Hosted App * Slack * Okta SSO & SCIM * GitHub Checks * Common MCP Errors * Error code reference * Changelog * GitHub * Get Started Inspector CLI SDK API MCPJam Inspector home page Search... ⌘ K Ask Assistant ⌘ I * GitHub Search... Navigation GitHub Checks Get Started Inspector CLI SDK API GitHub Checks Copy page Copy page Run an eval suite against the MCP server built from every pull request, and read the result as a check run, a pull-request comment, and a machine-readable JSON block. Copy page Copy page Connect a repository and MCPJam runs the eval suite bound to it against the MCP server built from each pull request’s head commit , then reports back as a check run named MCPJam Evals . Nothing about the check reads your default branch. It builds the pull request, starts the server it produces, runs the suite against that server, and reports what it measured. GitHub Checks is configured in the hosted app, at Settings → Integrations → GitHub . Every write on this page — connecting, installing, disconnecting, and changing per-repository settings — requires an organization owner or admin. Members can open the page to see the current setup, but all controls are disabled and a note explains why. Connect a GitHub account starts it. You sign in to GitHub so we can confirm which accounts you administer, then pick one from the list that comes back — an organization, or your own account. Installing the app is not on its own proof that an installation is yours to connect, which is why the sign-in step is there and not skippable. Accounts that already have the app appear in that list ready to connect. Install on another account , in that same list, is not generally available yet. During the beta the MCPJam app can only be installed on the MCPJam organization, so there is no other account for it to reach — following it sends you to the existing installation rather than offering a choice. Even once that opens up, GitHub sends an administrator who already has the app somewhere to that installation instead of asking which account to use. So for now, connect an account that already appears in the list. A GitHub account connects to one MCPJam organization at a time , for the whole account rather than per repository. Two organizations cannot split the repositories of one GitHub account between them. If the account is already connected elsewhere, connecting it again is refused — free it up in the organization holding it, or use a different account. Which organization is holding it is rolling out : until it reaches your deployment the refusal names nothing, so ask an owner of the GitHub account, who can see the installation. After it reaches you the account picker marks the account before you click, and names the holding organization when you are a member of it — when you are not, it still cannot name it, because that is a fact about an organization you cannot see. The pull-request comment described below is rolling out: it is enabled per deployment by an operator, and until it is on for yours a connected repository gets the check run and no comment. The per-repository toggle is visible before then and records your choice; nothing is posted until the rollout reaches you. ​ mcpjam.yaml MCPJam can usually work out how to build and start a server on its own. When it cannot — or when you would rather say it explicitly than let detection guess — commit an mcpjam.yaml at the root of the repository . A committed file is authoritative: MCPJam runs what it says and never falls back to detection. A file that is present but invalid fails the check rather than being ignored, because ignoring it would run a server the author never declared. mcpjam.yaml version : 1 checks : build : npm ci && npm run build start : node dist/server.js port : 8080 path : /mcp env : LOG_LEVEL : debug ​ Fields Field Required What it is version Yes Must be 1 . checks.build Yes The command that installs and compiles. An install-only ecosystem still declares one — npm ci , uv sync --frozen . checks.start Yes The command that starts the server. It must keep running. checks.port Yes An integer, 1–65535. The port your server listens on. checks.path Yes The MCP endpoint path. Must start with / . checks.transport No Only streamable-http , which is also the default. Version 1 of this file is HTTP-only. checks.env No Environment variables to set before start . Unknown keys inside checks: are rejected — inside the authoritative section a stray key is a typo that would otherwise silently change what runs. Unknown keys at the top level are ignored, so the file can grow new sections without breaking older checks. The file itself is capped at 32 KB. ​ env checks.env is a flat mapping of at most 20 entries. Names must be uppercase environment-variable names ( ^[A-Z][A-Z0-9_]*$ , up to 64 characters) and values must be strings of up to 1024 characters — quote anything that looks like a number, a boolean, or an empty value, or the file is rejected rather than coerced. Never put a credential in checks.env . This file is committed to the repository, so it is readable by anyone who can read the repository, and MCPJam stores its values verbatim alongside the check. It is for the flags a server needs in order to boot — LOG_LEVEL , FIXTURE_MODE — and for nothing that would matter if it were printed in public. ​ Binding and ports The port is what has to match; the bind address is not. MCPJam reaches your server through the sandbox’s own host bridge, from inside the box, so a server bound to loopback is reachable. What fails is listening on a port other than the one checks.port declares, or a process that exits after startup — both are reported as an unhealthy server, and the check body says so. If your framework needs a HOST (or equivalent) variable set to bind the way you want, set it in checks.env like any other boot flag. Only add it if your framework actually needs it — it is not something MCPJam requires. ​ The pull-request comment Alongside the check run, MCPJam posts one comment per pull request and updates that same comment in place on every later push. It is posted for every conclusion, including a pass — so the comment is always current, and there is never a thread of stale verdicts to scroll past. The comment carries: * The verdict line — what the check concluded on this head commit. * What the result is not. A check that could not produce a verdict says so in the comment rather than leaving you to infer it from a colour. * “What to fix” — one line per failing case, saying what to go and change. Each line names the case and the one thing to look at; it does not claim to know why the case failed. * A collapsed JSON block , described below, for coding agents. Comments are posted for every connected repository by default, and an admin can turn them off per repository — see Turning comments off . ​ The machine-readable JSON block The last section of the comment is a collapsed, fenced JSON block. It exists so a coding agent working on the pull request can read the result without parsing prose, and it is stable enough to be worth parsing. It is a bounded projection of MCPJam’s eval decision contract — not the whole contract, and not a dump of the run. It carries: Key What it says schemaVersion The version of this projection. Read it before anything else. The head SHA The commit this result describes. A result whose SHA is not the tip is out of date. The outcome and conclusion What MCPJam measured, and what it reported to GitHub. Trial counts How many trials ran, passed, and failed. One entry per failing trial Its first failed stage , its failure category, and the tool names the case expected versus the ones it observed. omittedFields The fields of the contract this projection deliberately leaves out. Two things to know when reading it: * It is a projection, and it says so. Anything the projection drops is named in omittedFields , so a consumer can tell “MCPJam did not include this” from “MCPJam measured nothing here”. Do not infer an absent field’s value. * The first failed stage is a location, not a diagnosis. It says where the chain stopped — connection, discovery, selection, the tool call, the response, or the user’s request being satisfied — and nothing about why. Treating it as a cause is how a change lands in the wrong file. ​ Review comments on removed tool names When a failing case asserts a tool that was never called, MCPJam looks for that tool name in the pull request’s own diff. If the diff removes a line containing the name in quotes, and the name does not come back on any added line, MCPJam leaves a single review comment on that removed line. That comment is a location, not a cause . It says: the case expected this tool, this line is where the name stopped existing, start here. It does not claim the removal is a mistake — renaming or deleting a tool on purpose is an ordinary thing for a pull request to do, and when it is, the fix is to update the case in MCPJam rather than to put the line back. MCPJam never requests changes. Review comments are posted as plain comments, so they never sit as a blocking review on your pull request. ​ What appears in a comment, and what never does Comments and review comments are public on a public repository . What MCPJam writes there is deliberately bounded. What can appear: * fixed MCPJam copy — the same sentences on every repository; * the head SHA, the check-run id, and the pull-request number; * links back into MCPJam; * eval case titles, as you wrote them; * tool names; * stage reasons and counts; * the same clamped build and server output the check run already shows. What never appears: * organization, project, or installation identifiers; * model or provider names; * prompts; * tool-call arguments; * credentials — evidence strings are credential-redacted before they are rendered. If a case title or a tool name is itself sensitive, treat it as published: those are the two fields that carry your own words into a public comment. ​ Per-repository switches Each connected repository has three labeled switches: Switch What it controls Checks Whether the eval suite runs on pull requests to this repository. Conformance Whether a conformance check runs alongside the eval suite. Disabled and dimmed while Checks is off, because it is a sub-setting of checks. Comments Whether MCPJam posts a pull-request comment with the result. Independent of Checks — you can pause checks while still allowing MCPJam to comment, or turn comments off while checks keep running. ​ Turning comments off Comments are on for every connected repository by default , including repositories connected before the feature existed. An admin opts a repository out; there is nothing to opt in to. Go to Settings → Integrations → GitHub , find the repository in Connected repositories , and turn off the Comments switch. This setting changes nothing about the check itself — whether it runs, what it measures, and how it reports are all decided elsewhere. Only the comment stops. Was this page helpful? Yes No Previous Common MCP Errors Solutions to common MCP server connection and configuration issues Next ⌘ I discord github website linkedin x Powered by This documentation is built and hosted on Mintlify, a developer documentation platform On this page * mcpjam.yaml * Fields * env * Binding and ports * The pull-request comment * The machine-readable JSON block * Review comments on removed tool names * What appears in a comment, and what never does * Per-repository switches * Turning comments off Assistant Responses are generated using AI and may contain mistakes. Contact support