Test Runner

Configuration for the SQL test runner (npgsqlrest --test) — write tests for your endpoints as plain .sql files and run them against the real endpoint pipeline, in-process. For the full walkthrough (test file anatomy, HTTP blocks, assertions, isolation patterns, migrations, Docker, template databases), see the Testing Guide.

The TestRunner section is a top-level configuration section (a sibling of NpgsqlRest, not nested inside it). It only has an effect when the client runs with the --test argument — it is completely inert during normal server operation.

sh

npgsqlrest ./config.json --test
npgsqlrest ./config.json ./test-config.json --test --watch
npgsqlrest ./config.json --test --testrunner:filter=login --testrunner:tag=smoke

Overview

json

{
  "TestRunner": {
    "FilePattern": "",
    "Filter": "",
    "Tag": "",
    "ExcludeTag": "",
    "ConnectionName": "",
    "MaxParallelism": 0,
    "FailFast": false,
    "PerTestTimeout": "30s",
    "JUnitOutput": null,
    "Keep": false,
    "DetailedReport": false,
    "AllowEmpty": false,
    "Coverage": null,
    "CoverageThreshold": null,
    "LoggerName": "NpgsqlRestTest",
    "ResponseTempTable": {
      "Name": "_response",
      "MultiNamePattern": "_response_{n}",
      "Columns": {
        "Status": "status",
        "Body": "body",
        "ContentType": "content_type",
        "Headers": "headers",
        "IsSuccess": "is_success"
      }
    },
    "Steps": {},
    "Setup": [],
    "Teardown": []
  }
}

Settings

Setting	Type	Default	Description
FilePattern	string	""	Glob selecting test files. Empty disables discovery.
Filter	string	""	Narrow the discovered set by path (substring or glob).
Tag	string	""	Run only files carrying at least one of these tags.
ExcludeTag	string	""	Skip files carrying any of these tags (exclude wins).
ConnectionName	string	""	ConnectionStrings entry to run tests against instead of the main connection.
MaxParallelism	int	0	Max test files running concurrently. 0 = processor count.
FailFast	bool	false	Stop scheduling new tests after the first failure/error.
PerTestTimeout	string	"30s"	Per-test-file timeout ("30s", "5m", seconds, or hh:mm:ss). 0 disables.
JUnitOutput	string	null	Optional path for a JUnit XML report.
Keep	bool	false	Skip Teardown so a failed run's state can be inspected.
DetailedReport	bool	false	Richer console report (passed assertions, failing SQL, notices).
AllowEmpty	bool	false	"No tests discovered" exits 0 instead of 4.
Coverage	bool?	null	Endpoint-coverage summary. null = on for full runs, quiet when narrowed; true/false = always/never.
CoverageThreshold	int?	null	Fail an otherwise-passing run (exit 2) when coverage is below this percentage.
LoggerName	string	"NpgsqlRestTest"	SourceContext name of the runner's log channel.
ResponseTempTable	object	see below	Naming and columns of the per-HTTP-block response temp table.
Steps	object	{}	Named, reusable steps for Setup/Teardown and per-file annotations.
Setup	array	[]	Run-once setup steps, before endpoint discovery, in written order.
Teardown	array	[]	Run-once teardown steps, always (best-effort), in written order.

Every setting is also available as a command-line override with the standard configuration syntax: --testrunner:filter=login, --testrunner:coveragethreshold=100, and so on.

FilePattern

Glob (same engine as SqlFileSource.FilePattern) selecting the test files. Empty disables discovery — --test then exits with code 4 (no tests found).

Two common layouts:

json

// Co-located: app.sql (endpoint) next to app.test.sql (test), same tree
{ "TestRunner": { "FilePattern": "./sql/**/*.test.sql" } }

json

// Separate tree: endpoints in ./sql, tests in ./tests
{ "TestRunner": { "FilePattern": "./tests/**/*.test.sql" } }

The co-located layout works because SqlFileSource.SkipPattern defaults to "*.test.sql", so test files are never exposed as endpoints.

Filter

The fast path for iterating on one test:

sh

npgsqlrest ./config.json --test --testrunner:filter=login

Matched against each file's cwd-relative path: a value without wildcards is a substring match; with wildcards it uses the same glob engine as FilePattern. Empty runs everything discovered.

Tag and ExcludeTag

Tag filtering (comma- or whitespace-separated lists, case-insensitive). A test file declares its tags with a header annotation:

sql

-- @tag smoke, auth

Tag runs only files carrying at least one of the listed tags; ExcludeTag skips files carrying any of them — exclude wins when both match. Composes with Filter (both must pass).

sh

npgsqlrest ./config.json --test --testrunner:tag=smoke --testrunner:excludetag=slow

Tags declared in an included annotation profile (via \i/\ir in the file header) count as if written in the file. See the TEST TAG annotation.

ConnectionName

A ConnectionStrings entry to run the tests against instead of the app's main connection. In test mode it becomes the connection used for endpoint type-checking (Describe) and execution, so it can point at a dedicated test database that a Setup step creates first — it does not need to exist at startup.

json

{
  "ConnectionStrings": {
    "Default": "...Database=app_db...",
    "Admin": "...Database=postgres...",
    "Test": "...Database=app_test_{rnd5}..."
  },
  "TestRunner": {
    "ConnectionName": "Test"
  }
}

Random tokens

{rnd1}…{rnd10} are random lowercase tokens (length = the digit), generated once and stable for the whole run, usable in any connection string or Setup/Teardown SQL — so app_test_{rnd5} resolves to the same name in the connection string, the create database step, and the drop database step. Need several distinct tokens of the same length? Indexed instances {rnd5_1}…{rnd5_9} are each independent.

MaxParallelism

Maximum number of test files running concurrently; 0 means processor count. Each test file runs on its own non-pooled physical connection (fresh session — no temp-table, GUC, or prepared-statement carryover), so parallel files cannot see each other's uncommitted state.

FailFast

Stop scheduling new tests after the first failure or error. In-flight tests still finish and are reported.

PerTestTimeout

Per-test-file timeout. Accepts "30s", "5m", "1h", a plain number of seconds, or "hh:mm:ss". 0 disables. A timed-out file is reported as an error (exit code 2).

JUnitOutput

Optional path to also write a JUnit XML report — the standard CI artifact (GitHub Actions, GitLab, Jenkins all consume it). Console output is always printed regardless. Assertion names (the second column of a boolean-SELECT assertion) become the JUnit test-case names.

json

{ "TestRunner": { "JUnitOutput": "./test-results.xml" } }

Keep

Skip Teardown so a failed run's state (the test database, fixture rows) can be inspected. Remember to clean up manually — with {rnd}-named databases each kept run leaves one behind.

DetailedReport

Richer console report: lists passed assertions (✓), prints the full failing SQL statement, and shows captured raise notice output for passing tests too (notices always show under failing tests).

This shapes the report only — for diagnostic logging of every executed query and HTTP invocation, raise the runner's log channel instead:

json

{
  "Log": {
    "MinimalLevels": {
      "NpgsqlRest": "Off",
      "NpgsqlRestClient": "Off",
      "NpgsqlRestTest": "Verbose"
    }
  }
}

AllowEmpty

Treat "no tests discovered" as success (exit 0) instead of exit 4. Useful for repos where a test tree may legitimately be empty.

Watch mode

Watch mode is enabled by the top-level Watch configuration section ("Watch": { "Enabled": true }) or its CLI shorthand --watch — it is not a TestRunner setting, because the same section drives both watch flavors (test watch with --test, server watch without). In test mode: run everything once, then re-run on changes until Ctrl+C.

• A changed test file re-runs alone (Filter still applies).
• A changed endpoint file (matching SqlFileSource.FilePattern) triggers an in-process endpoint rebuild — sources re-read, re-described, endpoint registry swapped atomically — followed by a full rerun, with the endpoint delta reported (+ POST /api/new, - GET /api/x (endpoint dropped — check its SQL file for errors)). To make this safe, watch mode forces SqlFileSource.ErrorMode from Exit to Skip; non-watch --test keeps Exit for CI.
• A database routine change (create/replace/drop/comment on functions or procedures, detected by polling the routine discovery query — Watch:DatabasePollingInterval, default 2s) rebuilds endpoints and re-runs everything (— change detected (database) —).
• Any other changed .sql under the test tree (an included fixture or profile, whose dependents are unknown) re-runs everything.

Teardown runs once, on exit — synchronously inside the SIGINT/SIGTERM handler, so the test database is dropped even when the watch process is stopped through a wrapper like bun run or npm run; a second Ctrl+C force-quits. A graceful stop exits 0 regardless of test outcomes — watch is not for CI gating.

Coverage and CoverageThreshold

Endpoint-coverage summary after the run: exercised N of M testable endpoints, plus the exact list of untested ones:

code

19 passed, 0 failed, 0 error(s)  —  19 assertions in 9 files

endpoint coverage: 2/2 (100%)

Coverage is tri-state:

Value	Behavior
null (default)	Report after full runs; stay quiet when the run is narrowed by Filter/Tag (a deliberately partial run would just nag).
true	Always report, including narrowed runs.
false	Never report.

CoverageThreshold (0–100) turns it into a CI gate — it always reports, regardless of Coverage or narrowing: an otherwise-passing run below the threshold exits 2. Set it to 100 and forgetting to write a test for a new endpoint fails the build, naming the endpoint.

"Covered" means invoked at least once by a test — execution, not assertion depth (the same semantics as code coverage). Endpoint kinds the runner rejects (SSE, upload, login/logout, outbound proxy) are excluded from the ratio and counted separately.

LoggerName

SourceContext name of the runner's own log channel (default "NpgsqlRestTest"); set its level independently under Log:MinimalLevels. Discovery and parsing log at Debug, each executed query and HTTP invocation at Verbose, raise notice output by its severity.

ResponseTempTable

Each HTTP block's response is captured into its own temp table on the test's connection, created fresh (no IF NOT EXISTS — a duplicate name fails the test loudly).

Setting	Default	Description
Name	"_response"	Table name when the file has one HTTP block.
MultiNamePattern	"_response_{n}"	Name pattern when the file has 2+ blocks; {n} is the 1-based block ordinal.
DebugTable	null	Debugging aid: also mirror every response into this permanent table (see below).
Columns.Status	"status"	int — HTTP status code.
Columns.Body	"body"	text — response body (cast to ::jsonb to assert on JSON).
Columns.ContentType	"content_type"	text — response content type.
Columns.Headers	"headers"	jsonb — response headers.
Columns.IsSuccess	"is_success"	boolean — true for 2xx.

A null or empty column name omits that column. A per-block override is available with the # @response <name> directive inside the HTTP block.

DebugTable — inspect responses after the run

Temp tables vanish with the test's rollback and connection, so they cannot be examined afterwards — and re-issuing the request from an .http file cannot reproduce a response that depended on the test's uncommitted fixtures. Set DebugTable (e.g. "_responses_debug") and every captured response is also mirrored into a permanent table, written on a separate autocommit connection — immune to rollbacks, recreated at the start of every run (it always holds the last run):

sh

npgsqlrest ./config.json --test --testrunner:responsetemptable:debugtable=_responses_debug

One table covers everything — each HTTP block adds one row: captured_at, test_file, block (that block's response-table name: _response, a _response_{n} ordinal, or the # @response name), method, path, status, body, content_type, headers, is_success. After the run, open a query editor:

sql

select test_file, block, status, body::jsonb
from _responses_debug
where status >= 400;

The temp-table semantics are unchanged; enabling it prints a loud warning — it is a debugging aid, do not enable in CI. In the fresh-test-database workflow combine it with Keep, or teardown drops the database (and the mirror with it).

Steps

Named, reusable steps (name → step object, same shape as Setup/Teardown entries). Reference them by name in Setup/Teardown, or from an individual test file's header annotations (-- @setup, -- @teardown):

json

{
  "TestRunner": {
    "Steps": {
      "CreateDatabase": { "Sql": "create database app_test_{rnd5}", "ConnectionName": "Admin" },
      "ApplyMigrations": { "Command": "bun db up", "WorkingDirectory": "./db" },
      "DropDatabase": { "Sql": "drop database if exists app_test_{rnd5} with (force)", "ConnectionName": "Admin" }
    }
  }
}

A step object is one of:

Shape	Runs
{ "Sql": "..." }	SQL text, statement by statement, on the test connection — or on any named ConnectionStrings entry via "ConnectionName".
{ "SqlFile": "..." }	A SQL file, statement by statement, same connection rules.
{ "Command": "...", "WorkingDirectory": "..." }	An OS shell command — migration runners, Docker, pg_dump, anything.

Referencing an unknown step name is a configuration error (exit 3).

Every step also has an Enabled flag (default true): a disabled step is simply ignored wherever it is referenced — skipped with a debug log line, never an error. The default configuration ships disabled example steps covering the typical scenarios (create/drop a {rnd}-named test database, apply a schema file, run a migration tool, start/stop a Docker PostgreSQL) — copy one into your config, adjust names and connections, and flip Enabled to true instead of typing it from scratch:

json

{
  "TestRunner": {
    "Steps": {
      "CreateTestDatabase":  { "Enabled": false, "ConnectionName": "Admin", "Sql": "create database app_test_{rnd5}" },
      "DropTestDatabase":    { "Enabled": false, "ConnectionName": "Admin", "Sql": "drop database if exists app_test_{rnd5} with (force)" },
      "ApplySchema":         { "Enabled": false, "SqlFile": "./migrations/schema.sql" },
      "RunMigrationTool":    { "Enabled": false, "Command": "echo replace with your migration tool command", "WorkingDirectory": "." },
      "StartDockerPostgres": { "Enabled": false, "Command": "docker run -d --name npgsqlrest-test-pg -e POSTGRES_PASSWORD=postgres -p 54329:5432 postgres" },
      "StopDockerPostgres":  { "Enabled": false, "Command": "docker rm -f npgsqlrest-test-pg" }
    }
  }
}

Setup and Teardown

Run-once lifecycle around the whole run. Setup runs before endpoint discovery — which is what makes the create-a-fresh-database workflow possible: by the time endpoints are described against ConnectionName, the database exists and is migrated. Steps run in the exact order written; each entry is a step name from Steps or an inline step object.

json

{
  "TestRunner": {
    "Setup":    [ "CreateDatabase", "ApplyMigrations" ],
    "Teardown": [ "DropDatabase" ]
  }
}

Teardown runs always (best-effort), even when the run fails — and it is guaranteed beyond the happy path: from Setup onward the runner intercepts SIGINT (Ctrl+C) and SIGTERM and runs Teardown synchronously in the signal handler, and a process-exit hook covers hard exits (e.g. a broken endpoint SQL file under SqlFileSource.ErrorMode: Exit). Keep: true skips Teardown deliberately.

Exit codes

Code	Meaning
0	All tests passed (or a graceful watch-mode stop).
1	At least one assertion failed.
2	At least one error (SQL error, timeout, unsupported endpoint, an interrupted run, or a failed coverage gate).
3	Setup or configuration error.
4	No test files found (AllowEmpty: true turns this into 0).

Related

• Testing Guide — the full walkthrough with scenarios: transactions, fixtures, test databases, template clones, migrations, Docker, CI
• Test file annotations — @setup, @teardown, @connection, @tag
• SQL File Source — endpoint files and SkipPattern
• Logging — Log:MinimalLevels, including "Off" to mute a channel