Skip to content

Test Gate

New in v2.5.0. If your project has a test suite, Whittl runs it after the AI says it is finished and reports what the change broke, before you see "done". The AI no longer gets to skip the tests under pressure, and you no longer find out three edits later that change number one broke something.

The gate came from a specific request: "before it gives me back changed code it has to run the regression tests, and if they fail, fix them, repeat until tests pass." v2.4 shipped the skill that teaches the model to do that. v2.5.0 adds Whittl-side enforcement so it cannot be forgotten.

How it works

  1. Baseline. Before the AI touches anything, Whittl runs your tests once and records which ones already fail. Those are yours, not the AI's, and they never count against it.
  2. Edits. The AI does its work through the tool loop as usual.
  3. Gate. When the loop finishes, Whittl runs the tests again and compares. Only failures that are new since the baseline are reported.
  4. Report. The status bar shows Test Gate: passed (no new failures) or Test Gate: N new failure(s), and a failure report lands in the chat with the failing test names and the relevant excerpt of the runner's output.

The report is written so the AI can act on it directly. In the current release, that next step is yours to trigger: reply with something like "fix the failing tests" and the model gets the report plus the code. Automatic re-prompting inside the same generation, up to the configured iteration cap, is the planned v2.5.x follow-up; the cap and timeout settings below already exist for it.

What counts as "having tests"

The gate looks for, in this order:

  1. A pytest.ini in the project root, or a [tool.pytest] section in pyproject.toml
  2. A tests/ directory containing test_*.py files
  3. Any test_*.py file in the project root

If none of those exist, the gate silently stays out of the way. Nothing is run and nothing is reported.

Tests run with pytest in the project's own Python, the same runtime Test Run uses, so a venv project tests inside its venv. Suites written with unittest work too: pytest collects unittest.TestCase classes as long as pytest itself is installed in that runtime.

What it does not do

  • It does not block the result. The edits are already in your project when the report appears. The gate is a report, not a veto. Use History & Archives to roll back if you decide the change is not worth keeping.
  • It does not run on every kind of generation. It hooks the tool-use loop, which is how modifications and Agent Mode tasks run. A first-time generation of a brand-new project has no baseline to compare against.
  • It does not fix pre-existing failures. By design. If you have three tests parked as broken, the AI is not sent to fix them, and they do not count against its changes.

Settings

In settings.json under your Whittl data directory. Not yet in the Preferences dialog.

Key Default Meaning
test_driven_iteration_enabled true Run the gate at all.
test_driven_max_iterations 3 Iteration cap for the planned automatic fix loop; shown in the report today.
test_driven_test_timeout runner-dependent How long a single test run may take before the gate gives up on it. A timed-out run is treated as "could not check", never as a failure.

Working with the gate

Keep the suite fast. The gate runs your whole suite twice per generation. A suite that takes a minute makes every modification a minute slower. If you have slow integration tests, mark them and deselect them in pytest.ini.

Write tests for the behaviour you care about. The gate can only protect what a test asserts. Field testing showed the pattern clearly: the projects that stayed healthy across dozens of AI edits were the ones with a small suite of behaviour tests, not the ones with none.

Pair it with the skill. The bundled test-driven-iteration skill teaches the model to run tests itself before claiming completion, and to decide when a change even needs a test. The skill makes the model try; the gate makes sure it cannot skip.

What's next