ikieEngineeringGet started
All posts

CITestingDev tools

We tested only what changed, and PRs did not get faster

The day after moving CI to one Mac Studio, we made PRs run only the tests that touch what changed. The median did not move and the slow end got slower. Here is why, and the other things we fixed on the same Mac that day.

한국어

In the previous post we moved CI onto four runners on one Mac Studio. That Mac also runs iOS builds, simulators and several Claude Code sessions. The homework left over was "run less on the same machine". This post covers what we measured while doing it, including the part that did not work as hoped.

Only the tests that touch the change

Every PR used to run all 263 test files. Now it picks only the tests that reach the changed files.

  • Pick by the import graph. vitest finds the tests that import a changed file (vitest list --changed).
  • Declare the tests that read files as text. Some tests do not import code; they read files as text to enforce a rule (line length, imports between layers). Those declare @guards <folder> in their top comment and run when that folder changes.
  • Things everything depends on run everything. A change to dependencies, test config, DB migrations or CI config runs the whole suite.
  • The full suite runs on main every three hours. Running it on every merge would cost more than it saves.

The test lock changed too. Tests used to run one at a time on this machine; now there are two slots. A picked run takes one slot, a full run takes both.

Result: the median did not move

We compared the 16 green PR runs (out of 21) in the two and a half hours after the change with the earlier measurement.

BeforeAfter
See the numbers
TimeBeforeAfter
Whole PR (median)150 s150 s
Whole PR (slow end, p90)275 s306 s
Test job (median)90 s88 s
Test job (slow end, p90)192 s236 s
Time per job in PR CI runs. After = 16 PR runs on October 17, 14:24–17:00 UTC.

We opened 15 test logs and counted how many files each one picked. Some PRs picked only 3, but more than half (8) picked 144–147. Two ran everything (262–263). Runs that picked about the same number still ranged from 64 s to 251 s, and that was set by the load on the Mac rather than by the count. All four runners are the same computer.

See the numbers
Files pickedPRs
3–25 files5
144–147 files8
All 262–263 files2
Test files picked, counted from 15 PR test logs. The full runs were PRs that changed dependencies or CI config.

One barrel file tied every test together

Following why half the suite got picked, even a PR that added a chart component to the blog picked 144 tests. Among its changes, the one that pulled tests in was a single line in the screen feature list (surfaces.ts). That line alone picks 139. Every new screen file has to be listed there under its feature, so almost every PR that adds a screen edits this file.

Only the code graph (the ontology) and one test actually read that list. But app code imports the policy package's barrel (index.ts), and the barrel re-exported the list. To the import graph, almost every test in the app looked like it depended on it.

BeforeAfter
See the numbers
CaseBeforeAfter
Tests picked for a one-line edit to surfaces.ts1392
Test files counted with vitest list --changed for the same edit.

We took the list out of the barrel and made its two readers import the file directly. The other half of the cause is policy and time-math files that really are used all over the app, so we left those alone. Picking them is correct.

Picking only affected tests is as honest as the import graph. One "re-export" line makes the graph lie.

The time limit that overrode CI

On a busy Mac a test that normally takes 4 s could take over 30 s, so CI gave tests a 120 s limit. But some test files set their own limit (20–60 s), and the value in the file beat the CI flag. So those files broke only when the Mac was busy. We removed the per-file limits and added a test that blocks them from coming back.

The same day brought one more. The first full run on the three-hour schedule went red: the setup that unpacks the DB dump (beforeAll) hit a 10 s limit. The --hookTimeout flag from CI does not reach each test project, so the default 10 s was still in place (--testTimeout does reach them). We moved the limit into the config file.

The CI that never compiled Swift

CI tested the web app and the shared packages but never built the iOS app. When the public API contract changes, the iOS code has to follow; if it didn't, we found out only after merging, when a Mac session broke. That really happened: a field in the contract became required, the iOS demo transport was not updated, and main broke.

Now a PR that changes something iOS reads (the app, the API contract, app policy, design tokens) compiles the app and its tests without signing. It does not take a simulator, since Mac sessions share them, and it keeps a build folder per runner so only what changed is rebuilt. This job is a required check and takes a median of 124 s.

Why we did not turn on the Xcode compilation cache

Xcode 26 has a cache that stores compile results by content. Every thread builds in a fresh working folder (a git worktree), so sharing the cache between folders looked like a big win. We measured with two folders.

See the numbers
CaseClean build
Cache off93 s
On, empty cache124 s
On, cache filled by another folder88 s
On, again in the same folder25 s
Measured on a busy Mac, so seconds wobble. We also counted cache hits.

With the cache another folder had filled, all 38 app sources and 294 package sources missed. The cache key includes the folder's absolute path. Adding path-mapping settings by hand left the misses as they were and broke the build. The only win is rebuilding in the same folder (82 s without the cache, 25 s with it), but threads almost always build fresh in a new folder. Turning it on would add only the empty-cache cost (about 30%) and disk, so we left it off.

The disk was 97% full

The same day the Mac's disk reached 97% (65 GB left). Working folders stayed behind after threads finished, and each one, with packages and iOS build folders, was 5–15 GB.

  • Clean up what was left. We removed 107 leftover working folders and their build folders, orphaned build folders, and 14 simulators threads had created. Free space went to 441 GB.
  • Cleanup at the end of the work. Instead of a separate cleanup job, opening and closing a working folder became one pair of commands. Closing removes the folder and its build folders and releases the simulator it held. If there are changed files or unpushed commits, it keeps the folder and says why. Opening first clears leftover folders merged more than a day ago.

What we learned

  • When you turn on a device to cut work, measure that it cut work. We turned on affected-only tests, and measuring showed it was picking half the suite. The cause was not the tests but one line in a barrel file.
  • On one machine, load sets the time. The same number of tests took anywhere from 64 s to 251 s. The slow end is set by how much else runs on the same Mac, not by the picking.
  • Keep a value in one place. With time limits in three places, tests broke only under load. Keep the setting in one place and block it from being written elsewhere.
  • Cleanup is part of the work. Pairing open and close left nothing behind, better than cleaning up separately.

The ikie teamikie is a calendar for your events and to-dos in one place. Here we write about the problems we run into while building it, and how we solve them.