In the previous post we moved CI onto four runners on one Mac Studio. That Mac also runs iOS builds, simulators and several Claude Code sessions. The homework left over was "run less on the same machine". This post covers what we measured while doing it, including the part that did not work as hoped.
Only the tests that touch the change
Every PR used to run all 263 test files. Now it picks only the tests that reach the changed files.
- Pick by the import graph. vitest finds the tests that import a changed file (
vitest list --changed). - Declare the tests that read files as text. Some tests do not import code; they read files as text to enforce a rule (line length, imports between layers). Those declare
@guards <folder>in their top comment and run when that folder changes. - Things everything depends on run everything. A change to dependencies, test config, DB migrations or CI config runs the whole suite.
- The full suite runs on main every three hours. Running it on every merge would cost more than it saves.
The test lock changed too. Tests used to run one at a time on this machine; now there are two slots. A picked run takes one slot, a full run takes both.
Result: the median did not move
We compared the 16 green PR runs (out of 21) in the two and a half hours after the change with the earlier measurement.
We opened 15 test logs and counted how many files each one picked. Some PRs picked only 3, but more than half (8) picked 144–147. Two ran everything (262–263). Runs that picked about the same number still ranged from 64 s to 251 s, and that was set by the load on the Mac rather than by the count. All four runners are the same computer.
One barrel file tied every test together
Following why half the suite got picked, even a PR that added a chart component to the blog picked 144 tests. Among its changes, the one that pulled tests in was a single line in the screen feature list (surfaces.ts). That line alone picks 139. Every new screen file has to be listed there under its feature, so almost every PR that adds a screen edits this file.
Only the code graph (the ontology) and one test actually read that list. But app code imports the policy package's barrel (index.ts), and the barrel re-exported the list. To the import graph, almost every test in the app looked like it depended on it.
We took the list out of the barrel and made its two readers import the file directly. The other half of the cause is policy and time-math files that really are used all over the app, so we left those alone. Picking them is correct.
Picking only affected tests is as honest as the import graph. One "re-export" line makes the graph lie.
The time limit that overrode CI
On a busy Mac a test that normally takes 4 s could take over 30 s, so CI gave tests a 120 s limit. But some test files set their own limit (20–60 s), and the value in the file beat the CI flag. So those files broke only when the Mac was busy. We removed the per-file limits and added a test that blocks them from coming back.
The same day brought one more. The first full run on the three-hour schedule went red: the setup that unpacks the DB dump (beforeAll) hit a 10 s limit. The --hookTimeout flag from CI does not reach each test project, so the default 10 s was still in place (--testTimeout does reach them). We moved the limit into the config file.
The CI that never compiled Swift
CI tested the web app and the shared packages but never built the iOS app. When the public API contract changes, the iOS code has to follow; if it didn't, we found out only after merging, when a Mac session broke. That really happened: a field in the contract became required, the iOS demo transport was not updated, and main broke.
Now a PR that changes something iOS reads (the app, the API contract, app policy, design tokens) compiles the app and its tests without signing. It does not take a simulator, since Mac sessions share them, and it keeps a build folder per runner so only what changed is rebuilt. This job is a required check and takes a median of 124 s.
Why we did not turn on the Xcode compilation cache
Xcode 26 has a cache that stores compile results by content. Every thread builds in a fresh working folder (a git worktree), so sharing the cache between folders looked like a big win. We measured with two folders.
With the cache another folder had filled, all 38 app sources and 294 package sources missed. The cache key includes the folder's absolute path. Adding path-mapping settings by hand left the misses as they were and broke the build. The only win is rebuilding in the same folder (82 s without the cache, 25 s with it), but threads almost always build fresh in a new folder. Turning it on would add only the empty-cache cost (about 30%) and disk, so we left it off.
The disk was 97% full
The same day the Mac's disk reached 97% (65 GB left). Working folders stayed behind after threads finished, and each one, with packages and iOS build folders, was 5–15 GB.
- Clean up what was left. We removed 107 leftover working folders and their build folders, orphaned build folders, and 14 simulators threads had created. Free space went to 441 GB.
- Cleanup at the end of the work. Instead of a separate cleanup job, opening and closing a working folder became one pair of commands. Closing removes the folder and its build folders and releases the simulator it held. If there are changed files or unpushed commits, it keeps the folder and says why. Opening first clears leftover folders merged more than a day ago.
What we learned
- When you turn on a device to cut work, measure that it cut work. We turned on affected-only tests, and measuring showed it was picking half the suite. The cause was not the tests but one line in a barrel file.
- On one machine, load sets the time. The same number of tests took anywhere from 64 s to 251 s. The slow end is set by how much else runs on the same Mac, not by the picking.
- Keep a value in one place. With time limits in three places, tests broke only under load. Keep the setting in one place and block it from being written elsewhere.
- Cleanup is part of the work. Pairing open and close left nothing behind, better than cleaning up separately.