Several Claude threads change the ikie web app at the same time. Over two days starting October 15, more than 130 PRs touching the screens were merged. Type checks and unit tests don't know how CSS draws a screen. Once, a rule meant for narrow screens leaked onto wide ones and the week view's timeline got a height of 0 (#857). All the code was right; only the screen was empty.
So we keep pictures of the key screens, and every PR that touches them takes the pictures again and compares them pixel by pixel. This is Playwright's screenshot comparison (toHaveScreenshot).
- 10/13The first nine: home in light and dark, the item sheet, the inbox, design components (#355)
- 10/15Phone and wide screens shot together by two workers (#482)
- 10/17Moved to the Mac runner, with only the browser in Linux on Docker (#694)
- 10/17Followed a half-drawn sheet to the database and turned JIT off (#755)
- 10/19Added tests that measure position instead of pixels (#857)
- 10/20Docs home and a single doc make it thirteen (#899)
What we shoot
Seven screens, on a phone (iPhone 15 size) and a wide screen (1440×900). On wide screens the inbox always sits beside home, so it's shot on its own only on the phone. Thirteen pictures in all.
| Screen | Phone | Wide | What it shows |
|---|---|---|---|
| Home, light | ○ | ○ | Day list, timeline and Now card; the side column on wide |
| Home, dark | ○ | ○ | The same board in dark colors |
| Item sheet | ○ | ○ | An event with tasks inside it, opened |
| Inbox | ○ | The list of undated tasks | |
| Docs home | ○ | ○ | Recently viewed docs, meeting notes for today's events |
| A single doc | ○ | ○ | Cover, icon and subpages |
| Design components | ○ | ○ | Every component on /design |
The same picture every day
The hard part of screenshot testing isn't the comparing. It's taking the same picture again. Clocks, data, fonts, motion and late responses all shake the picture. We stopped them one by one.
| What shakes it | How we stop it |
|---|---|
| The server's now | Start the server with node --import and freeze Date at one moment (Wednesday 9:30 AM in Seoul) |
| The browser's now | Put it at the same moment with Playwright's page.clock |
| What's on the board | Empty the test database and fill a fixed day (five events, six tasks, six docs) through the public API |
| Text like "added yesterday" | Move the database's creation times to that morning with SQL, keeping their order |
| Language and time zone | Fixed to Korean and Asia/Seoul |
| Motion | Reduced motion on, animations off, text caret hidden |
| Fonts | Wait until fonts load, and always shoot with the same version of a Linux browser |
| Things that arrive late | Wait until the scroll position and page height hold still for 0.3 seconds; shoot the sheet only after its spinner is gone |
| Leftover noise | Up to 1% of pixels may differ |
There's a reason to freeze both clocks. The server reads the first screen's board ahead of time; after that the browser fetches. If the two disagree on "today", dates on one screen don't match.
Filling the board through the public API is deliberate too. Data made the way the app makes it shows the same values the app would. Rows inserted straight into the database can miss computed fields. Since this empties a database, it refuses and stops if the address isn't this machine (localhost).
Each operating system draws fonts its own way. Even on Linux, pictures shot in a Claude session's container came out slightly different from CI's, so retakes used the pictures CI took (#636). Now they are always shot in a pinned version of Playwright's official Linux image. Since CI moved to the Mac, the app runs on the Mac and only the browser runs in Linux on Docker. That story is in the Mac runner post.
When it runs
Only on PRs that change files that reach the screen: screen code, copy, the calculation layer, design tokens and the app policy interpreter. PRs that change only tests or other packages skip it.
From October 12 to 20 it ran 700 times. 643 passed, and 31 were cancelled because a newer commit arrived on the same PR. 26 failed.
Sorting the 26 failures
We read the CI log of each failure and sorted the causes. One log had been deleted and couldn't be read.
Most were screens changed on purpose
11 of the 15 changed pictures were the item sheet, the screen that changed most in this period: a prerequisites section arrived, it was matched to the iOS chip sheet, and it was polished. In those cases the same PR retakes the pictures (--update-snapshots) and ships them with the code. 18 PRs added or retook pictures, and sheet pictures changed most often (12 times wide, 7 on the phone). A reviewer can compare two pictures instead of reading the CSS.
The day we merged on red
On October 15 the event invites PR (#546) added an "Invite" button to the sheet's attendee group. The wide sheet picture changed by 46,799 pixels (4%) and turned red, but the PR merged without retaking it. Screenshot comparison wasn't a required check.
After that main was red, and a PR that never touched the sheet (#542) also went red with exactly 46,799 pixels and merged red too. The identical pixel count was the clue: it meant the same cause. The very next PR (#547) only retook the picture.
A red check merged past becomes the next person's problem, and the next person spends their time hunting for what they broke.
A half-drawn sheet pointed at the database
The most valuable failure wasn't a picture. On October 17, on the security review PR (#755), both the phone and wide tests stopped before shooting the sheet. The sheet opens first with the board's values and shows a spinner next to the time while it fetches the details (alerts, repeat notes). The test shoots after the spinner goes away, and the spinner never went away.
Following it, the query that reads the sheet details took 5.7 seconds on the test database. Running it took a few milliseconds. The rest was Postgres JIT compilation.
One migration turned JIT off for new connections to this database (alter database … set jit = off). The same details came back in 0.08 seconds. The results are identical; only the execution strategy changes. Where we don't own the database and can't change the setting, it leaves a notice and moves on, because a deploy shouldn't stop over an execution strategy.
This test wasn't built to measure performance. But because it waits for "everything fetched" before shooting, a wait that never ended showed up as a failure.
Name what you wait for, and the wait itself becomes a test. Had we just waited a fixed time, we would have shot the half-drawn sheet and moved on.
Some we couldn't explain
Some pictures changed on PRs that didn't change the screen. On a PR that only touched iOS, the phone sheet differed by 5,150 pixels, and on two PRs the same day the phone design page differed by exactly 3,208 pixels. The second looks like one shared cause, but we never pinned down either.
Pictures aren't enough
A picture only says "different". It doesn't say what's wrong or why. The zero-height timeline (#857) would show up in a picture, but after the fix we added a test that measures position: "the week view timeline fills its board cell (over 300px), sits to the right of the side column on wide screens, and the narrow-screen all-day row is hidden." On the code before the fix, it fails with a height of 0.
Mouse and keyboard behavior on wide screens is also tested as behavior, not pixels: a single click on an empty slot, dragging snaps to 15 minutes (5 with Alt), the right-click menu, trackpad side swipes, shortcuts and so on. There are 12 of these tests. They only read the same fixed day.
What's left
- iOS isn't compared by picture. iOS UI tests check behavior. Screenshot comparison is web only so far.
- We shoot only Korean. Text overflowing or clipping in the other five languages isn't guarded by pictures yet.
- It isn't a required check. You can merge past a red picture, like on the day we did.
What we learned
- Freezing is harder than comparing. Clocks, data, fonts, motion and late responses each have to be stopped before the same picture comes out.
- A pixel count is a fingerprint. When different PRs fail by exactly the same number of pixels, it's the same cause.
- Name the condition you wait for. We wrote "until the spinner is gone" instead of "wait one second", and that's how 5.7 seconds became visible.
Running only the tests a change touches is in the second CI post, and the principle of doing each job in one place is in the principles post.