ikieEngineeringGet started
All posts

TestingPlaywrightCIPostgres

A screenshot test caught a 5.7-second database query

We photograph thirteen key web screens with the server and browser clocks frozen, and compare them pixel by pixel with saved images. It ran 700 times in eight days, and we sorted all 26 failures. One of them pointed not at a picture but at the database.

한국어

Several Claude threads change the ikie web app at the same time. Over two days starting October 15, more than 130 PRs touching the screens were merged. Type checks and unit tests don't know how CSS draws a screen. Once, a rule meant for narrow screens leaked onto wide ones and the week view's timeline got a height of 0 (#857). All the code was right; only the screen was empty.

So we keep pictures of the key screens, and every PR that touches them takes the pictures again and compares them pixel by pixel. This is Playwright's screenshot comparison (toHaveScreenshot).

  1. 10/13The first nine: home in light and dark, the item sheet, the inbox, design components (#355)
  2. 10/15Phone and wide screens shot together by two workers (#482)
  3. 10/17Moved to the Mac runner, with only the browser in Linux on Docker (#694)
  4. 10/17Followed a half-drawn sheet to the database and turned JIT off (#755)
  5. 10/19Added tests that measure position instead of pixels (#857)
  6. 10/20Docs home and a single doc make it thirteen (#899)

What we shoot

Seven screens, on a phone (iPhone 15 size) and a wide screen (1440×900). On wide screens the inbox always sits beside home, so it's shot on its own only on the phone. Thirteen pictures in all.

ScreenPhoneWideWhat it shows
Home, light○○Day list, timeline and Now card; the side column on wide
Home, dark○○The same board in dark colors
Item sheet○○An event with tasks inside it, opened
Inbox○The list of undated tasks
Docs home○○Recently viewed docs, meeting notes for today's events
A single doc○○Cover, icon and subpages
Design components○○Every component on /design

The same picture every day

The hard part of screenshot testing isn't the comparing. It's taking the same picture again. Clocks, data, fonts, motion and late responses all shake the picture. We stopped them one by one.

What shakes itHow we stop it
The server's nowStart the server with node --import and freeze Date at one moment (Wednesday 9:30 AM in Seoul)
The browser's nowPut it at the same moment with Playwright's page.clock
What's on the boardEmpty the test database and fill a fixed day (five events, six tasks, six docs) through the public API
Text like "added yesterday"Move the database's creation times to that morning with SQL, keeping their order
Language and time zoneFixed to Korean and Asia/Seoul
MotionReduced motion on, animations off, text caret hidden
FontsWait until fonts load, and always shoot with the same version of a Linux browser
Things that arrive lateWait until the scroll position and page height hold still for 0.3 seconds; shoot the sheet only after its spinner is gone
Leftover noiseUp to 1% of pixels may differ

There's a reason to freeze both clocks. The server reads the first screen's board ahead of time; after that the browser fetches. If the two disagree on "today", dates on one screen don't match.

Filling the board through the public API is deliberate too. Data made the way the app makes it shows the same values the app would. Rows inserted straight into the database can miss computed fields. Since this empties a database, it refuses and stops if the address isn't this machine (localhost).

Each operating system draws fonts its own way. Even on Linux, pictures shot in a Claude session's container came out slightly different from CI's, so retakes used the pictures CI took (#636). Now they are always shot in a pinned version of Playwright's official Linux image. Since CI moved to the Mac, the app runs on the Mac and only the browser runs in Linux on Docker. That story is in the Mac runner post.

When it runs

Only on PRs that change files that reach the screen: screen code, copy, the calculation layer, design tokens and the app policy interpreter. PRs that change only tests or other packages skip it.

See the numbers
RunnerTime per run (median)
GitHub runner (478 runs)2 min
Mac runner (154 runs)2.7 min
Only first attempts that passed. The Mac runner shares one machine with the rest of CI, so in busy moments (the top 10%) a run takes about 10 minutes.

From October 12 to 20 it ran 700 times. 643 passed, and 31 were cancelled because a newer commit arrived on the same PR. 26 failed.

Sorting the 26 failures

We read the CI log of each failure and sorted the causes. One log had been deleted and couldn't be read.

See the numbers
CauseFailures
The picture changed15
The app didn't build4
The runner wasn't ready4
The screen was half drawn2
No log1
The four build failures were the PRs' own type errors; the four runner failures were the Mac's Docker being off or the test database not starting.

Most were screens changed on purpose

11 of the 15 changed pictures were the item sheet, the screen that changed most in this period: a prerequisites section arrived, it was matched to the iOS chip sheet, and it was polished. In those cases the same PR retakes the pictures (--update-snapshots) and ships them with the code. 18 PRs added or retook pictures, and sheet pictures changed most often (12 times wide, 7 on the phone). A reviewer can compare two pictures instead of reading the CSS.

The day we merged on red

On October 15 the event invites PR (#546) added an "Invite" button to the sheet's attendee group. The wide sheet picture changed by 46,799 pixels (4%) and turned red, but the PR merged without retaking it. Screenshot comparison wasn't a required check.

After that main was red, and a PR that never touched the sheet (#542) also went red with exactly 46,799 pixels and merged red too. The identical pixel count was the clue: it meant the same cause. The very next PR (#547) only retook the picture.

A red check merged past becomes the next person's problem, and the next person spends their time hunting for what they broke.

A half-drawn sheet pointed at the database

The most valuable failure wasn't a picture. On October 17, on the security review PR (#755), both the phone and wide tests stopped before shooting the sheet. The sheet opens first with the board's values and shows a spinner next to the time while it fetches the details (alerts, repeat notes). The test shoots after the spinner goes away, and the spinner never went away.

Following it, the query that reads the sheet details took 5.7 seconds on the test database. Running it took a few milliseconds. The rest was Postgres JIT compilation.

One migration turned JIT off for new connections to this database (alter database … set jit = off). The same details came back in 0.08 seconds. The results are identical; only the execution strategy changes. Where we don't own the database and can't change the setting, it leaves a notice and moves on, because a deploy shouldn't stop over an execution strategy.

This test wasn't built to measure performance. But because it waits for "everything fetched" before shooting, a wait that never ended showed up as a failure.

Name what you wait for, and the wait itself becomes a test. Had we just waited a fixed time, we would have shot the half-drawn sheet and moved on.

Some we couldn't explain

Some pictures changed on PRs that didn't change the screen. On a PR that only touched iOS, the phone sheet differed by 5,150 pixels, and on two PRs the same day the phone design page differed by exactly 3,208 pixels. The second looks like one shared cause, but we never pinned down either.

Pictures aren't enough

A picture only says "different". It doesn't say what's wrong or why. The zero-height timeline (#857) would show up in a picture, but after the fix we added a test that measures position: "the week view timeline fills its board cell (over 300px), sits to the right of the side column on wide screens, and the narrow-screen all-day row is hidden." On the code before the fix, it fails with a height of 0.

Mouse and keyboard behavior on wide screens is also tested as behavior, not pixels: a single click on an empty slot, dragging snaps to 15 minutes (5 with Alt), the right-click menu, trackpad side swipes, shortcuts and so on. There are 12 of these tests. They only read the same fixed day.

What's left

  • iOS isn't compared by picture. iOS UI tests check behavior. Screenshot comparison is web only so far.
  • We shoot only Korean. Text overflowing or clipping in the other five languages isn't guarded by pictures yet.
  • It isn't a required check. You can merge past a red picture, like on the day we did.

What we learned

  • Freezing is harder than comparing. Clocks, data, fonts, motion and late responses each have to be stopped before the same picture comes out.
  • A pixel count is a fingerprint. When different PRs fail by exactly the same number of pixels, it's the same cause.
  • Name the condition you wait for. We wrote "until the spinner is gone" instead of "wait one second", and that's how 5.7 seconds became visible.

Running only the tests a change touches is in the second CI post, and the principle of doing each job in one place is in the principles post.

The ikie teamikie is a calendar for your events and to-dos in one place. Here we write about the problems we run into while building it, and how we solve them.