ikieEngineeringGet started
All posts

PerformancePostgresServer

We filled the database with 10,000 users, and something else broke first

Production got a little slower, so we measured where ikie would fall over first, with a test database of 10,000 sign-ups and an estimate for 1,000 people online at once. The board query went from 527 ms to 5 ms and the alarm cron from 462 seconds to 40 ms. But the first thing to break wasn't the board. It was the calendar sync that ran every time someone opened the app.

한국어

In mid-October, production numbers got a little worse. Over three days the median for GET /api/v1/days, which serves the board, went from 77 ms to 138 ms, and /home from 100 ms to 230 ms. There were few users yet and nobody had complained, but it meant things were getting slower as data piled up, not as users arrived.

So while fixing what was slow, we went one step further. What breaks first at 10,000 sign-ups, and at 1,000 of them opening the app on the same morning?

First, where does the time go?

Vercel's metrics only show total request time, so we couldn't tell sign-in checks from computation from which query had grown. Every public API and MCP response now carries a standard Server-Timing header (sign-in check, use case, query count and total, overall). A request over 150 ms writes one log line with its per-phase times and its four slowest queries. A query that runs more than ten times in one request is logged as a group.

Read queries over 60 ms are re-run once per query every 15 minutes as EXPLAIN ANALYZE inside a read-only transaction, logging planning, execution and disk reads. This never blocks the response and nothing decides anything from it. It told us the production board query spent 21–60 ms planning and 86–99 ms executing, with no disk reads.

Reading my board scanned everyone's events

The board found its candidates among everyone's items, then filtered by "can this person see it". The time-range indexes covered everyone too, so more sign-ups meant more of other people's events, and a slower board for me.

To measure it, we built a test database with the production schema and 10,000 users: local Postgres 16 with about 5.3 million items (each person with Google Calendar events, repeats, birthdays, work logs and an inbox), about 4 million alarms, 3,000 Live Activity tokens, roughly 9 GB. Before and after were measured on the same database.

The fix decides "where this person can reach" first: calendars they can read, things placed on their board, shares, and invites. Only inside that set does it pick by time range, using per-calendar indexes and an end-time condition together.

BeforeAfter
Show the numbers
UserBeforeAfter
Someone with only 11 events527 ms5 ms
Someone with two Google calendars564 ms13 ms
Someone with a shared calendar626 ms13 ms
Someone with a lot of work logs668 ms17 ms
Month board query time for four kinds of user on the 10,000-user test database. The today board went from 249–281 ms to 5–11 ms.

The key row is the person with only 11 events, still at 527 ms. It was slow because of other people's data, not theirs.

Two crons that run every minute

What collapsed hardest wasn't the board. It was work that runs every minute with nobody asking.

  • The alarm cron checked the visibility rule against every alarm of everyone with a device, every minute. With 4 million alarms one run took 462 seconds. A job that runs every minute took almost eight.
  • The Live Activity cron computed the board of every token holder in turn, every minute. That was 46.8 seconds at 100 users, so about 23 minutes per run at 3,000.
BeforeAfter
Show the numbers
CronBeforeAfter
Alarm cron (4 million alarms)462 s0.04 s
Live Activity cron (3,000 tokens)1,400 s0.105 s
Time for one run. The Live Activity 1,400 s is extrapolated from 46.8 s measured at 100 users. The alarm cron takes 210 ms on its first run, then 40–60 ms.

Both changed from "look at everything, then pick" to "find only what to pick". For each alarm offset in use (10 minutes before, an hour before…), the alarm cron uses an index to find items starting at the moment that would fire now, and checks visibility only for the alarms it picked. All-day and floating times don't know their zone, so they get ±15 hours; repeats get ±1 hour for daylight saving. The Live Activity cron also finds tasks about to start first, then only the devices of people who can see them.

Changes like these drop things easily. So each cron has a test that compares "look at everything" with "pick first" every 10 minutes across more than a day, with New York's daylight-saving start, time-only repeats, moved occurrences, shared calendars and birthdays. We confirmed it fails when the daylight-saving margin is removed or the birthday math is off.

What breaks first at 1,000 people online

From here on it's an estimate, not a measurement. We didn't run 1,000 virtual users. We counted the requests the app sends on one real account (read only) and multiplied by per-request database time from production logs, assuming each person opens the app once, pages seven times and edits twice every 10 minutes.

The result surprised us. The weak point wasn't reading boards. It was the calendar sync that ran every time the app opened. One sync in production ran 1,402 queries and took 10 seconds, 6.6 of them in the database. The query count was the same every time: the work scaled with what you have, not with what changed.

Show the numbers
WorkEstimate
Calendar sync on every app open117 s
Board reads5–15 s
Estimated database work (seconds) piling up each second with 1,000 people online. Sync is the morning rush (about 1,000 syncs a minute × about 7 s of database each); board reads are the average (about 48 a second × 0.1–0.3 s).

No database survives 117 seconds of work arriving every second. So sync moved from the app to the server.

  • Write only what changed. Syncing a task source (Notion and others) opened a transaction per task and rewrote links that hadn't changed. Now it writes nothing when nothing changed. Reading every calendar's raw responses now happens only when the adapter asks; Google has incremental tokens and needs none.
  • Sync on notification. Google and Outlook get a webhook per calendar that fires on change; when one arrives we respond, then sync that account only. Webhooks are renewed before they expire (six days).
  • On app open, only what can't notify. iCloud and Naver (CalDAV) and .ics subscriptions don't notify, so they sync with "Sync now": at most once a minute per account, never twice at once, and if a write or notification arrived meanwhile, exactly once more afterwards.

By the estimate, about 1,000 syncs a minute become 70–100. We're still collecting production numbers after the change.

Fewer requests, and batched

The estimate showed one more thing. A person sends about 44 requests every 10 minutes, and two thirds of them re-ask for months that answered "unchanged" seconds earlier.

  • Ask less. A month answered within the last minute isn't asked again, and after an edit, a push or a sync only the current month and the edited month are asked. Coming back to the app within a minute went from 6 requests to 1. The cost is that edits by people sharing a calendar can show up to a minute late.
  • One request per moment. Opening the app asked separately for home, rules, calendars, plan and several months, each repeating the sign-in check and settings read. Now requests from the same moment (within 10 ms) go out as one GET /api/v1/view, and the answer is split back into each original response.
  • Remember the sign-in check. Every request first ran two queries for session and user (usually 12–40 ms). Now they're remembered for 60 seconds. Signing another device out can take up to 60 seconds to land.
BeforeAfter
Show the numbers
MomentBeforeAfter
First open (nothing on the device)124
Reopen63
Back to the app after more than a minute72
Requests sent by the iOS app. Median of three runs on a real account, read only. One of the four on first open goes only when the language setting changed.

A performance device must give the same result on or off, and the domain must not know it exists.

Batching happens only inside the adapter that calls the server, not in screens or stores. Screens and models still ask "this month, please" and "home, please" separately; the adapter collects calls from the same moment, sends them at once and splits the answer back. The web uses the open-source dataloader; iOS uses a small adapter in the same style. What gets batched together lives in one app policy shared by web and iOS, and with batching off, requests go out separately. Tests check that on and off give the same answers. How we went on to cut requests after edits, pushes and opening a sheet is in We left the screens alone and made the app ask the server less.

What we learned

  • We didn't know until we measured. We worried about board reads. What broke first was the sync that ran on every app open, even when nobody did anything.
  • The slow part was always "look at everything, then pick". The board, alarms and Live Activities all had the same shape. Finding what to pick first cut thousands of times.
  • Every speedup has a same-answer test, and we check that the test fails on the old code or with a margin removed. A fast wrong answer is the worst outcome.
  • Measurements and estimates are written apart. The 10,000-user data was measured; 1,000 online is an estimate. We'll add production numbers here once they're in.

The ikie teamikie is a calendar for your events and to-dos in one place. Here we write about the problems we run into while building it, and how we solve them.