ikieEngineeringGet started
All posts

SyncCalendarsDesign

We merge external calendars field by field, three ways

So that edits made in Google and in ikie to the same event both survive, ikie remembers the last value both sides agreed on and compares three versions for each group of fields. One engine reconciles Google, iCloud, Outlook, iPhone calendars and .ics subscriptions this way, and never throws away fields it doesn't understand.

한국어

ikie connects Google Calendar, iCloud (CalDAV), Outlook, the calendars on your iPhone and .ics subscriptions, and keeps them in sync both ways. The moment you connect one, the same event lives in two places. Say that in the morning you push a meeting back 30 minutes in ikie on your phone, and meanwhile a colleague renames the meeting in Google. What should survive the next sync?

Both. The time should be ours and the title should be Google's. This post covers the rules we set to make that happen, and what we ran into while keeping them.

Overwriting from one side loses an edit

The simplest approach is "last write wins". Treat the event as one lump and overwrite it with whichever side was edited later. In the example above, if the colleague edited last, the title is right but our new time snaps back. To the user, an edit just reverted for no reason.

Comparing each field separately solves most of this. But knowing who changed a field takes more than two versions. Seeing that our value and the remote value differ doesn't say whether we changed it or they did. That needs a third value.

Three versions: base, ours, theirs

For every external event, ikie keeps a binding row, and in it the last value both sides had in common. That's the base. When syncing, we compare base against ours and base against theirs separately. Whichever side differs from the base is the side that changed.

GroupBase (last synced)Ours (ikie)Theirs (Google)Merged
TitleWeekly syncWeekly syncWeekly sync (2F)Weekly sync (2F)Only they changed it
Time10:00–11:0010:30–11:3010:00–11:0010:30–11:30Only we changed it
LocationRoom ARoom BRoom CWhoever edited laterBoth changed it: counted as a conflict
NotesAgendaAgendaAgendaAgendaNobody changed it
Three versions are compared for each group of fields. A cell that differs from the base (highlighted) shows who changed it. If one side changed it, that side wins. If both did, the later edit wins and we count a conflict.

The rule fits in three lines. If we didn't change it, take theirs. If they didn't change it, keep ours. If both changed it, take whichever was edited later. One pure function makes this call, with no database or network in sight.

export function merge3(base, local, remote, localNewer): MergeResult {
  if (!base) return { merged: remote, conflicts: [] };
  const merged = { ...remote };
  const conflicts: SyncUnit[] = [];
  for (const u of PUSH_UNITS) {
    const localChanged = unitKnown(u, base) && !unitSame(u, base, local);
    const remoteChanged = !unitSame(u, base, remote);
    const takeLocal = localChanged && (!remoteChanged || localNewer);
    if (localChanged && remoteChanged && !unitSame(u, local, remote)) conflicts.push(u);
    if (takeLocal) for (const k of SYNC_UNITS[u]) merged[k] = local[k];
  }
  return { merged, conflicts };
}

If both sides changed a field to the same value, it isn't counted as a conflict. Think of two people fixing the same typo. An event we've never seen has no base, so we take theirs as is.

"Later" is decided by edit time. We read our own edit time in microseconds, which is Postgres's timestamp precision, so even two edits in the same millisecond can be told apart. If the remote doesn't say when it was edited, we compare against the last sync time.

Fields merge in groups

Some fields can't be merged one at a time. Take our start and their end, and you might get an event that ends before it begins. So fields that must change together are handled as one group (a unit).

GroupFields
TimeTime kind, start and end date, start and end time, start and end time zone
RecurrenceRepeat rule (RRULE), added dates (RDATE), excluded dates (EXDATE)
The other twelveKind, title, notes, location, URL, color, busy/free, visibility, conference link, alerts, travel time, my RSVP

There are fourteen groups. Kind and conference link are pull-only: Google doesn't let an event's type change after creation, and conference links are created by the provider.

This structure also helps when we add a new group. When we started syncing alerts, bases saved before that had no alert fields. A naive comparison would read every event as "we changed the alerts" just because the base lacked them, and the first sync would overwrite the provider's alerts with ours. So a group the base doesn't know isn't treated as our change, and the first time around we take theirs.

We don't throw away fields we don't understand

External calendars carry plenty of fields ikie doesn't know. Apple writes properties starting with X-APPLE-…, and Google events carry attachments, email reminders and hidden values other apps left behind. Rewrite an event with only the fields we know and send it back, and all of that is gone.

So we keep the response exactly as received (raw) in the binding row, and when sending we send only the groups that changed.

  • Google gets a PATCH with only the changed groups. For fields Google replaces wholesale, we merge with the raw copy before sending. In the hidden extended properties we change only our own key and leave other apps' keys alone. In the attendee list we change only my RSVP row and send everyone else's exactly as received.
  • CalDAV (iCloud, Naver) gets the received .ics file back, with only the changed groups rewritten. Properties we don't know and separately edited occurrences stay untouched. The sequence number (SEQUENCE) goes up only when time or recurrence changes, since nothing else is worth re-notifying attendees about.

A test holds this. It changes only the title and location of a weekly iCloud event, then checks that the result still has Apple's X-APPLE-TRAVEL-ADVISORY-BEHAVIOR:AUTOMATIC line and both separately edited occurrences, and that the sequence number didn't go up.

Some providers can't hold certain fields. iPhone calendars can't hold color, visibility or travel time. For those fields we put the base value in place of the received one, so we don't read it as "the provider deleted it", and our value survives. Nobody lists what each provider can't hold by hand. It's read from the model declaration, which records each field's mapping to every provider. For a provider like Outlook that keeps only one alert, both sides' alerts are compared only up to what the provider can hold, so the alerts it can't hold aren't read as deleted.

One engine, many adapters

There is one sync engine. It doesn't know which provider it's talking to. It only knows a shape called a port (CalendarPort).

export interface CalendarPort {
  unsupported?: readonly SyncUnit[];
  alarmSlots?: number;
  calendars(): Promise<RemoteCalendar[]>;
  changes(cal: CalendarRef, known: StoredEvent[]): Promise<RemoteChanges>;
  get(cal: CalendarRef, ev: StoredEvent): Promise<RemoteEvent | null>;
  create(cal: CalendarRef, fields: SyncFields): Promise<RemoteEvent>;
  update(cal, ev, fields, units: SyncUnit[], occurrences: OccurrenceOut[]): Promise<RemoteEvent>;
  remove(cal: CalendarRef, ev: StoredEvent): Promise<void>;
}

The only thing that really differs between providers is how you find out what changed. That difference stays inside each adapter.

ProviderHow we find what changed
GoogleIncremental sync token. The first sync fetches everything from 365 days back
iCloud, Naver (CalDAV)Calendar version (ctag) and RFC 6578 sync tokens
OutlookGraph's delta returns only expanded occurrences and can't carry a series, so we compare the event list's ids and changeKeys with the versions we remember
iPhone calendarsThe server can't call the device, so the device uploads one .ics per series with a content hash. Edits we need to send are applied by the device
.ics subscriptionsRead only. Split per UID and versioned by a hash that leaves out DTSTAMP, which changes on every fetch

When sending, all three writable providers (Google, CalDAV, Outlook) get "only change this if it's still the version I saw" (If-Match). If the provider changed in the meantime and refuses (412), we refetch just that event and merge it, and whatever of ours remains in the merged value goes out on the next sync.

Occurrences of repeating events

A repeating event arrives as one series plus exceptions for individual occurrences. An occurrence's address is its original start, just as inside ikie (We don't store repeating events). Google sends occurrence exceptions as if they were separate events, so we apply the series first and attach exceptions after. CalDAV puts exceptions in the same file with a RECURRENCE-ID.

Sending occurrences follows the same group rule. Each occurrence keeps its own base, and we send only the groups that differ from it. This rule came from a bug. When you replied "going" to one occurrence of a recurring meeting someone else organized, we also sent its time and status, which you hadn't changed. Google refused a time edit from a non-organizer and rejected the whole call, reply included. After we compared against the occurrence's own base and sent only the reply, replies went through.

What we fixed after running into it

  • Missing once doesn't mean deleted. Change which calendars are selected on the iPhone, or lose calendar permission, and that calendar drops out of the upload. We used to read that as "every event was deleted". Now a missing calendar isn't deleted, and the range where we treat events as deleted is limited to the period the device actually read. A huge .ics subscription works the same way: if we only took the first 5,000 events, we note that it was truncated and delete nothing that time.
  • Without the stored original, fetch it again. When a CalDAV event's stored .ics was missing, creating a fresh one and overwriting lost the event's identity (UID) and every unknown field with it. Now we fetch that event again and patch on top of it.
  • The same moment can be written differently. Looking up a Google occurrence by its original start in UTC returned nothing, because Google writes the same moment with a time-zone offset. Occurrence edits were skipped silently. Now we read by occurrence id first, and if that fails we pick the neighbouring occurrence with the same original start.
  • The slow part was round trips, not computation. A single sync once took 17 seconds. CPU was 0.2 s. The rest was round trips piling up as calendars were fetched one after another, about 19 per Google account. Now we fetch four calendars at a time, and one failure no longer stops the rest.

What we can't do, we refuse

We can't reconcile everything. When we can't, we don't do half a job. We refuse clearly.

  • Splitting a repeating series tied to an external calendar with "this and following" is refused with a 409. The engine can't yet guarantee which series each remote occurrence belongs to after a split.
  • Events nested inside another item, floating times with no zone, and untimed tasks aren't sent out. External calendars have no place for them.
  • Recurrence an iPhone calendar can't represent (RDATE, unsupported repeat units) isn't downgraded and sent. It isn't sent at all.

What we learned

Most of syncing external calendars came down to knowing exactly who changed what. To do that we wrote the last agreed value down as a fact, grouped the fields that change together, and gathered the judgment into one pure function. The only thing that differs per provider is how to find what changed, so that lives inside the adapters. And we never threw away what we didn't understand. If ikie deletes something a user added in another app just because we don't know what it is, then from that app's point of view, ikie broke it.

The ikie teamikie is a calendar for your events and to-dos in one place. Here we write about the problems we run into while building it, and how we solve them.