Two-Stage Fanout

A group message has two very different requirements. First, everyone in the group should agree that message 104 comes after message 103. Second, the system may need to write that message into 10,000 recipient inboxes, push mobile notifications, update unread counts, and fan out WebSocket events. Doing all of that inline in the ordered path makes one large group block every later message.

handleMessage(group_id, message):
  seq = nextSequence(group_id)
  saveMessage(group_id, seq, message)

  for member in getMembers(group_id):
    writeInbox(member, group_id, seq, message)
    pushNotification(member, group_id, seq)

  # A 50,000-member group keeps the sequencer busy for seconds.
  # Later messages wait even though they only need a sequence number.

• Ordering bottleneck: the component that decides sequence numbers is forced to wait for slow delivery work.
• Tail latency amplification: one slow inbox write or push provider call delays the whole group stream.
• Retry confusion: if delivery partially succeeds and the worker retries, some recipients may see duplicates unless inbox writes are idempotent.

Two-stage fanout creates a narrow ordered stage and a wide parallel stage. Stage 1 is a per-group sequencer. It consumes messages partitioned by group, assigns a monotonic sequence number, persists the canonical message, and emits a delivery job. Stage 2 is a fleet of delivery workers. They split the job into recipient tasks and write inboxes or feeds in parallel.

Stage 1: ordered sequencer, partitioned by group_id

on IncomingMessage(group_id, client_msg_id, body):
  seq = incrementAndGet("group:" + group_id + ":seq")
  insert GroupMessage(group_id, seq, client_msg_id, body)
  publish DeliveryJob(key=group_id, value={group_id, seq})

Stage 2: parallel delivery workers, partitioned by recipient_id or shard

on DeliveryJob(group_id, seq):
  message = load GroupMessage(group_id, seq)
  recipients = loadMembers(group_id)

  for shard in split(recipients, 1000):
    publish RecipientBatch(key=shard.id, value={group_id, seq, shard})

on RecipientBatch(group_id, seq, recipients):
  for recipient in recipients:
    upsert Inbox(recipient, group_id, seq)  # unique(recipient, group_id, seq)
    enqueuePush(recipient, group_id, seq)

The trick is that order is attached to the message before fanout begins. Every recipient inbox write carries the same {group_id, seq}pair. Delivery workers can run in any order, retry independently, and still write deterministic inbox positions because the sequence was already chosen by the ordered stage.

• Canonical message table: store one durable copy keyed by{group_id, seq}. Inboxes reference it instead of duplicating the full message body everywhere.
• Recipient inbox index: write a row keyed by{recipient_id, group_id, seq} so retries become upserts rather than duplicates.
• Client rendering: clients sort by sequence number and can detect gaps, such as seeing 103 and 105 while 104 is still being delivered or fetched.

Two-stage fanout does not remove fanout amplification; it controls where that amplification happens. A million-member group still creates a lot of delivery work. The design wins because stage 1 can keep assigning sequence numbers while stage 2 absorbs, throttles, retries, and sheds delivery work according to product priority.

incoming messages topic (key=group_id)
  -> sequencer lag should stay tiny
  -> emits delivery jobs quickly

delivery jobs topic (key=group_id)
  -> can accumulate during spikes
  -> workers split into recipient batches

recipient batch topic (key=recipient_shard)
  -> horizontal scale lever
  -> retries and dead letters isolated by shard

Backpressure tools

• Separate queues: sequencer lag and delivery lag are different SLOs. Alert on both, but do not let delivery lag stop order assignment unless storage is at risk.
• Shard big jobs: split a giant group into recipient batches so one job does not monopolize a worker for minutes.
• Priority lanes: online users, mentions, or paid tenants may use higher-priority delivery queues than cold inbox backfills.
• Rate limits: push providers and downstream databases need token buckets so retries do not amplify an outage.

The same structure appears anywhere one ordered event must update many personalized views. It pairs naturally with the broader fanout trade-off between write-time and read-time fanout described in fanout on write versus read.

• Group chat: sequence per conversation, write per-recipient inbox rows, and push notifications through independent workers.
• Social feeds: assign a post version once, then fan out to follower home timelines in batches while preserving author-post order where the product requires it.
• Enterprise notifications: persist a canonical alert, then deliver to user inboxes, email, Slack, and mobile push with separate retry policies.

• Sequencer hot spots:a huge group is still a single order key. If it exceeds one partition's capacity, you need product-level compromises, substreams, or a more specialized ordering service.
• Gaps are normal during delivery: recipients may receive seq 105 before seq 104 is written to their inbox. Clients should fetch missing canonical messages or show a loading gap.
• Membership snapshots: decide whether delivery uses the member list at send time or delivery time. Most chat systems snapshot or version membership to avoid surprising recipients.
• Deletes and edits: sequence them too. An edit event at seq 110 should not race unpredictably with the original message or with moderation actions.
• Observability: track sequencer lag, delivery lag, inbox write failures, push failures, and per-group hot keys separately.