Bottom line. This system is the front door between 100,000 endpoint sensors and the cloud services that consume their telemetry. It terminates 100,000 authenticated long-lived connections, turns what they send into an ordered durable record at about 370 MB/s, and delivers cloud commands back to individual sensors or whole fleets. Three decisions shape everything else: durability lives in the cloud, with the sensor spool covering only network partitions, ingest runs in two stages so producers and consumers evolve independently, and the envelope is fixed while payloads stay opaque. Every service below is stateless or owns its state outright, so any one of them can fail without losing data the system has acknowledged.
| Dimension | Value | What it forces |
|---|---|---|
| Customers | ~100 | Tenant isolation is cheap; per-customer quotas do the work |
| Sensors | ~100,000 | 100k concurrent connections, ~500 per session node at N=200 |
| Events per sensor per day | ≤ 10,000 | 11,574 events/s average — the number that sizes everything |
| Event size | 1–64 KB, 32 KB average | ~370 MB/s average, ~741 MB/s worst case |
| Messages per sensor per day | ≤ 100 | 116/s total. Egress is latency-bound, not throughput-bound |
| Event types | 11 | Must be extensible without touching ingest |
| Message types | 4 | Same |
| Operating systems | One | OS metadata is specified in Appendix A, not enforced |
Ingest bandwidth dominates. Command traffic is three orders of magnitude smaller, so it is designed for latency and fanout reach, not volume.
Eleven sensor → cloud types. Only SensorStart is protocol; the rest are telemetry and take the
normal ingest path.
| Event | Note |
|---|---|
SensorStart, SensorStop |
Agent lifecycle |
SensorHeartbeat |
Telemetry every 24 h — not a liveness check; protocol keepalive covers that |
ProcessStart, ProcessEnd, ProcessHeartbeat |
Process lifecycle; heartbeat carries long-running processes |
FileWritten, FileModified, FileDeleted |
File system activity |
NetworkConnectIn, NetworkConnectOut |
Connection records |
Four cloud → sensor types, all persisted before delivery. They differ in how the queue may treat them, which matters at fanout (§9):
| Message | Semantics | Queue treatment |
|---|---|---|
ConfigUpdate |
Policy snapshot per customer | Coalesce — newest wins, older entries droppable |
SensorUpdate |
Agent version change | Coalesce by target version |
ThreatIntelUpdate |
Indicator set snapshot | Coalesce — newest wins |
ResponseAction |
One-shot action: isolate, kill, quarantine | Never coalesced, own priority lane, longest retention |
| Decision | Choice | Buys | Costs |
|---|---|---|---|
| Durability lives in the cloud; the sensor spool covers partitions | Sensor buffers in memory and on disk only while disconnected or unacked; the cloud log is the record. At-least-once delivery, idempotent consumers | Endpoints can crash, be wiped or be decommissioned without holding the system's only copy. Ack latency is the durability budget, so batches stay small and flushes frequent | Unacked data at the moment an endpoint dies is lost — accepted, and bounded by a short ack window |
| Ingest runs in two stages | Stage 1 admits and appends a raw batch. Stage 2 decodes, validates and deduplicates into the structured log consumers read | Each side deploys and evolves independently; a bad stage-2 deploy is recovered by replaying the raw log; new event types need no stage-1 change | One extra hop of latency, and raw retention for the replay window |
| Fixed envelope, opaque payload | Ingest reads routing metadata and never parses payloads | New event types are a registry entry plus a consumer change — no ingest redeploy | Consumers must skip types they do not know |
Each is load-bearing: remove one and the others stop working.
| Service | Responsibility | State | Scaling unit | If it fails |
|---|---|---|---|---|
| Session tier | Terminate connections, authorize, admit, apply flow control, ack after durable append | Per-connection only | Pod | ~10k sessions drop; sensors reconnect and replay their spool |
| Raw ingest log | The admitted record: batches as received, short retention | The record itself | Partition | Sensors stop getting acks and slow down |
| Normalizer | Decode, validate, dedupe, emit coverage markers, write structured | None beyond offsets | Partition | Ingest continues; only structured lag grows |
| Structured ingest log | Ordered per-sensor record that consumers read | The record itself | Partition | Consumers stop advancing; replay after recovery |
| Event ingress log | Cloud messages accepted from producers, before targeting | The record itself | Partition | Producers get backpressure; nothing is lost |
| Fanout | Resolve a message's targets to sensors, write per-sensor queue entries | None | Stateless | Delivery delayed; ingress log holds the message |
| Command router | Tenant check, persist to per-sensor queue, attempt live push | None | Stateless | Publishes back up to the caller; queued messages unaffected |
| Per-sensor queue | Hold messages for offline sensors: priority, cap, TTL, prune on ack | Per-sensor queues | Shard by sensor | Delivery degrades to requeue |
| Session directory | Map sensor → session node, TTL | Ephemeral | Stateless | Live push degrades to queue-only: latency cost, no loss |
| Customer cache | Which customers exist | LRU, 100 entries | In-process | Falls back to registry; unknown customers rejected |
| Consumers | Detection, analytics, export | Their own checkpoints | Consumer group | Their lag grows; alert before retention runs out |
| Cold export | Convert aged segments to columnar storage | — | Batch job | Cold queries delayed; hot path unaffected |
Every arrow is a call or a read, labelled with its interface. Cylinders are state a service owns.
(sensor_id, epoch, seq) against a per-sensor watermark, so replaying the
raw log is safe.ingest_ts ordering and per-customer ingest metrics.How dedupe is performed, and where its state lives. There is no separate dedupe database. The
state is one 12-byte tuple per sensor — (epoch, high-water seq) — and the log is its source of
truth:
| Step | Mechanism |
|---|---|
| Compare | Incoming (epoch, seq) against the sensor's watermark |
seq ≤ watermark, same epoch |
Duplicate — drop, count, no write |
seq = watermark + 1 |
Append, advance watermark |
seq > watermark + 1 |
Gap — append, emit a coverage marker for the hole |
| Lower epoch | Regression — new epoch issued at the session tier (§5.2) |
| Same identity, different content hash | Collision — quarantine the record and alert; never pick a winner |
The watermark is rebuilt from the log tail on restart rather than held only in process memory, so a normalizer restart re-derives it instead of accepting a replay it already committed. The per-sensor cache in front of it is an optimisation: a compacted per-key state table, if one is available, replaces the rebuild on cold start. Collision detection needs a bounded recent-hash window per sensor, because the watermark proves which identities are committed, not what their content was.
message_id.ResponseAction has its own lane so an isolate command cannot
queue behind 3,000 stale config updates.customer_id travels on the message and is checked, not trusted. A mismatch is rejected
and alerted: the failure it prevents is a kill command reaching the wrong customer's endpoint.| Interface | Between | Contract |
|---|---|---|
| Sensor link | sensor ⇄ session tier | TLS, length-prefixed frames. SensorStart, ACCEPT, REJECT, BATCH, ACK, SKIP, DRAIN; commands arrive as MESSAGE with ACK or NACK |
| Raw append | session tier → raw ingest log | Append batch, return committed offset. No decoding |
| Normalize | raw log → normalizer | Read from offset; write structured records; idempotent by watermark |
| Consumer read | structured log → consumers | Read from an offset; total order per sensor; independent cursors |
| Message publish | command sources → event ingress | CloudMessage(msg_id, customer_id, target, type, priority, ttl, payload) |
| Target resolution | fanout → endpoint metadata | Resolve a selector (customer, OS, version, tag) to sensor IDs |
| Delivery | session tier → sensor | Live: push then expect ACK/NACK. Offline: replay from the queue cursor |
| Operational | load balancer, ops tooling | Liveness, readiness, drain trigger. Readiness reflects log reachability |
Five boundaries. Telemetry flows left to right; commands right to left. Solid edges are load-bearing, dotted edges are advisory and may be stale or wrong.
Telemetry in
Sensor → buffer → BATCH → session terminator (authorize, admit) → raw append → durable → ACK watermark → sensor trims → normalizer → structured log → consumers
Commands out
Command source → event ingress → fanout (resolve targets, coalesce) → per-sensor queue → live push or replay → sensor ACK or NACK → prune
Reconnect
Sensor → SensorStart(epoch, watermark) → compared with log tail → ACCEPT with authoritative epoch → replay unacked → resume
Draining deregisters from the directory only after the flush, so nothing is acked and then
lost.
Three terms used throughout. A watermark is the highest sequence number such that everything at or below it is committed; the sensor trims to it and the cloud expects from it. A spool is the sensor's buffer for the disconnected window. A coverage marker is an event a sensor or the pipeline emits when telemetry is dropped, so the gap is explained rather than discovered later.
Every event needs a stable identity, because three things depend on it: deduplicating a replay, acknowledging a range so the sensor can release its buffer, and detecting that something is missing. Only the first is achievable without an ordered identifier.
| Option | Size per event | Ack shape | Dedupe | Verdict |
|---|---|---|---|---|
(epoch, seq) |
8 bytes | One integer: the contiguous watermark | Compare against a per-sensor watermark | Chosen |
| UUIDv7 per event | 16 bytes | A list of ids, or a watermark the cloud computes | Requires retaining a set of seen ids — expensive at 11.5k/s | Rejected |
| No id, rely on TCP ordering and replay everything | 0 | Byte offset | Impossible | Rejected |
The identifier costs 8 bytes on a 32 KB event: 0.025%. Positional encoding — a batch header
holding base_seq, event ids implied by index — would recover those 8 bytes and is rejected: a
sensor may split already-numbered events into separate batches by priority, so position no
longer maps to a contiguous run. Allocating a range at send time fails too, because the buffer
is released by ack and an ack names sequence numbers, so identity must exist before send.
Sequence numbers restart when a sensor's local state is lost, and a restart at zero would regenerate numbers the cloud has already seen, silently discarding real events as duplicates. An epoch closes each numbering generation. It changes only when the counter restarts, so a normal restart resumes mid-sequence.
| Sensor presents | Meaning | Cloud response |
|---|---|---|
| Equal to its high-water | Normal resume | Accept, echo the epoch back |
| Absent | New install, or local state unreadable | Accept, issue high-water + 1 |
| Lower than high-water | Rolled back or forked state | Accept under a new epoch; refuse the stale one; coverage marker; alert |
| Higher than high-water | Corrupt counter, or the log lost records | Do not adopt. Record the divergence, alert, continue from the log tail |
The epoch is issued by compare-and-set (set to E+1 where current = E). Two session nodes doing
read-then-write would mint the same epoch twice, which is the collision epochs exist to prevent.
| Field | Carried by | Cost | What it buys |
|---|---|---|---|
customer_id, sensor_id |
Session, asserted once | 0 per event | Routing, partitioning, tenancy |
epoch: u32 |
Session, re-asserted per batch | 4 bytes per batch | Closes each numbering generation |
seq: u64 |
Sensor, at send | 8 bytes per event | Dedupe, ack, gap detection |
event_type |
Sensor | Registry index | Routing to consumers, schema selection |
event_ts |
Sensor clock | 8 bytes | When it happened — the only usable time after a disconnected window |
ingest_ts |
Session tier | 8 bytes | When we learned it; latency and skew detection |
payload |
Sensor | 1–64 KB | The event itself, opaque to ingest |
payload_schema: u16 |
Sensor | 2 bytes | Candidate for removal — if event_type pins the schema version, this duplicates the registry |
Four schemas, owned by different parties. Keeping them separate is what allows new event types without an ingest change.
| Schema | Owner | Enforced by | Changes how |
|---|---|---|---|
| Envelope | This design | Session tier bounds, normalizer validation | New optional fields only; consumers ignore unknown |
| Endpoint identity | This design | Append-only metadata store | Additive; see Appendix A |
| Payload | Sensor engineering, per event type | Producer and consumer, not ingest | Versioned in a schema registry, by event_type |
| Cloud message | Cloud services, per message type | Fanout and sensor | Versioned in a registry, by message type |
Rules that make the envelope stable:
customer_id and sensor_id, never a copy of
endpoint metadata. OS and version live in the metadata store (Appendix A), so correcting a
host's OS does not rewrite history.Each consumer derives its own projection from the same envelope. There is no single "event identity" tuple; there are keys for different jobs:
| Job | Key |
|---|---|
| Deduplication | (customer_id, sensor_id, epoch, seq) |
| Ordering scope | (customer_id, sensor_id, epoch) |
| Routing and authorization | (customer_id, sensor_id) |
| Physical position | (partition, offset), assigned at append |
Metadata is decoupled from storage format, so repartitioning or replacing the log changes a projection, not the producer.
Terminology is deliberately generic; §15 maps it to what specific products call these things.
hash(customer_id, sensor_id). Order is guaranteed per sensor, which is
the only order the system promises.| Tier | Horizon | Encoding | Volume | Notes |
|---|---|---|---|---|
| Raw | 1–2 h | Same as wire | ~2.7 TB | Exists to be replayed after a bad normalizer deploy |
| Hot | 24–48 h | Protobuf records on the log | 32 TB (96 TB at RF=3) | Live consumers |
| Warm | 7 d | Protobuf records on the log | 224 TB (671 TB at RF=3) | Investigation and replay |
| Cold | 90 d | Parquet, partitioned by customer and date | 2.88 PB raw | Analytical scans, export, compliance |
Hot data is written as Protobuf records: one envelope per event, appended in order, read sequentially from an offset by consumers that need every event.
Cold data is converted to Parquet at export. The reasons are structural, not stylistic:
| Property | Protobuf records | Parquet |
|---|---|---|
| Write cost | Append, no transformation | One conversion per segment, batch |
| Read pattern served | Sequential, per event | Column scans, filtered aggregates |
| Compression | Per record | Per column: repeated customer_id and sensor_id collapse, timestamps delta-encode |
| Selective read | Decode whole envelope | Read two columns out of ten |
| Ordered replay | Native | Must sort on (sensor_id, epoch, seq) |
Conversion happens once, on segments that will never be re-ingested, and the payload column stays opaque bytes. Where payloads are text or JSON, extraction into columns is a consumer-owned enrichment, not something the gateway does.
Retention, not throughput, dominates cost. Choosing a horizon is a product decision, and the price belongs next to it.
| Mechanism | Bounds | Where |
|---|---|---|
| Admission control | New connections per customer per second | Session tier |
| Flow control | Bytes and events per customer once connected | Session tier |
| Backpressure accounting | Whose entries are evicted under pressure — never another customer's | Session tier, queue |
| Quotas | Total throughput and storage per customer | Logs, cold store |
Placement is separate from fairness. Shuffle sharding looks like the answer and is not, at this tenant count: with 40 nodes and a shard of 3, any two tenants share a node about 14% of the time, so one noisy tenant still touches fourteen others. Explicit placement is stronger and simpler: a tenant's partitions go on a chosen subset of brokers, and the largest customer, who can be 20% of the fleet, gets dedicated capacity.
Telemetry is many producers to few consumers. Commands are the mirror image: few producers to 100,000 endpoints. Fanout is the service that makes that asymmetry work, and it is the piece most likely to be asked about.
Producer → event ingress log → fanout (resolve target, coalesce) → per-sensor queue entries → live push or replay → ACK or NACK → prune
Fanout is a queue-driven worker, so a producer that publishes is done as soon as the ingress append commits. Resolution and delivery are asynchronous, retryable, and independent of producer liveness.
| Target | Resolution | Entries written |
|---|---|---|
| Single sensor | Direct | 1 |
| Customer fleet | All known sensors for the customer | up to ~5,000 |
| Selector | Metadata query: OS family, agent version, tag, last-seen window | N, computed |
| Segment | Precomputed membership snapshot | N, from snapshot |
Selector targeting is the reason endpoint metadata exists as a service rather than as fields scattered through telemetry (Appendix A): "patch every sensor on build 4.1.x" is a metadata query, not a scan of the event log.
| Hazard | Control |
|---|---|
| A fleet-wide message becomes 100k queue entries | Store the body once; queue entries reference it by msg_id |
| 100k sensors receive and act at the same instant | Jittered delivery, spread over a window proportional to fleet size |
| Repeated config pushes crowd out an isolate command | Priority lane for ResponseAction; coalescing for the other three types |
| A message fanning out twice after a retry | Dedupe entry on (msg_id, sensor_id) |
| A sensor that has been offline for weeks accumulates stale config | Coalesce to the newest entry per type; TTL drops what expired |
| A selector matches 80k sensors by mistake | Fanout previews the count and requires an explicit threshold override |
| Property | Guarantee |
|---|---|
| At-least-once to each targeted sensor | Yes — queue entry persists until ACK |
| Coalescing | ConfigUpdate, SensorUpdate, ThreatIntelUpdate; never ResponseAction |
| Ordering | Per sensor, by priority lane then arrival |
| Expiry | TTL per type; stale config is worse than none |
| Cross-tenant delivery | Rejected at the router when customer_id does not match the target sensor |
| Failure | Effect | Handling |
|---|---|---|
| Session node dies | ~10k sessions drop | Jittered reconnect elsewhere; spool replays unacked. Reconnect storm is the real risk |
| Endpoint crashes, is wiped, is decommissioned | Its unacked buffer is gone | Accepted, and bounded by a short ack window. Coverage cannot be reported by a dead host, so this window is the metric that is watched |
| Raw log degraded | Cannot accept | Stop acking, slow down, buffer; gap markers if exceeded |
| Normalizer down | Structured log stops advancing | Ingest and acks continue; raw retention must cover the outage |
| Bad normalizer deploy | Bad structured records | Roll back and replay the raw log — the reason it exists |
| Structured log degraded | Consumers stop | Consumers resume from their offsets after recovery |
| Event ingress degraded | Producers cannot publish | Producers retry; per-sensor queue and directory are unaffected |
| Registry unavailable | Cannot authorize new sessions | Serve known customers from cache; reject unknown ones; live sessions unaffected |
| Directory stale | Live push fails | Fall back to queue — latency cost only |
| Sensor restart | Buffer must not be renumbered | Epoch persists; resume from the high-water mark |
| Sensor state rollback | Superseded epoch regenerates used numbers | New epoch, stale namespace closed, coverage marker, alert |
| Sensor clock step | event_ts unreliable |
Ordering never depends on it; alert on divergence from ingest_ts |
| Queue cap reached | Cannot persist more for one sensor | Evict by priority, retaining ResponseAction longest |
| Consumer falls behind retention | That consumer loses data permanently | No repair after the fact; alert on lag as a fraction of retention |
| Duplicate delivery | Consumer sees an event twice | Idempotent on (customer_id, sensor_id, epoch, seq) |
A record that is never accepted is never acked, so it is retried forever and everything behind it stalls. One bad record must not become a permanent, silent outage of one sensor's telemetry, so every retrying hop needs a terminal outcome.
| Hop | Poison looks like | Terminal outcome |
|---|---|---|
| Sensor → session tier, per event | Oversize, undersize, undecodable envelope | Drop the event, ack past it, coverage marker |
| Sensor → session tier, per batch | Corrupt frame, bad length, decompression failure — nothing ackable inside | Sensor-side hash on buffered records catches it before send; SKIP over a sequence range as backstop |
| Session tier → raw log | Malformed structure | Cannot occur for the batch framing the tier wrote; envelope errors surface in the normalizer |
| Normalizer | Undecodable envelope, unregistered type | Record a coverage marker, skip, count. Replayable from raw, so never silently lost |
| Fanout → sensor | A message the sensor cannot apply | Explicit NACK, drop, alert, no retry |
| Structured log → consumer | A record a consumer cannot decode | Dead-letter after N attempts, with a marker |
Two rules make this safe:
CustomerExists(customer_id) is the only supporting API.
Encoding is decided in two places, and they are independent.
On the sensor link: not this system's choice, with one constraint. The sensor transport — framing, compression, keepalive cadence — is sensor-engineering territory and is treated as out of scope beyond the frame contract the gateway must parse. The constraint this design imposes is that the envelope's fields must be readable without decoding the payload.
Inside the backend: Protobuf, for three reasons:
It flips if consumers need random access to single fields without full deserialization — search and hunting over hot telemetry. That is a consumer-side access pattern, so it can change per tier without touching ingest, which is exactly what the two-stage split buys.
Bounds are part of the contract. 64 KB maximum per event, a fixed maximum batch count and compressed size, and the same limits applied to the decompressed stream so a compressed batch cannot become a memory amplifier.
| Operator question | Metrics |
|---|---|
| Is telemetry flowing? | Ingest rate and bytes/s by customer and type; raw append latency p50/p99; ack latency p99; raw-to-structured lag |
| Is anything lost silently? | Coverage-gap markers by reason and by stage; buffer depth distribution per cell plus threshold-breach events; epoch regressions; dedupe drops; consumer lag as a fraction of retention |
| Is the command path working? | Ingress-to-fanout lag; queue depth per sensor; time-to-delivery by path; NACKs; coalesced vs dropped messages |
| Is a tenant misbehaving? | Admission rejections, throttle events, connect rate, connected ÷ known sensors |
| Property | Target | Note |
|---|---|---|
| Durability of acked telemetry | No acknowledged event lost | A correctness property. Exceeding it is a defect, not error budget |
| Ack latency | p99 ≤ 250 ms at design load | This is the window an endpoint's buffer must cover |
| Raw-to-structured lag | p99 ≤ 5 s | Stage 2 is allowed to lag; it must not lag indefinitely |
| Consumer lag | p99 ≤ 10% of retention | Alert on the fraction, never on absolute lag |
| Coverage-gap markers | Zero in steady state | Any marker is an alert |
Alert on causes — buffer growth, queue depth, admission rejections, stage lag — not symptoms. Rates of change carry more signal than absolute values, because every threshold here depends on fleet size and time of day.
Out of scope by requirement
Out of scope by design
Open
ThreatIntelUpdate may be dropped entirely when a newer snapshot
exists, or must be applied in order.None of these are load-bearing; each primitive has more than one home. The vocabulary stays generic and maps as follows.
| This document | Partitioned-log products | Queue products | Notes |
|---|---|---|---|
| Partitioned log | Topic partition | — | Ordered, replayable, retained |
| Partition key | Message key | — | Same hash, same ordering guarantee |
| Offset | Offset | — | Per-consumer position |
| Consumer group | Consumer group | — | Independent readers |
| Retention | Retention ms / bytes | — | Time or size bound |
| Per-sensor queue | Log with per-key cursor | Queue with per-key lists | Needs priority and prune-on-ack |
| Ephemeral directory | — | KV with TTL | Hint only |
| Primitive | Satisfied by |
|---|---|
| Durable ordered log | Redpanda (preferred): one binary, no JVM, no ZooKeeper, lower tail latency at this write rate. Kafka is the fallback and the safe answer if the deployment is already Kafka-shaped. Either satisfies the contract; the contract is partitions, keys, offsets, consumer groups and retention |
| Per-sensor queue | The same log with per-key cursors; or a KV store with per-sensor lists |
| Ephemeral directory | Redis or etcd with TTL; or gossip within the session tier |
| Customer cache | In-process LRU — 100 entries needs no infrastructure |
| Cold store | Object storage, S3 class, Parquet files |
| Session tier | Stateless processes behind a load balancer, TLS terminated at the node |
One OS is assumed, so none of this is required. It is specified because telemetry without endpoint context is difficult to interpret, and because selector-based command targeting (§9.2) needs a queryable metadata surface.
Three rules shape it:
sensor_id is the key. Everything else is an attribute you filter
on, so cardinality matters more than completeness.sensor_id; metadata is joined at query time. A
corrected OS version does not rewrite 90 days of events.| Group | Fields | Why it earns space |
|---|---|---|
| Host identity | host_id, hostname, domain_role |
Correlate events across reinstalls; distinguish duplicates |
| OS | family, name, version, build, kernel, arch |
Vulnerability matching and patch targeting |
| Device | device_class, virtualization, hypervisor, instance_id |
Distinguish physical, VM and container; ephemeral hosts produce bursty churn |
| Agent | agent_version, policy_id, config_hash, capabilities |
Selector targeting, drift detection, feature negotiation |
| State | last_seen, buffer_depth |
Connected ÷ known sensors, buffer growth alerts |
Not part of the base design. The system is sized here for 100 customers and 100,000 sensors; this appendix records how it grows when those numbers move, and which parts move with them.
| Scale | Events/s | Ingest | Hot 24 h at RF=3 | Partitions at ~25 MB/s |
|---|---|---|---|---|
| 100k sensors | 11,574 | 370 MB/s | 96 TB | ~15, so provision 2–3× |
| 1M sensors | 115,740 | 3.7 GB/s | 960 TB | ~150 |
| 10M sensors | 1.16M | 37 GB/s | 9.6 PB | ~1,500 |
Connections, bytes and fanout destinations do not grow at the same rate. Partition count tracks throughput, not key cardinality: 10M sensors still needs ~1,500 partitions, not 10M. That matters because partition count is the one number that is expensive to change later.
| Order | Constraint | Why |
|---|---|---|
| 1 | Cost, mainly read amplification | Three consumer groups reading the full stream at 1M sensors is ~11 GB/s of cross-AZ traffic, triple the ingest. The bandwidth bill scales with consumers, not producers |
| 2 | Partition count | One ordered log and one writer per partition; the count is chosen once and changing it remaps keys |
| 3 | Deploy churn | 1M sessions means a million TLS handshakes per roll, and the handshake is the expensive part |
| 4 | Directory | 1M entries with constant churn; needs sharding and a staleness budget rather than a TTL and hope |
| 5 | Fanout write amplification | A fleet-wide message is 1M queue entries. The body is shared; the entries are not |
| 6 | Metric cardinality | Per-sensor series stop being viable in the low thousands of sensors |
A cell is a complete stack — session tier, logs, queues, fanout — owning a set of customers.
| Rule | Value | Rationale |
|---|---|---|
| Tenants per cell | ~100, soft cap | Keeps the whole customer registry in one process, so CustomerExists stays a cache lookup and fail-closed stays viable |
| Hard cap | The cell's resource limit: bytes/s and connections | 100 light tenants and 100 telemetry-heavy tenants are not the same cell |
| New tenant | First cell with headroom; stand up a new cell when none has it | Provisioning is control-plane work, not data-plane work |
| Whale tenant | A dedicated whole or partial cell | An isolated cell runs below capacity, and that idle cost is acceptable: security telemetry has little idle time, and isolation buys blast radius and deploy independence |
| Whale larger than a cell | Shard the whale cell by hash(sensor_id) |
The shard becomes the unit of ordering and delivery |
Cells buy three things at once: bounded blast radius, independent upgrade cadence, and bounded metric cardinality by construction — roughly 100 customers per cell means ~100 per-customer series per cell, regardless of fleet size.
Sharding by hash(sensor_id) works provided the shard is the unit of ownership for everything at
once:
| Owned per shard | What breaks if it is split |
|---|---|
| Session affinity | A sensor briefly live in two shards duplicates delivery and breaks ordering across the seam |
| Log partitions | The same problem one layer down |
| Per-sensor queue | A command lands in the wrong shard and is never delivered |
| sensor → shard map | Everything above becomes guesswork |
Two mechanics worth settling in advance:
hash mod N. mod N remaps nearly every
sensor when N changes; a ring moves about 1/N.ConfigUpdate. First contact happens before any config has been
delivered.Forced disconnect, not forced purge. The control plane has two ways to move a sensor, and neither discards data:
| Mode | Mechanism | Cost |
|---|---|---|
| Soft, time-based | A ConfigUpdate asks the sensor to reconnect before a deadline, at a jittered time |
Slowest, but spreads the reconnect over minutes |
| Forced disconnect | The old cell closes the session, so the sensor reconnects immediately and the router resolves its customer to the new cell | Instant, but produces a reconnect burst that has to be staggered by shard or batch |
Both are scheduling actions, not data actions. Nothing is purged on either side: the sensor keeps its buffer and epoch, replays from its watermark after reconnecting, and the old cell's directory entry expires by TTL. That is what makes a move cost latency instead of events.
Three things a move does have to get right:
Per-sensor series are wrong in the low thousands of sensors. Per-sensor state is unavoidable. Keeping the two apart is the whole discipline:
| Thing | Where it lives | Cardinality |
|---|---|---|
| Watermark, buffer depth, last-seen, throttle budget | Per-connection state in the session tier | 1M structs at ~200 B is 200 MB |
| Throughput, errors, gap markers | Metrics by cell and by customer | ~100 series per cell |
| A single outlier — one sensor at 100× the norm | An event on threshold crossing | Bounded by rate, not by fleet |
| Distribution of buffer depth | Histogram or sketch | Fixed |
Per-sensor throttling needs no per-sensor series: a local token bucket in the connection struct, with the budget recalculated periodically from the customer quota divided by their connected count, which the cell already knows.
| Concern | Boundary |
|---|---|
| Session tier, logs, queues, fanout | Per cell |
| Cold storage | Shared lake. Analytic queries are off the hot path, and per-cell lakes either duplicate storage or force a federating query layer |
| Fleet-wide fanout in a whale cell | Per shard, so one command becomes S fans. Dedupe on (msg_id, sensor_id) and jitter still apply; completion tracking becomes per-shard and approximate |
| Tenant → cell map, billing, control plane | Global, and kept boring |
Invariant. None of this touches the envelope, the identity scheme, the ack rule or the two-stage split. Scaling changes placement and counts. If a scaling plan requires changing the envelope, the envelope was wrong.