Spike: Push Problem Events To Sana

Goal

Move problem-event delivery from routine new-sana polling to Mathspace pushing events close to real time.

Today, new-sana calls Mathspace /sana/get-problem-events (GPE) when it needs problem events. Problem events are not persisted in new-sana, and most GPE calls return no events while still creating load.

Plan

Phase 1: Initial Implementation

The async sync and persistence path runs alongside the current GPE and notification flows. A calculation reads persisted events when its student/catalog partition has a non-null watermark. A null watermark continues to use GPE.

Capability	State	Owner	Behaviour in this phase
Mathspace async sync	New	Mathspace	Reuses the current event sources and projection, coalesces sync triggers, and sends initial or delta histories.
Shared push protocol	New	Both services	Carries watermark-checked event chunks from Mathspace to new-sana.
Persisted Problem Event Store	New	new-sana	Persists immutable problem events, tracks partition completeness, serialises writers, and supports reset.
Calculation integration	New	new-sana	Reads persisted events for a non-null watermark, uses GPE for a null watermark, and gates snapshot persistence.
Existing GPE flow	Retained	Both services	Supplies problem events for null-watermark calculations. A cursorless request also triggers a separate async backfill.
Existing /v4/problem-event-notification flow	Retained	Both services	Continues alongside async push.

Phase 2: Transitional Period

Persisted events are the normal read source. Null-watermark requests may still use the GPE path, which triggers async backfill, or use the incomplete-partition path, which calculates only from available Lantern events, triggers the same async initialisation, and does not persist a snapshot.

Capability	State	Owner	Behaviour in this phase
Mathspace async sync	Primary	Mathspace	Handles source-driven delta syncs and request-driven initial syncs.
Shared push protocol	Primary	Both services	Carries all initial and delta event delivery into new-sana.
Persisted Problem Event Store	Primary	new-sana	Serves every partition with a non-null watermark.
Calculation integration	Dual path	new-sana	Reads persisted events when available and supports both null-watermark request behaviours.
Existing GPE flow	Transitional fallback	Both services	Supplies a complete problem-event view for null-watermark requests using this path and triggers separate async backfill.
Existing /v4/problem-event-notification flow	Retained	Both services	Continues alongside async push.

Demand-driven backfill initialises active student/catalog partitions naturally. Inactive historical partitions can remain incomplete.

Phase 3: End State

GPE and routine new-sana polling are absent. Async push is the only problem-event delivery path. A null-watermark calculation uses only the available Lantern events, triggers async initialisation, returns a best-effort result, and does not persist a snapshot.

Capability	State	Owner	Behaviour in this phase
Mathspace async sync	Retained	Mathspace	Delivers source-driven delta syncs and request-driven initial syncs.
Shared push protocol	Retained	Both services	Remains the only problem-event delivery channel into new-sana.
Persisted Problem Event Store	Authoritative when complete	new-sana	Serves persisted events for every non-null partition and exposes null partitions as incomplete.
Calculation integration	Retained	new-sana	Uses available Lantern events only for null partitions, triggers initialisation, and does not persist snapshots from those calculations.
/sana/get-problem-events	Removed	Both services	No new-sana calculation calls GPE.
/v4/problem-event-notification	Removed	Both services	Completion and test-mode triggers drive async push directly.

Implementation

Mathspace Async Sync Implementation

Event Sources And Projection

Mathspace remains the canonical source of:

• regular workout problem completions from Problem
• test-mode original completions from TestModeProblemAuditLog
• test-mode manual teacher overrides from TestModeProblemAuditLog
• test-mode consistency-review overrides from TestModeProblemAuditLog

The async worker reads the canonical sources through the current GPE event projection and produces the same event shape.

Test-mode source rows are immutable audit events. An original completion creates a row when marks are available. Manual and consistency-review overrides append a row only when the effective correct/incorrect result changes. Each row has its own occurred_at, so an override may arrive after the original completion.

Source timestamp watermarking relies on rows being timestamped when completion or update occurs, using “now” for _cached_completed_at or occurred_at. A row that becomes visible after a worker captures its high-water therefore has a later event timestamp and is found by a later sync.

Sync Triggers And Scheduler

Mathspace schedules a sync after:

• a student completes a problem
• a test-mode problem event is created
• a cursorless GPE request is made for an incomplete student/catalog partition
• new-sana requests initialisation for an incomplete partition

Completion and audit-log scheduling happens after the source transaction commits. Work scheduled by GPE or new-sana is performed by a sync task that reads the canonical sources.

Scheduling coalesces per internal student/catalog identity:

1. Start a short transaction and lock the unique sync row.
2. Increment requested_version.
3. If task_started_at is null or more than 10 minutes stale, set task_started_at = now and running_version = requested_version.
4. Otherwise save only the new requested_version.
5. Commit the transaction.
6. If this scheduler acquired the lease, enqueue one Dramatiq task after commit.
7. When the task finishes, lock the row again. If it no longer owns running_version, exit without changing state.
8. If it still owns the version and requested_version > running_version, refresh the lease, promote running_version, and enqueue one follow-up. Otherwise clear the task lease.

Sync State And Worker

Mathspace has one sync row per internal student<>catalog pair. The worker uses those identities for source scanning and resolves the Sana-facing user_id and catalog_id when building the push request.

The row contains:

• latest_synced_at: wall-clock high-water of the last sync fully accepted by new-sana
• task_started_at: async task lease timestamp
• requested_version: latest requested work version
• running_version: version currently allowed to finalise state

latest_synced_at is the wall-clock source-scan boundary across both source streams, not the timestamp of the latest event. It advances after both streams are exhausted through the captured sync high-water and new-sana accepts that boundary.

Each worker:

1. Reads latest_synced_at as its starting watermark.
2. Captures the current wall-clock time as the task-local sync high-water.
3. Processes both source streams over (latest_synced_at, sync_high_water].
4. Generates the current GPE event shape and sorts events by timestamp.
5. Sends bounded intermediate chunks whose watermarks are the latest complete event timestamps included in those chunks.
6. Sets the last chunk's watermark to the wall-clock sync high-water rather than its latest event timestamp. The last chunk may contain events; it is empty only when the sync contains no events.
7. Records latest_synced_at only after that last chunk is accepted and the task still owns running_version.
8. Never replaces a later stored Mathspace watermark with an earlier one.

Source pagination and transport chunking are separate. Mathspace may page each source internally, but an intermediate transport chunk covers a complete event-timestamp range across both sources. An event-timestamp group cannot be split between chunks. The last chunk also covers the event-free range between its final event timestamp and the wall-clock sync high-water.

A failed worker rebuilds from accepted state rather than retrying a frozen payload. A handled send failure leaves latest_synced_at unchanged and schedules a delayed retry with backoff. A worker crash is recovered when the Mathspace task lease becomes stale.

Shared Push Protocol

Mathspace pushes initial and delta sync chunks to:

POST /v4/problem-events

Example chunk:

{
  "user_id": "123",
  "catalog_id": "catalog-id",
  "previous_watermark": null,
  "watermark": 1779850800.123,
  "events": [
    {
      "timestamp": 1779850800.123,
      "correct": true,
      "sana_topic_id": "topic-id",
      "problem_template_id": 12345,
      "difficulty": 0.42,
      "discrimination": 1.0
    }
  ]
}

Fields:

• user_id: existing Sana-facing user id
• catalog_id: existing Sana-facing curriculum/catalog id
• previous_watermark: partition watermark required immediately before this chunk, or null for the first chunk of an initial sync
• watermark: safe partition watermark after this chunk is durably applied; an intermediate chunk uses its latest complete event timestamp, while the last chunk uses the wall-clock sync high-water
• events: events in the current GPE shape

After a chunk succeeds, the next chunk uses its watermark as previous_watermark. Intermediate chunk boundaries are therefore tied to event timestamps. Mathspace sets the last request's watermark to the wall-clock time captured when the sync started, even when the latest event in that request has an earlier timestamp. If the sync has no events, that last request has an empty events list. In either case, the watermark records the completed source-scan boundary independently of the events in the request.

Every accepted chunk is immediately readable up to its safe watermark. After new-sana has accepted through the sync high-water, Mathspace can advance latest_synced_at to the same wall-clock time.

On a pre-write watermark mismatch, new-sana rejects without writing:

{
  "error": "watermark_mismatch",
  "acceptable_previous_watermark": 1779850900.0
}

A failed response after possible partial application returns the current acceptable_previous_watermark. Mathspace rebuilds work from that boundary. If the returned value is null, the partition was reset and Mathspace reinitialises its sync state.

The endpoint uses the existing new-sana service authentication. tenant_id comes from new-sana configuration and is not part of the payload.

Persisted Problem Event Store Implementation

Event Partition

Problem events use catalog_id in the Cassandra partition key so completeness and writes apply to one student/catalog:

PRIMARY KEY ((tenant_id, user_id, catalog_id), timestamp, problem_template_id, sana_topic_id)

The partition has a static completeness field:

watermark double | null

A null watermark means incomplete. A non-null watermark means problem events are complete through that source-scan boundary. Intermediate values are complete event timestamps; the last chunk writes the wall-clock sync high-water, which may be later than every stored event timestamp.

Event rows and watermark progress are written atomically, so rows should not exist with a null watermark. If they do, new-sana reports the invariant violation for investigation.

The event merge identity is:

tenant_id + user_id + catalog_id + timestamp + problem_template_id + sana_topic_id

This identity relies on the existing constraints that a problem template has one entry for a LanternSkill, a student problem has a unique timestamp, LanternSkills are not shared across catalogs, and topic IDs are unique within a catalog. Multiple topic mappings from one source row can legitimately produce multiple events at the same timestamp.

Duplicate events merge over the existing Cassandra row. Delivery order is not trusted; application order is based on event timestamp. Raw Mathspace timestamp floats are stored unchanged. ProblemEventStore coerces them to new-sana's millisecond datetime representation for consumers, matching the current GPE path.

Unknown catalog or topic events are stored. Unresolved events are not selected by problem-event consumers.

Writer Coordination And Ingestion

Sync and reset writers are serialised per (tenant_id, user_id, catalog_id) through a separate Cassandra coordination table:

PRIMARY KEY ((tenant_id, user_id, catalog_id))

The lock row contains an opaque owner id and its most recent acquisition or refresh timestamp. Acquisition uses a conditional insert. An existing lock may only be replaced once its timestamp is older than the configured grace period. Refresh and release are conditional on the caller still owning that id, so a stale owner cannot refresh or delete its replacement.

Each ingestion request creates an internal owner id, holds the partition lease while applying its chunk, and releases the lease before returning. Reset uses the same lease for the duration of its operation. Another ingestion request or reset receives a retryable lock response until the current owner finishes or becomes replaceable after the grace period.

For each chunk, new-sana:

1. Creates an internal owner id and acquires the partition lease.
2. Compares the current partition watermark with previous_watermark.
3. Validates event timestamps against the range covered by the chunk.
4. Sorts events and writes bounded, logged single-partition batches. Each batch makes its event rows and static watermark update visible atomically.
5. Keeps all events with the same stored timestamp in the same batch and transport chunk.
6. Advances watermark after each successful batch to the latest complete timestamp durably represented by that batch.
7. Sets the partition watermark to the request watermark after writing the event batches.
8. Releases the lease before returning success.

If a batch fails, the chunk fails and the current watermark remains the repair point.

Reset

A reset targets one (tenant_id, user_id, catalog_id) partition and claims the same writer lease as sync ingestion:

1. Delete rows from newest to oldest in bounded logged batches.
2. In the same batch, set the watermark to the latest event timestamp still remaining, so readers cannot observe deletes without the matching safe watermark.
3. When no events remain, set watermark = null and release the lease.
4. Stop before another batch if lease ownership is lost.

Other sync writers cannot accept events while reset owns the lease. A stale Mathspace task is rejected by the partition watermark check or its Mathspace running_version check and cannot advance latest_synced_at.

After reset, a null-partition request follows the calculation behaviour for its current phase and triggers a new async initial sync.

Calculation Integration

The shared problem-event reader is used by every calculation that includes problem events, including adaptive-engine, status, history, tracking, report, snapshot-generation, and background flows.

Read Selection And Incomplete Partitions

For a non-null partition watermark, the reader returns persisted events up to that watermark. A persisted-store read failure fails the request; it does not fall back to GPE.

For a null watermark:

• In Phase 1, and for Phase 2 requests using the GPE path, the reader uses GPE for the current calculation. Every cursorless GPE request schedules a separate async backfill, while cursor-bearing pagination requests do not. The backfill task reads the canonical sources.
• For Phase 2 requests using the incomplete-partition path, and for every Phase 3 request, the calculation uses only the available Lantern events without problem-event history and does not persist a snapshot.

The incomplete-partition path:

1. Calculates a best-effort result only from available Lantern events.
2. Calls the Mathspace async-sync trigger endpoint with the Sana-facing user_id and catalog_id.
3. Mathspace coalesces the trigger into an async initial sync whose first push uses previous_watermark: null.
4. Reports a trigger-call failure without failing the calculation. A later null-partition request tries again.
5. Does not persist a snapshot.
6. Returns the best-effort result.

Accepted chunks become readable immediately up to their watermark, so reporting improves progressively while the initial sync runs.

A new student should have little or no history, so initial reporting remains close to real time. A student returning after a long period of inactivity may have a large history and experiences slower reporting while the cold partition is populated.

Snapshot Safety

Snapshots are timestamped with the latest event included in their model-fitting context. That timestamp may come from any event source.

Lantern events and problem events have different availability:

• Lantern events are pushed synchronously and are available from the existing persisted store without projection.
• Problem events are projected and pushed asynchronously. Their persisted watermark records how far that stream is complete.

A calculation can therefore see a later Lantern event before an earlier problem event has been registered. If it writes a snapshot at the Lantern event's timestamp, a later calculation starting from that snapshot skips the delayed problem event because its timestamp is before the snapshot. That omission is then baked into the snapshot.

The existing snapshot store remains separate:

PRIMARY KEY ((tenant_id, user_id, calculation_method_id), catalog_id, timestamp)

Snapshot handling follows the problem-event read path:

• A calculation using the incomplete-partition path does not persist a snapshot.
• A calculation using persisted problem events may persist only when snapshot_timestamp <= problem_event_watermark.
• A calculation using a complete GPE response may persist a snapshot because GPE supplies the complete bounded problem-event view for that request.

The watermark gate prevents that by allowing a snapshot only when problem events are complete through the snapshot timestamp. Live calculations may still use newer Lantern events; they simply cannot persist a snapshot beyond problem-event coverage.