Load test comparison — 2026-08-03 vs 2026-08-09

Comparison of two full load-test pipeline runs on the bridge-api LT environment. Generated from the two report.json artifacts; every number below is copied from them programmatically.

Summary

The two runs are not equivalent, and there are substantial differences between them.

ops/load-testing@aug3 — 2026-08-03, branch ops/load-testing, commit 25a2f54, image lt-25a2f541 — before the Aug 4–6 optimization work

dev@aug9 — 2026-08-09, branch dev, commit 65ef429 (merge of ops/load-testing), image lt-65ef4296 — after it

The two runs are separated by 41 non-merge commits. The merge itself changed no code — dev and ops/load-testing resolve to the same tree — so every difference below comes from those commits, not from the merge.

Run identity

ops/load-testing@aug3 (before) dev@aug9 (after)
Branch ops/load-testing dev
Started 2026-08-03T09:01:06Z 2026-08-09T13:33:48Z
Ended 2026-08-03T10:42:40Z 2026-08-09T15:15:28Z
Image tag lt-25a2f541 lt-65ef4296
Commit 25a2f54 — "ops: lt improvements" 65ef429 — dev tip (merge of ops/load-testing)
Report verdict PASSED PASSED
Scenarios 8 8
Blocking failures 0 0

Both runs used the same environment, so this is a like-for-like comparison:

25a2f54 (Aug 3) predates every optimization commit (Aug 4–6), so ops/load-testing@aug3 is a true pre-enhancement baseline and dev@aug9 contains the full set.

Latency by scenario

ops/load-testing@aug3 = 2026-08-03, image lt-25a2f541 (before) · dev@aug9 = 2026-08-09, image lt-65ef4296 (after)

All values in milliseconds. Lower is better; ✅/❌ mark changes beyond ±5%.

Scenario p50 ops@aug3 p50 dev@aug9 Δ p90 ops@aug3 p90 dev@aug9 Δ p95 ops@aug3 p95 dev@aug9 Δ max ops@aug3 max dev@aug9 Δ
smoke 10.9 13.6 +25% ❌ 90.1 76.3 -15% ✅ 794.5 442.1 -44% ✅ 5631.8 4877.2 -13% ✅
baseline 7.1 7.2 +1% 11.7 11.6 -1% 13.7 13.4 -2% 580.8 909.6 +57% ❌
stress 13.8 16.8 +22% ❌ 56.9 518.2 +811% ❌ 67.7 594.7 +778% ❌ 307 12760.9 +4057% ❌
login-storm 83.9 72.3 -14% ✅ 346.3 213.1 -38% ✅ 403.4 269.9 -33% ✅ 1035.5 675.2 -35% ✅
vlearning 9.6 10.5 +9% ❌ 15.8 17 +8% ❌ 18.3 19.7 +8% ❌ 258.6 226.2 -13% ✅
webhooks 6.6 7.5 +14% ❌ 8.9 10.2 +15% ❌ 9.7 11.3 +16% ❌ 212.2 212.2 0%
queue-cron 64.3 64.5 +0% 116.7 71.4 -39% ✅ 149.3 81.1 -46% ✅ 920.6 275.1 -70% ✅
soak 5.5 6.3 +15% ❌ 8.3 9.6 +16% ❌ 9.5 11.5 +21% ❌ 295.4 309.9 +5%

smoke runs only ~110 requests as a sanity check — its percentiles are noise, not signal. p99 is null in both runs for every scenario, so it is omitted.

Reliability

ops/load-testing@aug3 = 2026-08-03, image lt-25a2f541 (before) · dev@aug9 = 2026-08-09, image lt-65ef4296 (after)

Scenario 5xx ops@aug3 5xx dev@aug9 Err rate ops@aug3 Err rate dev@aug9 Dropped ops@aug3 Dropped dev@aug9
smoke 0 0 0 0 0 0
baseline 0 1 0 <0.0001% 0 0
stress 1 36 <0.0001% 0.0009% 302 7,797
login-storm 2 0 0.001% 0 0 0
vlearning 26 12 0.0021% 0.0009% 160 179
webhooks 0 0 0 0 0 0
queue-cron 0 0 0 0 0 0
soak 0 0 0 0 0 0

The stress regression

Every independent signal moved the same way, which is what makes this credible rather than noise:

Signal ops/load-testing@aug3 dev@aug9 Change
p90 (ms) 56.9 518.2 +811% ❌
p95 (ms) 67.7 594.7 +778% ❌
max (ms) 307 12760.9 +4057% ❌
Throughput (rps) 4397.1 4333.3 -1%
5xx responses 1 36 +3500% ❌
Dropped iterations 302 7,797 +2482% ❌
RDS connections (peak) 323 1030 +219% ❌
RDS CPU (peak %) 72.9 80 +10% ❌
API pods 10 12 +2 pods

dev@aug9 was slower on more pods while delivering less throughput — 12 API pods vs 10, yet lower RPS and an 8.8× worse p95. Extra capacity did not absorb it, which points at a shared bottleneck rather than per-pod saturation. The 3.2× jump in peak RDS connections is consistent with requests holding database connections for longer.

This regression does not reach the verdict. generate-report.mjs:531 filters stress out:

const blocking = report.scenarios.filter((s) => s.name !== 'stress' && s.status === 'FAILED');

stress ramps until something breaks in order to locate the knee, so its threshold breaches are treated as informational. Both runs therefore report PASSED despite the degradation above.

Infrastructure

ops/load-testing@aug3 = 2026-08-03, image lt-25a2f541 (before) · dev@aug9 = 2026-08-09, image lt-65ef4296 (after)

Scenario RDS CPU% ops@aug3 RDS CPU% dev@aug9 RDS conns ops@aug3 RDS conns dev@aug9 Pods ops@aug3 Pods dev@aug9
smoke — 1.6 — 7 10 10
baseline 39.7 43.7 202 167 10 10
stress 72.9 80 323 1030 10 12
login-storm 23 15.4 210 188 14 14
vlearning 59.5 62.1 263 265 10 10
webhooks 8.6 8.9 195 211 10 10
queue-cron 66.5 28.8 180 104 10 10
soak 19.5 23.6 118 103 10 10

Slow queries

From the RDS slow query log, digested and ranked by cumulative time (top 8 each).

ops/load-testing@aug3

Count Max (s) Total (s) Statement
5 0.822 3.93 SELECT user.id AS user_id, user.referred_by AS user_referred_by FROM users user WHERE user
2 1.714 3.4 SELECT user.id AS user_id, user.email AS user_email FROM users user LEFT JOIN email_notifi
1 3.127 3.13 SELECT COUNT(DISTINCT user.id) AS count FROM users user INNER JOIN users referral ON refer
1 3.021 3.02 SELECT user.id AS id FROM users user WHERE (user.is_deleted = false OR (user.is_deleted =
2 1.152 2.07 SELECT COUNT(N) AS cnt FROM users User WHERE ((User.referred_by > N) AND (User.is_deleted
2 1.042 1.8 SELECT user.country AS country, COUNT(*) AS members FROM users user WHERE user.role = '?'
2 0.79 1.48 SELECT COUNT(DISTINCT user.country) AS total FROM users user WHERE user.role = '?' AND use
1 1.344 1.34 SELECT COUNT(N) AS cnt FROM users user WHERE (user.is_deleted = false OR (user.is_deleted

dev@aug9

Count Max (s) Total (s) Statement
200 0.854 119.58 INSERT INTO users (id,firstName,lastName,email,mobile_number,country,password,email_verifi
5 1.376 4.9 SELECT user.id AS user_id, user.referred_by AS user_referred_by FROM users user WHERE user
2 2.797 4.81 SELECT user.id AS user_id, user.email AS user_email FROM users user LEFT JOIN email_notifi
2 2.574 4.3 SELECT user.country AS country, COUNT(*) AS members FROM users user WHERE user.role = '?'
2 2.284 3.85 SELECT COUNT(*) AS total_members, COUNT(DISTINCT CASE WHEN user.country IS NOT NULL AND us
1 2.772 2.77 SELECT u.id FROM (( SELECT /*+ INDEX(users idx_users_first_name) */ id, createdAt AS sort_
1 2.506 2.51 SELECT COUNT(*) AS total FROM (( SELECT /*+ INDEX(users idx_users_first_name) */ id FROM u
1 2.155 2.16 SELECT COUNT(N) AS cnt FROM users User WHERE ((User.role = '?') AND (User.is_deleted = fal

The dev@aug9 table is dominated by an entry with no counterpart in ops/load-testing@aug3: 200 slow INSERT INTO users totalling ~120 s, roughly 24× the next-worst statement. The ops/load-testing@aug3 The ops/load-testing@aug3 slow log contains only SELECTs.

Likely cause (hypothesis)

Between the two runs, these migrations landed on the users table:

The last one adds a STORED generated column plus an index on it:

ALTER TABLE `users` ADD COLUMN `admin_visible` TINYINT(1)
  GENERATED ALWAYS AS (`is_deleted` = 0 OR `status` IN ('Failed','Self_Removed')) STORED
ALTER TABLE `users` ADD INDEX `idx_users_admin_visible_created` (`admin_visible`, `createdAt`)

A STORED generated column is materialised and indexed on every insert and update, so it adds write amplification to the hottest write path. stress is the registration-heavy ramp, which makes it the scenario most exposed to slower users inserts. Slower inserts hold DB connections longer, which fits the 3.2× peak-connection jump and the latency collapse.

Status: strongly supported, not proven. The slow-insert entry and the connection spike are measured facts; the causal link to the generated column is inference. The read-side gains those indexes were built for are real and visible (queue-cron, login-storm), so this is a trade-off to measure, not an obvious mistake.

How to confirm

  1. Re-run only stress with idx_users_admin_visible_created dropped and compare p95 — LT_SCENARIOS=stress keeps it to a short run.
  2. Or time a bulk INSERT INTO users on a scratch copy with and without the column/index.
  3. If confirmed, options are: make the column VIRTUAL (not indexable the same way), narrow the index, or accept the write cost if admin query latency matters more than registration throughput.

Caveats when reading these numbers

Sources