I've reviewed everything. Here's my assessment and questions:.md

I've reviewed everything. Here's my assessment and questions:

D1 Review Summary

The config is well-structured. Key numbers:

53,136 total features (CSV) vs 50,497 (JSON grand_total) — there's a ~2,639 discrepancy. Likely the CSV includes the level1_standalone + ratio items that aren't counted in the JSON interactions block. Not a problem as long as D2 uses the CSV as the authoritative enumeration.
43 overall buckets × 29 venue buckets — sensible split
21 transform types with proper min-obs thresholds
6 interaction categories with domain-grounded pair selection
Implementation guide provides concrete Python for all 12 "new" transforms with performance estimates

Questions on Impact to D2 and Beyond

1. Grand total discrepancy — which number is authoritative?

The JSON says 50,497 but the CSV has 53,136 rows. The difference appears to be the level1_standalone (9,191) + ratio items (21) + cross-sectional prank (6,552) + companion counts (5,824) living in the CSV but the JSON grand_total only counting interactions (31,548) + some subset. For D2 implementation: should the spoke chat treat the CSV as the canonical feature list and just iterate through it, or should it reconstruct from the JSON config programmatically?

2. Execution strategy given 53K features × 1M rows:

The implementation guide flags that expensive transforms (autocorrelation, percentile rank, binned entropy) run at 5K–50K rows/sec. At 1M rows, that's 20–200 seconds per column. With 91 eligible stats, the expensive transforms alone could take 30–100+ minutes per bucket.

The guide recommends limiting expensive transforms to "top 10–15 columns." Has this already been encoded in the config (i.e., do the 53K features already reflect that restriction), or does D2 need to make a runtime decision about which columns get the expensive treatment?

3. Per-market iteration — how does this change D2?

You said D2–D5 will run iteratively per market. But the Level-1 features and companion counts are market-agnostic — they're rolling stats of base columns regardless of what you're predicting. Only the interactions (and later the selection gates) are market-dependent.

Does this mean D2 should be split into:

D2a (once): Generate all Level-1 features (9,191 standalone + 6,552 prank + 5,824 companion counts = ~21,567 features). Market-independent. Run once.
D2b (per market): Generate interactions (31,548) which may need different subsets depending on what the target is (e.g., sums matter more for OU, diffs for 1X2)?

Or do ALL 53K features get generated once, stored as Parquet, and then D3 (gating) is the step that runs per-market?

4. Storage format for 53K columns × 1M rows:

That's ~53 billion cells. Even at 8 bytes per float64, that's ~400 GB uncompressed. The handoff doc mentioned "temporary Parquet (too large for permanent DuckDB columns)." What's the storage plan?

One giant Parquet file (will be very wide — Parquet handles this poorly)?
Column-group Parquet files (e.g., one per interaction category)?
Generate-and-gate in chunks (generate a category, prune immediately, discard raw)?

5. The implementation guide uses min_periods=1 but the D1 spec says strict thresholding (require exactly W observations → else NULL). Which wins?

The implementation guide's rolling_grouped helper uses min_periods=1 and relies on companion counts for downstream reliability weighting. The D1 spec says "if window has fewer observations than window size, return NULL." These are contradictory. The companion-count approach is more flexible (lets the model decide), but the strict-threshold approach is cleaner. Which policy does D2 implement?

6. DuckDB vs pandas for D2 execution:

The implementation guide offers both pandas and DuckDB SQL approaches. Given you're on Bento with the live DuckDB file, DuckDB window functions would be ~10x faster for the vectorized transforms (rolling_mean, rolling_std, momentum, expanding_mean, ACWR, companion_count — probably 70% of features). Should D2 prefer DuckDB SQL where possible and fall back to pandas only for the custom-apply transforms (entropy, autocorr, percentile_rank)?

On First Market for Proof-of-Concept

My recommendation: Over/Under 2.5 (OU).

Reasons:

Binary target — simpler evaluation (AUC, Brier, ECE on 2 classes vs 3 for 1X2)
Strongest expected signal from sum-type interactions — your config has 39 universal sum pairs specifically designed for this market, giving immediate signal validation
Good baseline available — you have v0.7.1a OU metrics to compare against (ECE < 0.0091 target)
High data coverage — OU outcomes are available for essentially every match (unlike BTS which requires goal scorers data)
Moderate complexity — not as "easy" as BTS (which is almost a coin flip at 50/50) but not as hard as 1X2 (3-class)
If OU works, AH generalizes — these two markets share underlying signal (goal expectation), so OU success gives strong confidence AH will follow

Runner-up: Asian Handicap (AH) — also binary, uses difference-type interactions, but requires the handicap line as context which adds complexity.

Avoid for first run: 1X2 (3-class, hardest), BTS (ceiling concerns noted in v0.7.1 — the "floor=0.358" issue).

What are your thoughts on these 6 questions + the OU recommendation?