Date: 18 October 2026. Status: proposed — awaiting owner approval. Supersedes the backlink parts (S5–S7) of PHASE1_INTEL_PLAN.md. Parent roadmap: AUTOPILOT_PLAN.md §4.4.
Non-negotiable product rule: every step in this plan runs inside Growth Console: started from the UI or a schedule, run by the worker or the paired host AI runner, and visible in the UI. No step needs a VS Code chat, a terminal or a
python -m growth_console.intel …command. The CLI stays for debugging only.
| # | Owner experience | Root cause (evidence) |
|---|---|---|
| P1 | "Large keyword set × 4 markets, only ~150 opportunities" | Hard caps in backlinks.py: MAX_COMPETITORS = 15, MAX_PROSPECTS = 100, MAX_PAGE_LOOKUPS = 80. Only one source (competitor referring domains). S3 pulled SERPs for only 921 of 2,916 keyword rows. The keyword universe is 1,071 unique keywords, and only 36 rows have search volume. |
| P2 | "I don't know which keyword an opportunity is for" | intel_backlink_opportunities has no keyword, cluster or market-keyword link. Prospects are domain-level. The target page is guessed from page tokens in target_for(). |
| P3 | "I don't have content that this site would link to" | The competitor page that earned the link (url_to) is never stored. There is no check that we have an equivalent asset, and no way to create one. |
| P4 | "No visibility into processing" | Stages ran in chat or CLI. There is no run record, per-stage counts, funnel, exclusion reasons or cost per run in the UI. |
| P5 | "Not reusable or continuous" | There is no schedule or incremental run. AI calls B/C need a manual host command (.venv/bin/python -m growth_console.intel.ai_batch …). |
| P6 | "List is not actionable" | Findings show seller pages, wrong types and loose target mapping in the top 20 (PHASE1_INTEL_FINDINGS.md). There is no readiness checklist, outreach workflow, follow-up or verification. |
| P7 | Two disconnected backlink modules | /backlinks (backlink_opportunities, research-observation ledger with verification) and /intel/backlinks (intel_backlink_opportunities) don't share data. |
Goal: for each keyword cluster we care about, in each market, the owner can see who earns links for it, where those links come from, which of our pages should earn the same link, and whether that page exists yet. From there they can take the opportunity to a verified, live link without leaving the app. Discovery re-runs on a schedule and shows only what changed.
| Metric | Phase 1 | Target after this plan |
|---|---|---|
| Keyword-market rows feeding discovery | 2,916 (36 with volume) | ~20,000 (≈5k × 4 markets), ≥60% with volume |
| Keywords with SERP evidence | 921 | All tier A + B keywords (§4.2), refreshed monthly |
| Competitors analysed | 15 overall | Up to 50 per market (configurable) |
| Raw link candidates evaluated | ~1,000s, invisible | Tens of thousands, every rejection has a reason code |
| Qualified opportunities | ~110–150 | ≥1,000 qualified, each tied to ≥1 keyword cluster (measured, not guaranteed) |
| Opportunities with a linking page, keyword, our target page and route | Partial | 100% of ready rows |
| Asset gaps turned into content briefs | 0 | Every gap blocking ≥3 opportunities has a brief |
| Manual steps outside the app | All | 0 |
| Placed links verified automatically | 0 | 100%, weekly re-check |
Quality beats volume. A smaller list where every row is explained is better than a long one the owner can't use. The funnel (§6.1) shows both.
Primary persona: Owner/marketer (non-SEO-expert, limited time, B2B organic exporter). Secondary: VA/assistant doing submissions and outreach.
| ID | Scenario | Product flow (all in app) |
|---|---|---|
| E1 | First-time setup | /backlinks/setup wizard: markets → keyword sources (site content, seed products, competitor-derived) → competitor review (confirm, add, remove) → budget and caps → Estimate (dry run, $0) → Run discovery. |
| E2 | Watch it run | Run Center shows each stage live: status, items in/out, paid requests, cache hits, $ spent, warnings. Failures can be retried per stage, and the cache makes retries free. |
| E3 | Keyword-first discovery ("I want links for organic turmeric powder bulk in DE") | Keyword/Cluster page → our rank and page → competitors ranking → the competitor pages' link sources → opportunities for this cluster → asset readiness. |
| E4 | Prospect-first triage | Opportunities inbox, filtered by market, cluster, type, readiness, authority, new-since-last-run. Bulk accept, reject or snooze. Every row says why it exists. |
| E5 | Asset gap → content ("they link to a competitor's curcumin grading guide; we have nothing") | Asset Gaps board → create brief (pre-filled from competitor page + cluster) → AI content draft (existing ai.content-draft runner) → owner review → repository change/PR (existing content.repository-change) → publish → linked opportunities flip to ready on their own. |
| E6 | Outreach | Opportunity → readiness checklist → AI-personalised draft (host runner) → owner edits/approves → send (Gmail, later) or copy + mark sent → follow-up auto-scheduled → reply logged → placed. |
| E7 | Listings/directories | Listing tasks grouped by directory, with the listing kit pre-filled (copy buttons, image/logo links) and a submit link. Each task is marked submitted, then pending approval, then live. |
| E8 | Reject with learning | Reject reason (seller page, irrelevant, spam, paid only, no contact, duplicate…). The domain, pattern or type is down-weighted or blocklisted and won't come back in future runs. |
| E9 | Continuous mode | Weekly incremental and monthly full runs on a schedule. A digest lists new opportunities, newly unblocked ones (asset published), lost links and budget used. |
| E10 | Verify and monitor | A placed link is auto-verified: page live, link present, rel, anchor, target. It is re-checked weekly. A lost link raises an alert and a re-outreach task. |
| E11 | Impact | Results page: referring domains gained per month, links per cluster, rank movement of the target page's cluster before and after the link, outreach reply/placement rates by type. |
| E12 | Budget safety | Each run shows an estimate first and stops at the per-run/monthly cap. Above the approval threshold the owner must confirm. Data spend is shown per run and per stage. |
jobs system: intel.run (orchestrator) and
intel.stage (one per stage), claimed by the worker. Add these in the
worker.py tick() claim chain. They use the
existing lease, heartbeat, retry and recovery. There is no new process type.scheduled_evaluation. Saved
recipes: weekly-incremental, monthly-full, on-demand. Each has its own scope
(markets, tiers, sources) and budget.ai.intel-classify and ai.outreach-draft, following
the ai.generate/ai.content-draft pattern in AI_RUNNER.md.
If the runner is offline the UI shows "AI enrichment pending". This replaces the
manual ai_batch host command.growth_console/intel/stages.py, which both the CLI
and the job use, so the logic exists in one place only.| Stage | Purpose | Output | Main change from Phase 1 |
|---|---|---|---|
| D1 Keyword universe | Grow from site keywords to ≈5k per market | intel_keywords (+ source) |
Add Labs keyword_ideas / related_keywords seeded by catalogue products, plus keywords_for_site / ranked_keywords on confirmed competitors. Relevance filter (product vocabulary + negative terms). German set per market. |
| D2 Metrics | Volume, CPC, KD, intent | same | Labs bulk metrics as the main source (Ads volume is sparse for long-tail). Missing metrics stay empty. |
| D3 Clustering + tiering | Group keywords into topics; one target page per cluster | intel_clusters, intel_keyword_clusters |
Deterministic lemma/product grouping, then SERP-overlap merge (≥4 shared top-10 URLs). Tier A: primary/high volume; B: mid; C: long tail. AI only names clusters. |
| D4 Page mapping | Map each cluster to our best existing page | intel_cluster_targets |
Uses content front matter + our ranking URL + similarity. Flags no_matching_page (a content gap). Owner override persists. |
| D5 SERPs | Who ranks per cluster/market | intel_serp_results |
All tier A+B keywords (≈ one representative per cluster for C). Monthly refresh; weekly for tier A. |
| D6 Competitors | Real competitors per market and cluster | intel_domains |
Cap configurable (default 50/market). Owner confirmation UI. Classification: owner > rule > AI. |
| D7 Link sources (multi-source, §4.3) | Raw link candidates with exact page and the competitor page they link to | intel_link_candidates |
New: one row per candidate, with source, keyword/cluster provenance and competitor_url_to. |
| D8 Qualification | Filter and score | candidates qualified / rejected(reason) |
Every rejection stores a reason code. No silent caps. |
| D9 Page fetch + type | Crawl the linking page and site for type, route, contact, freshness | candidate enrichment | Same safe fetcher as today (robots, rate limit, SSRF guard). Batched across runs. Detects seller/product pages (P6 fix). |
| D10 Asset match | Does our site have an equivalent of what earned the competitor's link? | asset_status: ready / weak / gap |
Compares the competitor url_to page (title, H1, topic) with our cluster target page. A gap creates or attaches an Asset Gap (§5.3). |
| D11 Opportunity build | Merge candidates into actionable opportunities | link_opportunities (unified, §7) |
Dedup by prospect page; aggregate clusters, keywords, competitors; priority score; drafts. |
| D12 Diff + digest | What's new, unblocked or lost since last run | intel_run_diffs, notification |
Feeds "new since last run" and the weekly digest. |
| Source | How | Why it matters | Indicative cost |
|---|---|---|---|
| S-A Competitor link gap (domain) | backlinks/backlinks per competitor, mode=one_per_domain, limit 1,000 → gives url_from, anchor, dofollow, url_to in one request (replaces 80 single page lookups) |
Broad base, with the competitor page that earned each link | ~$0.06 per competitor; 50×4 markets (deduped ~120) ≈ $7 |
| S-B Keyword/cluster link gap (page) | backlinks/page_intersection on each cluster's top-ranking competitor URLs (≤20 targets) → pages linking to ≥2 of them |
Directly tied to a keyword cluster (fixes P2) | ~$0.03–0.06 per cluster; 300 tier-A/B clusters ≈ $10–18 |
| S-C SERP-native prospects | Free reuse of intel_serp_results: publisher, directory, roundup, association pages that rank for our keywords. Plus footprint SERPs ("organic turmeric" suppliers list, write for us, submit supplier, exhibitor list) |
Pages that already rank for our topics and list suppliers are the most relevant targets | ~$0.0012 per SERP; 2,000 footprint SERPs ≈ $2.4 |
| S-D Broken-link reclamation | Competitor/industry backlinks pointing to 4xx pages (is_broken) |
"Replace the dead link with ours" pitches; also shows which content to recreate | Included in S-A rows |
| S-E Unlinked brand mentions | DataForSEO Content Analysis search for "Growsian" / brand variants, minus pages already linking | Easiest wins | Verify price in dry run |
| S-F Structured sources (curated, maintained) | Export councils, organic associations, certification-body directories, trade-show exhibitor lists, B2B profile listings, per market | Reliable, repeatable listings | $0 (owner-maintained registry in UI) |
| S-G Own link monitoring | Our new and lost referring domains, weekly | Feeds E10 and reclaim tasks | ~$0.06 per run |
Indicative full-run data cost: ≈ $30–45 for the first full build (universe + metrics + SERPs + links), then ≈ $8–15 per month incremental. The UI dry-run estimate before each run is the source of truth. Prices must be re-verified from the live account. Default caps: $25/run, $40/month, owner-editable.
Reason codes (stored per candidate, shown in the funnel and on the domain page):
farm_pattern, spam_score>=N, links_to_most_competitors, seo_tool_name,
junk_tld, competitor_own_site, marketplace (routed to Listings instead of being
dropped), seller_product_page, non_indexable, off_topic, wrong_language,
owner_blocklist, owner_rejected_before, duplicate, no_route.
Priority score (0–100, components visible in a tooltip): cluster value (volume × tier × our rank gap) 30 · topical relevance 20 · authority (rank) 15 · competitor overlap (count, and whether competitors' cluster pages are linked) 15 · attainability (type + route) 10 · asset readiness 10 − spam penalty. Weights live in settings. Owner rejections feed a per-domain/pattern penalty.
Navigation: Backlinks → Overview · Keywords & Clusters · Opportunities ·
Asset Gaps · Outreach · Listings · Placed Links · Runs · Settings.
/intel/* keyword and competitor pages are folded in. The old /backlinks ledger is merged (§7).
ai.content-draft) → review → Create PR
(content.repository-change) → on publish (URL live check) the asset is
ready, the cluster target is updated, and blocked opportunities move to ready.keyword cluster → SERP (who ranks) → competitor → competitor page (url_to) → linking page (url_from, anchor, rel, first seen) → our target page → why scored.
It also shows a page snapshot (title, excerpt, detected type, contacts, submit form),
a readiness checklist (asset live, contact found, draft approved, claims verified),
activity timeline, notes, and similar opportunities on the same domain./ai settings), curated source
registry (S-F).keywords → clusters → SERPs → competitors → raw link candidates → after farm/spam filters → topical → typed + routed → qualified → asset ready / weak / gap → ready to act. Each arrow shows the count dropped and the top reasons.
Every opportunity stores the IDs of the run, source, keywords/cluster, competitor and competitor URL, and the fetched-page snapshot hash. Nothing in the UI is "magic". Each number links to the rows behind it.
opportunity_events (merged with the existing backlink_opportunity_events) logs
every status change, draft edit, send, reply, verification and AI suggestion, with
actor (owner, VA, system, AI) and timestamp.
intel_runs (recipe, scope, trigger, status, budget, spend, started/ended).intel_run_stages (run, stage, status, in/out counts, paid/cached requests,
cost, error, summary JSON).intel_clusters, intel_keyword_clusters, intel_cluster_targets (cluster →
our page, asset status, owner override).intel_link_candidates (run, source, url_from, url_to (competitor page),
competitor, cluster ids, anchor, rel, first_seen, rank, spam, state,
reason_code).link_assets (asset gaps and ready assets: title, format, clusters,
competitor examples, status gap → brief → draft → published, linked
content brief/PR ids, live URL).link_opportunities: merges intel_backlink_opportunities and
backlink_opportunities. Adds clusters, keywords, competitor_url_to, asset_id,
readiness flags, score components, follow-up dates and run provenance. The
existing verification fields and events carry over. Data migration keeps all
current rows and statuses.outreach_messages (draft, approved, sent, follow-up, reply; no automatic sending
without a connection).link_verifications (each check result).intel_domain_overrides (class, blocklist, penalty).| Task | Capability | Batch size | Fallback if offline |
|---|---|---|---|
| Classify ambiguous domains/pages (type, seller vs editorial) | ai.intel-classify |
≤100 rows/job | Rule class + "needs review" |
| Name clusters, propose asset formats for gaps | ai.intel-classify |
≤100 clusters/job | Rule-based names |
| Personalise outreach opening and pitch | ai.outreach-draft |
≤25/job | Deterministic template |
| Content draft for an asset gap | existing ai.content-draft |
1 brief/job | Brief only |
Inputs are public page or SERP data only, with no contacts or credentials. Outputs
are schema-validated and cached. The existing daily cap and single execution slot
apply. The runner is paired once in /ai. After that the owner never runs commands.
ready unless the
owner opts in (sponsored rel expected).| M | Scope | Owner-visible exit criteria |
|---|---|---|
| M1 Pipeline in product + Run Center | intel.run/intel.stage jobs, worker wiring, scheduler, intel_runs/intel_run_stages, Runs UI with Estimate/Run/Retry, budgets. Wrap existing S1–S6 stages as jobs. |
Owner clicks Run discovery and watches stages, counts and cost in the UI. Zero CLI. A scheduled weekly run happens on its own. |
| M2 Keyword linkage + clusters | D3/D4, clusters UI, re-derive keyword provenance for existing opportunities from cached SERP/backlink data ($0) | Every existing opportunity shows its clusters, keywords and market. Cluster detail page works. |
| M3 Scale discovery + funnel | D1 universe expansion, D2 Labs metrics, sources S-A…S-E, intel_link_candidates, reason codes, configurable caps, funnel UI |
≈20k keyword-market rows. ≥1,000 qualified opportunities (or a funnel that explains why not). Drill-down into rejections. |
| M4 Asset match + Asset Gaps → content | D10, link_assets, Asset Gaps board, brief → ai.content-draft → PR → publish check → auto-unblock |
Owner turns a gap into a published page in-app, and its opportunities become ready. |
| M5 Action workspace | Unified link_opportunities migration, opportunity detail with evidence chain, readiness checklist, ai.intel-classify + ai.outreach-draft jobs, outreach queue (copy + mark sent), listings tasks with kit, reject-with-learning |
Owner takes an opportunity from inbox to "contacted" or "submitted" with drafts. Rejections don't come back. |
| M6 Verify, monitor, report, digest | Verification crawler, lost-link alerts, own RD tracking (S-G), Results page, weekly digest, run diffs | Placed links verify themselves. Weekly digest. Impact per cluster. |
| M7 (optional) Gmail sending | Gmail connection (OAuth via existing connections/vault), rate-limited batch send, reply detection | Approved batch sent from the app with threads tracked. |
Build order: M1 → M2 → M3 → (M4 ∥ M5) → M6 → M7. M1 and M2 cost $0 in data. M3 is the first paid run (estimate shown before running).
ready opportunity has: linking page, type, market, ≥1 cluster with keywords,
competitor(s) and competitor page linked, our live target page, route (submit,
email or contact), draft, and priority score components.scripts/check_app.py
suite and new tests pass.| Risk | Mitigation |
|---|---|
| Volume without quality (link farms dominate this niche) | Reason-coded filters, seller-page detection, owner feedback loop, quality-first default views |
| Data spend creep | Dry-run estimate, per-run and monthly caps, cache, tiered refresh (A weekly, B monthly, C quarterly) |
| Asset gaps pile up | Ranked by "opportunities unblocked", effort S/M/L, with a weekly suggested top 3 in the digest |
| Outreach reputation and compliance | L1 batches, weekly cap, one follow-up, opt-out, no paid links by default |
| Host AI runner availability | AI is enrichment-only. Rules always produce a usable result |
| Two legacy backlink modules | One unified model and migration in M5, old routes redirect |