End to end: one project, four stacks, clone to drawn flow The other documents are organised by topic. This one is organised by time: a single project is ingested, linked and rendered, and at every step you see what runs and what the data looks like afterwards.
Read this first if you are new to the subsystem. The three companions go deeper on one thing each: tree-sitter-walkthrough.md (parsing), flow-views.md (the join and the view internals), frontend-mapping.md (what a "page" is).
On fidelity. Fragments marked ✅ real are copied from live output of project 270 or from this repository. The example project is marked 🔧 composed — no single real project uses all four stacks at once, so the four sources are assembled to exercise every path. Every mechanism described is real either way.
The example project 🔧 One project, four sources. This is normal: a Project has many ProjectSource rows, and linking merges all of them into one graph.
project 42 "acme-platform" │ ├── source 1 repo "web" Next.js 14 App Router (REACT + TYPESCRIPT) ├── source 2 repo "admin" React 18 + React Router (REACT + TYPESCRIPT) ├── source 3 repo "orders-api" Spring Boot 3 (JAVA) └── source 4 repo "billing-api" FastAPI (PYTHON) Files that matter:
web/ src/app/(shop)/orders/page.tsx ← Next.js route /orders src/hooks/use-orders.ts ← the hook that actually fetches src/api/client.ts ← axios instance, baseURL "/api"
admin/
src/App.tsx ← <Route path="/refunds" element={
orders-api/ src/main/java/com/acme/OrderController.java ← @RestController @RequestMapping("/api/orders") src/main/java/com/acme/OrderService.java src/main/java/com/acme/OrderRepository.java ← extends JpaRepository<Order, Long> src/main/java/com/acme/model/Order.java
billing-api/ app/api/v1/invoices/router.py ← APIRouter(prefix="/invoices") app/services/invoice_service.py app/repositories/invoice_repository.py app/models/invoice.py ← class Invoice(Base) Phase 1 — Ingest Runs once per source. Four POST /api/v1/project-source calls, one per repo. Each is independent: nothing is joined yet, and no repo knows the others exist.
Step 1 — the request arrives ProjectSourceController → ProjectSourceService.ingest(request, zipFile, projectId)
{ "repoName": "orders-api", "gitUrl": "https://github.com/acme/orders-api", "language": "java" }
Step 2 — get the code onto disk
cloneRepository(gitUrl, repoName, provider, token, user) → JGit clone into temp-clones/orders-api-
The random suffix is why Step 5 must relativise paths — node ids hash the file path, and a different clone directory each run would change every id.
Step 3 — pick the scan root resolveScanRoot(folder, repo) → if folder/repo is a directory it wins, else folder. Lets a monorepo be ingested one subfolder at a time.
Step 4 — find the files, decide the language FileDiscoveryService.discover(scanRoot, "java") → List<DiscoveredFile{path, language}>
codekg.ignore-patterns drops /target/, /node_modules/, /.git/, … LanguageDetector.detect(path) is a pure extension lookup: .java→JAVA, .py→PYTHON, .tsx→REACT, .ts→TYPESCRIPT the filter is a family: declaring web as react admits REACT, TYPESCRIPT and JAVASCRIPT, because the API client lives in .ts and dropping it would lose every URL What each source yields:
Source Declared Languages admitted web react REACT (.tsx), TYPESCRIPT (.ts) admin react REACT, TYPESCRIPT orders-api java JAVA billing-api python PYTHON Step 5 — parse and extract, 10 files at a time parseFile(df, scanRoot, repoId, commitHash), on a fixed pool of 10:
String relativePath = relativizeToScanRoot(scanRootPath, df.path); // "src/main/java/com/acme/OrderController.java" TSParser parser = parserFactory.getParser(df.language); TSTree tree = parser.parseString(null, sourceString); ... LanguageAstExtractor extractor = extractorMap.get(df.language); ExtractionResult result = extractor.extract(ctx); extractorMap was built at startup by @PostConstruct initExtractorMap() from every bean's supports(). One extractor per language, enforced by Collectors.toMap throwing on a duplicate.
What each stack produces here — details in tree-sitter-walkthrough.md:
File Extractor Detection Result OrderController.java JavaAstExtractor readAnnotations → RouteRegistry annotation table endpoints: ["GET /api/orders", "POST /api/orders"], entity_type: ctrl OrderRepository.java JavaAstExtractor walkTypeIdentifiers on extends JpaRepository<Order, Long> implements_entities → Order router.py PythonAstExtractor decorator table + APIRouter(prefix="/invoices") endpoints: ["GET /invoices/{invoice_id}"] invoice.py PythonAstExtractor — a CLASS node; DB tier comes later from the /models/ path token orders/page.tsx ReactAstExtractor LayerModel.screenRoute(path) screen_route: "/orders" use-orders.ts TypeScriptAstExtractor extractApiUrl → normalizeUrl api_calls: ["/orders"] — the client's baseURL is not in the source, so only the suffix is recorded admin/src/App.tsx ReactAstExtractor collectRouteDeclarations route_decl_refs: {"ROUTES.REFUNDS": "RefundsPage"} admin/src/data/routes.ts TypeScriptAstExtractor collectRouteTable route_table: {"ROUTES.REFUNDS": "/refunds"} Note RefundsPage.tsx gets nothing at this stage. Its route is declared in App.tsx and its name in routes.ts — three files that only meet in Phase 2.
Step 6 — resolve what can be resolved within one repo Method Adds SymbolTableBuilder.build(parsedFiles) Language → {simpleName → nodeId} ReferenceResolver.resolve(...) CALLS, EXTENDS, IMPLEMENTS, REFERENCES_TYPE edges — in-repo only ModuleImportResolver.resolve(...) file-to-file IMPORTS edges dedupeById first-wins collision drop the childrenByParent loop back-fills code_slots GraphEnricher.enrich(graph) rolls each method's calls up to its class → class_calls_entities, called_by This is why OrderController → OrderService exists but orders/page.tsx → OrderController does not: they are in different repos, and nothing here crosses a repo boundary.
Step 7 — write the YAML YamlGraphWriter.write(graph) → output/codekg/graph_orders-api.yaml
✅ Real, a node from project 270 at this stage (all non-null fields):
type: CLASS name: Home language: REACT startLine: 8 endLine: 10 contentHash: 4d5dbea8a75143a99df0eba7f26fcc24 commitHash: 0b4113ce1c7f770a179670350defb9dd09b97979 parsedAt: 2026-09-07T15:41:03.853625560Z signature: Home() metadata: {componentKind: functional} node_id: 5cfc95111b0c6496 fully_qualified_name: page.Home repo: front file_path: web/src/app/page.tsx source_type: source_code entity_type: other node_id is a 16-hex-char SHA-256 prefix of repo|relativePath|NodeType|qualifiedName (NodeIdGenerator).
Step 8 — upload, and mark the project stale
String s3Key = s3StorageService.buildKey(S3StorageService.PREFIX_SOURCE, "source_nodes.yaml");
String s3Uri = s3StorageService.uploadFile(Path.of(result.getSourceNodesPath()), s3Key);
...
projectRepository.updateIsOutdatedGraph(projectId, true);
A naming trap: the file on disk is graph_
After Phase 1 there are four independent graphs. No frontend node points at any backend node. is_outdated_graph = true says so.
Phase 2 — Link Runs once per project. POST /api/v1/ingest/project/42 → IngestionController.ingestProject → LinkageEngineService.initiateProjectIngestion. This is the only trigger. Nothing below refreshes without it, and every read API serves whatever the last pass produced.
Step 9 — merge the four sources initiateProjectIngestion downloads each source's YAML; mergeAndWriteYaml concatenates their nodes, edges and connections into one source_nodes.yaml, dropping duplicate ids.
Now — and only now — nodes from web and orders-api are in the same map.
Step 10 — load, and fold edges into slots loadAllNodes(dir) → Map<node_id, Map<String,Object>>. Two normalisations matter:
the tree-sitter field names are aliased to the legacy ones (id→node_id, qualifiedName→fully_qualified_name, repoId→repo, filePath→file_path), defaulting source_type: source_code edges: are folded into slots on the source node via EDGE_TYPE_TO_SLOT: IMPORTS→imports_entities, CALLS→calls_entities, IMPLEMENTS→implements_entities, … From here on there are no edge objects — only lists of ids in slots. That is the graph's real shape.
Step 11 — resolve route-table references resolveRouteRefs(codeNodes). Must run before Step 12, because it is what turns a constant into a URL.
For the admin app:
1 routes.ts has route_table: {"ROUTES.REFUNDS": "/refunds"} table key via propertyKey → "REFUNDS" 2 a page has route_refs: ["ROUTES.REFUNDS"] propertyKey → "REFUNDS" → "/refunds" 3 isBackendCall("/refunds", endpointIndex) false — it is a page route, so it is not added to api_calls; Step 14 picks it up instead For a real API constant (ROUTES.INVOICES: "/api/v1/invoices") the answer is true and the URL is appended to the referencing node's api_calls, indistinguishable from a literal fetch.
Step 12 — the join: frontend → backend linkBackendUi(codeNodes, allNodes). This is the step the whole system exists for.
12a. Build the index. Every node with a non-null endpoints, whatever its language:
endpointIndex.put(normPath(stripVerb(String.valueOf(ep))), e.getKey()); Contributed by Endpoint string Index key OrderController.java (Java) GET /api/orders /api/orders OrderController.java (Java) POST /api/orders /api/orders router.py (Python) GET /invoices/{invoice_id} /invoices/{invoice_id} stripVerb removes ^[A-Z]+\s+; normPath drops the query string and fragment, lowercases, and strips one trailing slash. Nothing here knows which framework produced which string.
12b. Match every api_calls entry. Exact hash lookup first, then tail matching with the longest declared key winning.
use-orders.ts recorded /orders, because the axios baseURL is configuration and not in the source. So:
Attempt Result endpointIndex.get("/orders") miss — the declaration is /api/orders matchesTail("/api/orders", "/orders") segments [api, orders] vs [orders], overlap 1, orders≡orders → true 12c. Write the edge — two reciprocal list entries, and that is the edge:
addStructuralEdge(e.getValue(), "calls_backend", backendId); addStructuralEdge(allNodes.get(backendId), "serves_ui", e.getKey()); ✅ Real, a linked frontend node from project 270 after this step:
node_id: b78048262a92741d name: WmThemePromptModal language: REACT repo: front file_path: web/src/components/engagements/wm-theme-prompt-modal.tsx api_calls: ['/api/a2a/{}/vcs/{}', '/api/a2a/{}/reply'] calls_backend: ['72edf31e1451d03a', '7aa6db917d84effb'] imports_entities: ['dc5d44928fc5d261', 'eac087d64644aed2', ...] calls_entities: ['aea191a20d1c508f', 'cd2ba2b9584add13', ...] Worth noticing: that is a component, not a page — wm-theme-prompt-modal.tsx. It holds the calls; Phase 3 folds them into whichever page renders it.
The four combinations, all through the same code:
Frontend Backend Why it matches Next.js /orders Spring GET /api/orders tail match — the client's baseURL is missing from the call Next.js /api/v1/invoices/{} FastAPI GET /invoices/{invoice_id} tail match — the /api/v1 mount prefix is missing from the declaration, and {}/{invoice_id} are both wildcards React /api/orders Spring GET /api/orders exact hash hit React /refunds (nothing) not an API path; handled by Step 14 Rows 1 and 2 are abbreviated on opposite sides, which is exactly why matchesTail is symmetric.
Step 13 — the other structural phases linkInheritance (extends_entities, implements_entities), linkComponentUsage (renders_entities), linkHookUsage, linkMethodCalls (Java field_injections → calls_entities).
linkInheritance is what closes the Spring Data gap: OrderRepository extends JpaRepository<Order, Long> yields implements_entities → Order, which is the only record that the repository persists Order — save/findById are inherited, so no call edge exists at all.
Step 14 — page-to-page navigation linkPageNavigation(codeNodes, nameIndex):
build pageRoutes from the non-API half of every route table → {"ROUTES.REFUNDS": "/refunds"} stampDeclarations(route_decls, …) for literal paths, and stampDeclarations(route_decl_refs, …, pageRoutes::get) for references — resolves ROUTES.REFUNDS → /refunds, finds the node named RefundsPage, and writes screen_route: "/refunds" on it index screens by route: declared stamps first, then screenRoute(file_path) derivations — a node with a stamp contributes no derived entry resolve every nav_targets string against that index with matchesExactly (equal segment counts — a page route is absolute). A hit writes flows_to_entities; a miss writes dangling_nav_targets, because a link to a page that does not exist is a real defect So the admin app's routes exist only after this step, and only because three files were merged in Step 9.
For web, nothing needs stamping — a Next.js route is derivable from the path. ✅ Real: project 270 has 0 nodes with screen_route, and all 26 of its pages are still found. That is not a bug; it is file routing.
Step 15 — cluster and emit nodeClusteringService.cluster(nodes) writes cluster_id (all-or-nothing; skipped on failure), then emitGraph dumps everything to knowledge_base.yaml, uploads it to S3 and stores the path on Project.linkedYmlPath.
After Phase 2 one file holds every node from all four repos, with cross-repo slots populated.
Phase 3 — View Runs on every request. Three endpoints, same four steps each: resolve Project, read linkedYmlPath, download from S3 to a temp file, build(...), delete the temp file.
Step 16 — Backend Flow GET /api/v1/projects/42/backend/flows?depth=6
1 index, drop tests —
2 collapseToModules every symbol → its file, keyed "
APP orders-api └─ CONTROLLER OrderController GET /api/orders └─ SERVICE OrderService └─ REPOSITORY OrderRepository └─ DB Order ← via implements_entities, not a call APP billing-api └─ CONTROLLER router GET /invoices/{invoice_id} └─ SERVICE invoice_service └─ REPOSITORY invoice_repository └─ DB Invoice ✅ Real for project 270: 17 controllers, 117 endpoints, 12 reaching a database.
Step 17 — Frontend Flow GET /api/v1/projects/42/navigation?depth=4
1 index; drop tests, backend and proxies only web and admin survive 3 screenRouteOf(module) — stamp first, then path /orders (path-derived), /refunds (stamped in Step 14) 4 foldSharedComponents a inside a component becomes a link from the page that renders it 5 foldAppChrome the navbar links to pages but is rendered by none → its links credited to every page 6 adjacency from flows_to_entities page → page 9 expand, one slot reserved per page so a dense graph cannot report 26 pages and draw 21 ✅ Real for project 270: 26 pages, 335 links, 5 unreachable. Most of those links come from foldAppChrome, correctly.
Step 18 — Full Flow GET /api/v1/projects/42/user-journeys?depth=6
2 collapseToModules + principalSymbol orders/page.tsx is named OrdersPage, not page 3 markServerModules stamps journey_layer on frontend-language modules reachable from a route — nothing here, but this is what stops an Express backend being a PAGE 5 foldFrontendChain page.tsx → use-orders.ts → client.ts: their api_calls and calls_backend merge into the page 7 pass 1 matchRoute picks the URL; endpointView makes endpoint::GET /api/orders; handlerView narrows the controller to the one handler → INVOKES, ROUTES_TO 8 pass 2 nextLayerTargets, each hop strictly deeper in rank → CALLS 9 findEntryPoints frontend, is a screen, reaches the backend Result:
APP web └─ PAGE OrdersPage /orders └─ ENDPOINT GET /api/orders via_url /orders └─ CONTROLLER OrderController └─ SERVICE OrderService └─ REPOSITORY OrderRepository └─ DB Order Step 5 is the one to remember: page.tsx contains no fetch at all. The call is in a hook two imports away, and folding is what attributes it to the screen.
✅ Real for project 270: 44 journeys, 60 frontend-to-backend hops, 114 leaf paths of which 56 reach a table.
Step 19 — the UI draws it
FlowView → FlowTree → layoutFlowTree(roots, expanded) → one
The fourth view, and a real trap GET /api/v1/projects/{id}/knowledge-graph is a different projection — KnowledgeGraphQueryService, entity-level. ✅ Real for project 270:
632 nodes: CLASS 460, METHOD 172 with endpoints: 0 with screen_route: 0 Zero, on a project whose flow views find 117 endpoints and 26 pages. The cause was ENTITY_NODE_TYPES = {CLASS, INTERFACE, ENUM, METHOD} — it filtered out the MODULE nodes where a page's screen_route/api_calls and a Django or Express router's endpoints live. FUNCTION, STRUCT and MODULE have since been added; the capture above predates that.
The lesson generalises: the flow views read knowledge_base.yaml directly, precisely so they are not subject to another view's filter.
Who contributed what
web (Next.js) admin (React) orders-api (Spring) billing-api (FastAPI)
Route found by file path JSX decl → link time annotation table decorator table
Slot screen_route screen_route (stamped) endpoints endpoints
Also produced api_calls, nav_targets route_table, route_decl_refs implements_entities references_entities
Appears in Backend Flow no no yes, as a root yes, as a root
Appears in Frontend Flow yes yes no no
Appears in Full Flow yes, as a root yes, as a root yes, downstream yes, downstream
The whole thing on one page
FlowTree
view service
LinkageEngineService
S3
SourceIngestionService
ProjectSourceService
You
FlowTree
view service
LinkageEngineService
S3
SourceIngestionService
ProjectSourceService
You
PHASE 1 — once per repo (×4)
PHASE 2 — once per project
PHASE 3 — every request
POST /project-source
cloneRepository
ingestWithTreeSitter
discover → detect → extract → enrich
graph_
Ingest does not link. Four ingests produce four disconnected graphs. Step 9 is a separate HTTP call, and is_outdated_graph is the flag that says so. Cross-repo edges exist only after Step 12, and they are matched by URL string, not by anything a compiler would recognise. If a page shows no endpoint, that match failed. A page rarely holds its own API call. It is in a hook or a client module, and Step 18's fold is what moves it up.