2026-05-23 · 2 min read

For Nexus, indexing based purely on token size is a fundamentally flawed approach when dealing with codebases. Fixed-length chunking blindly cuts through source files with no awareness of structure — a chunk might start in the middle of a function, span across two unrelated methods, or completely omit the class a method belongs to. You lose critical context: which function does this line belong to? Which class owns this method? What are the imports and exports?

AST-Aware Chunking

The fix is to parse the source code into an Abstract Syntax Tree (AST) and chunk along semantic boundaries instead. Rather than "chunk every 500 tokens," we chunk by nodes — a method, a class, an interface, a module — units that actually mean something.

But a new problem surfaces immediately: some of these nodes are huge. A class with 30 methods will produce an enormous chunk if we embed the whole thing. That's where Recursive AST Chunking comes in.

Recursive Splitting with Token Budgets

The idea is simple: assign a token budget to each chunk. If a node exceeds the budget, don't embed it whole — recurse deeper into its children.

Class too large     →  split into Methods
Method too large    →  split into logical blocks

At the same time, we don't want to over-split and end up with dozens of tiny, context-free fragments. So we enforce both a minimum and maximum token threshold:

if node.tokens < 80 {
  merge with next sibling
}

This keeps chunks meaningful in both directions — not so large they're noisy, not so small they're useless.

One important caveat: the node hierarchy varies by paradigm. Object-oriented languages (classes → methods → blocks), functional code (modules → functions → expressions), and frontend code (components → hooks → JSX trees) each have their own logical structure. The chunker needs to be aware of this.

Preserving Context with Metadata

Chunking by AST node solves the structural problem, but introduces a new one: context loss. If you embed a method in isolation, the embedding has no idea which class it belongs to, what it imports, or what it exports. That information lives outside the node boundary.

The solution is to store rich metadata alongside each chunk:

{
  "repo": "crm-backend",
  "file": "services/payment.ts",
  "language": "typescript",
  "symbol": "PaymentService.createSubscription",
  "symbol_type": "method",
  "parent": "PaymentService",
  "imports": ["StripeClient", "PaymentRepo"],
  "exports": true
}

This metadata doesn't get embedded — it gets stored and used to reconstruct context at retrieval time. When you surface a chunk, you can stitch the parent class, relevant imports, and file path back in before passing it to the LLM.

LLM-Generated Chunk Summaries (Out of Scope for Nexus)

One step further would be passing each chunk through an LLM to generate a natural-language summary and embedding that instead of raw code:

Function: PaymentService.createSubscription
Parameters:
  - userId: string
  - planId: string

Purpose:
  Creates a recurring subscription for a user via Stripe.

Code:
  ...

This dramatically improves retrieval quality — semantic search over summaries tends to outperform search over raw syntax. However, for Nexus we're keeping this out of scope. Running every chunk through an LLM at index time would blow up cost fast.

Parsing: tree-sitter

For multi-language AST parsing, tree-sitter is the right tool. It's fast, incremental, and has grammars for virtually every popular language. For Nexus, we'll scope support down to a handful of widely-used languages and use tree-sitter as the unified parsing layer across all of them.

So the updated chunking strategy now looks like this:

1. File Classification — determine file type and language
2. Source Code Chunking — AST-aware, recursive, with metadata (with the known caveats above)