Vectorless RAG: retrieve by reasoning, not by similarity
Why "chunk, embed, top-k" struggles on long financial reports, and how a table-of-contents tree that an LLM navigates can find the right pages instead.
The default recipe for retrieval-augmented generation is simple: split documents into chunks, embed every chunk, embed the question, and hand the model the top-k most similar chunks. It works well for FAQs and short articles. It works much less well on a 300-page annual report.
Where similarity search breaks
Similarity is not the same thing as relevance. A question like "How did net interest margin move this year?" is semantically close to every paragraph that mentions margins, including the glossary, the risk disclaimer and last year's commentary. The paragraph that actually answers it may use different words, or sit in a table.
Chunking makes it worse:
- Tables get cut in half. The row you need ends up in one chunk and its column headers in another.
- References break. "See note 14" points to a chunk the retriever never sees.
- Structure disappears. A human analyst knows the answer lives in Management Discussion → Financial Performance. A flat list of chunks has no idea where it is in the document.
The idea: a table of contents the model can read
Vectorless RAG keeps the document's structure instead of throwing it away.
- Index: build a tree that mirrors the document, like a table of contents. Each node is a section with a short summary and the page range it covers.
- Retrieve: give the model the tree, not the text. It reasons about which branches could contain the answer, opens them, and drills down until it reaches the right pages.
- Answer: read only those pages, and cite them.
This is the approach behind PageIndex, an open-source project from VectifyAI. There is no vector database and no chunking: retrieval is a reasoning step over the tree.
def find_pages(question, node, llm, depth=0):
"""Walk the document tree like an analyst flipping to the right section."""
if not node.children:
return [node.pages]
menu = "\n".join(f"{i}. {c.title}: {c.summary}" for i, c in enumerate(node.children))
picks = llm.choose(
f"Question: {question}\nWhich sections could contain the answer?\n{menu}",
max_choices=2,
)
return [p for i in picks for p in find_pages(question, node.children[i], llm, depth + 1)]What I used it for
In my Equity Research Command Center, a researcher agent works from a master index of page summaries for each company's reports. Before the analyst agent writes anything, the researcher picks the pages that talk about the metrics that matter for that sector, such as NIM, NPA and CASA for a bank, and passes only those pages on. The analyst then has to quote exact figures from them.
The win was less about accuracy on paper and more about traceability: every number in the briefing points back to a page I can open.
Trade-offs
| Vector RAG | Vectorless (tree) RAG | |
|---|---|---|
| Best for | Many short, loosely structured docs | Few long, well-structured docs |
| Query cost | One embedding + a lookup | Several LLM calls while navigating |
| Explainability | "These chunks were similar" | "I opened section 4.2, pages 61–63" |
| Failure mode | Confidently retrieves the wrong paragraph | Picks the wrong branch, but you can see it |
It is not a replacement for vector search everywhere. For a support bot over thousands of short articles, embeddings are cheaper and good enough. For contracts, filings, manuals and policies, where the structure is the meaning, letting the model navigate is often the better retriever. The two also combine well: use vectors to pick the document, then a tree to find the page.
Takeaways
- Ask what a human expert would do. If they would open the table of contents, your retriever probably should too.
- Keep page numbers all the way to the final answer. Citations are the cheapest evaluation tool you have.
- Measure it on your own documents. The PageIndex authors report strong results on FinanceBench, but your reports are not their benchmark.