Copy to clipboardShould you read this?
If you read my earlier piece on building ZeidBot with LangChain, this one might make for interesting reading too. ZeidBot (a chatbot which answers questions about me) has been completely gutted and rebuilt and he is now harder, better, faster and stronger 🤖. If you didn't read that old article, don’t worry, you can read this and get the gist.
I am far from an authority in building AI systems, but you can play around with ZeidBot and make your mind up for yourself how good a job I’ve done. You can even go one step further and ask ZeidBot to summarise everything that I’ve written about him, you don’t even have to read the rest of this article - or anything on this website for that matter, just ask ZeidBot! Just please don’t abuse the poor guy - my friends took great pleasure in being nasty to him and the trauma has stuck…
TLDR: ZeidBot used to be a RAG pipeline with a vector database. It's now an agent with a compressed context file. It's better, faster, cheaper and about a third of the code. Why rebuild? Because AI got so much better in a matter of a few months and ZeidBot had to keep up.
Copy to clipboardWhat this article is and isn't
This article goes through the reasoning behind rebuilding ZeidBot: how context windows got bigger, how tool usage improved, what the efficiency gains actually were, and how the new system works end-to-end. It isn't a walkthrough, and it isn't an argument that other approaches are all now dead.
Copy to clipboardZeidBot v1: a child of small context windows
When I built the first ZeidBot, the architecture was a result of the constraints of the time. The models of that era could only handle a few thousand tokens of context. My site's content — every blog post, career page and education record — is far bigger than that. The AI model powering ZeidBot couldn’t handle all that context, so the context needed to be pre-processed and at response time only selected bits of curated context would be sent to the AI model to help ZeidBot answer.
That machinery was the whole architecture. The content on zeidbsaibes.com (blog articles, my career pages etc.) was chopped into 600-character chunks, each chunk embedded into a 3,072-dimensional vector with OpenAI's text-embedding-3-large, and everything stored in a Pinecone vector database. At question time, a classification step guessed the topic, a similarity search fished out the most relevant chunks, and gpt-4o-mini answered from those fragments. LangChain and LangGraph orchestrated the chain; LangSmith watched it run.
It worked, and building it taught me a lot, but all of that machinery was just a workaround for the context limitations at the time. Chunking, embedding, similarity search — none of it exists because it's the best way to understand information; it exists because of economic constraints imposed by the AI models at the time.
Copy to clipboardAI Economics: context windows and prompt caching
Sending context to an AI model used to be like sending an envelope through a letterbox and waiting for an envelope-sized response back. Then the little letterbox expanded into a pallet loading bay and then into the Suez Canal, big enough to fit a supertanker’s worth of context through. In two years, mainstream context windows went from 4–32k tokens to 200k and beyond — some models now accept over a million. Token costs have fallen dramatically enough to allow for enormous amounts of context to be shared without racking up the costs.
Alongside cheaper tokens AI providers have also introduced prompt caching. In simple terms if the bulk of your prompt is identical between requests, the AI provider caches it and charges a fraction for the repeat reads. More often than not you are sending similar bits and pieces along with each prompt (like instructions for the response, context summaries, chat history summaries etc.). For ZeidBot, these two changes flipped the economics entirely: sending a large summary of the whole site with every message went from unthinkable to very cheap.
Copy to clipboardZeidBot got tooled up
ZeidBot v1 didn’t really have any tools; it used cosine similarity: a mathematical guess about which context chunks smell most like the question to help inform its responses. Nobody — not the model, not me, the author — ever decided anything; the nearest vectors won. I didn’t have much control over how context got embedded in the vector database and as such what information was deemed to be relevant to the context when ZeidBot was answering questions.
Since that time models have become genuinely good at using tools. “Using a tool” is effectively an AI model deciding to go off and do a bit of relevant work — using a tool — to improve the answer it provides. Tooling has existed since 2023 but what's new is that models can now be trusted with tools. Early models would invent tools that didn't exist or use actual tools poorly. Today's models can more reliably judge: whether to reach for a tool and what to do with the tool. ZeidBot has been given a tool which allows it to retrieve relevant documents from my website’s content management system to provide as context to help its responses. Based on what questions are posed ZeidBot will decide when to reach for that tool.
He kind of works like this: ZeidBot constantly holds a summary digest of everything on the site in his brain, and will first try to answer questions off the top of his head. If a question needs more detail (e.g. "what exactly does Zeid know about JavaScript"), it reviews the summary digest for a list of pages on this site that might contain this information and then uses the getDocument tool to pull all the verbatim content in full from these pages. All this long-form context is then used to form a response to the deep-dive question.
Copy to clipboardZeidBot v2: the technical flow
The new system has two halves, and both are short enough to describe pretty quickly.
Building the summary digest. A button in my content management system admin kicks off the generator. It pulls every published document — posts, career pages, education, main pages — converts the rich-text to markdown, and then uses Claude Haiku to write a 100–180 token summary of each blog post's actual argument — the claims, not just the topic. Career, education and page content stay near-verbatim, because dates and job titles must not be paraphrased. The result is assembled into a single structured markdown document where every entry carries its url — for usage with the tool later if needed. The digest summary is regenerated whenever I publish something new.
Answering questions. Each chat message goes to Claude Haiku. The system prompt is three layers: ZeidBot's persona and grounding rules, then the entire digest (cached), then the conversation. The model answers directly from the summary digest when it can, calls the getDocument tool when it needs the full text, and the streaming UI shows visitors exactly what it's doing — you'll see "Reading: A Better Internet…" flash up before an answer grounded in that post. Retrieval, made visible, turns out to be rather good theatre.
That's it. No graph, no vectors, no pipeline. One document, one model, one tool.
Copy to clipboardThe efficiency gains
The best part of this rebuild is the amount of code and service deletion. Gone: the vector database (and costs), the embedding model, the chunking pipeline, the classification step, the admin buttons for re-embedding new website content, and the perpetual sync problem of keeping thousands of vectors faithful to a changing website. The entire knowledge layer is now one markdown document — I call it the site digest — stored in my CMS where I can read it, version it, and even hand-edit it.
That last point is important: the site digest is very easy for me to understand and customise. When v2 says something odd, I open the digest and look. Debuggability is really underrated.
Latency improved because two hops disappeared (no classification call, no embed-and-search round trip) — streaming starts almost immediately. And cost, counter-intuitively, went down despite sending far more tokens per message: the digest sits in the cached portion of the prompt, so after the first message Anthropic charges roughly a tenth for it, and Claude Haiku's pricing does the rest. The expensive thing in 2023 was tokens. The expensive thing in 2026 is complexity.
Copy to clipboardDon’t lose your RAG
This setup above works because my portfolio website is small. If your context is millions of documents or growing faster than you can summarise it, vector search remains a better approach. The decision rule I'd offer: measure your corpus in tokens, compare it to the context window of the model you'd actually use and the price you'd actually pay with caching, and only build retrieval infrastructure for content that doesn’t fit. The future probably looks like a blend where querying a pre-processed vector database is one of the many tools offered in an agentic AI setup.
Copy to clipboardSumming up
ZeidBot v1 was a workaround for small context windows, and it was the right build for its moment. ZeidBot v2 exists because context windows grew a lot, caching made big prompts cheap, and models learned to use tools. The rebuild deleted a vector database, an embeddings pipeline and two vendor dependencies, replaced them with one human-readable document and one tool, and made the bot faster, cheaper and easier to debug. There's always room for optimisation — but this time, the optimisation was subtraction.
A walkthrough of AI fundamentals which helps contextualise the current excitement around on Retrieval-Augmented Generation (RAG) and Agentic AI Applications.
How I built ZeidBot - a RAG chatbot using Vector DBs, LLMs, embedders and LangChain. Using tools within the AI-stack and building reasoning applications with precision and control.