Why grounded chatbots go stale (and why re-crawling everything is the wrong fix)
September 3, 2026
A chatbot that answers from your website has a second job nobody talks about. The first job -- ground every answer in your content instead of inventing one -- gets all the attention. The second job is quieter and just as important: keep that content current. A bot grounded in *last quarter's* site is still wrong. It just cites its sources while it does it.
You've probably seen the failure. You change a price, retire a product, update a returns policy -- and weeks later the bot is still confidently quoting the old one, footnoted to a page that no longer says that. The answer *looks* trustworthy. It's grounded. It's also out of date, which is its own kind of hallucination -- the confident-and-wrong kind, dressed as confident-and-sourced.
Here's why it happens. Most "chat with your website" tools crawl your site once, at setup: fetch the pages, split them into chunks, turn each chunk into a vector, store it. From then on every answer is retrieved from that store -- which means the store is a photograph of your site on the day you installed the bot. Your site keeps changing. The photograph doesn't. The longer it's been since setup, the more the bot is answering questions about a website that no longer exists.
The obvious fix is to re-crawl on a schedule, and most tools technically offer it -- re-index weekly, or a button to refresh. But watch what the naive version actually does: it re-crawls *every* page and re-embeds *every* chunk, whether or not anything changed. On a 50-page site that's tolerable. Across a few thousand sites it's a real bill -- so the default quietly becomes "weekly," or "on demand," or "never," and the bot rots between refreshes. The cost of the blunt fix is the reason knowledge goes stale. Freshness turns into something you pay extra for, so people don't.
The way out isn't to refresh more often. It's to refresh only what changed -- and it turns out you almost never have to guess, because the web hands you cheap change signals if you ask. Three of them, cheapest first:
1. Let the site tell you (event push). The best signal isn't one you go hunting for -- it's one the site sends you. A WordPress publish hook, a Shopify or Webflow webhook: the moment a page changes, the platform pings you, and you refresh exactly that page, exactly when it changed. Zero polling, zero waste, near-instant. If your CMS can push, nothing beats it.
2. Read the sitemap's timestamps. For everything else, a sitemap usually carries a <lastmod> date per URL. One request to the sitemap tells you *which* pages changed since you last looked, so you fetch only those and skip the rest without downloading them at all. A thousand-page site where three pages changed becomes three fetches, not a thousand.
3. Hash the content you do fetch. Even a page you decided to re-fetch might not have really changed -- a template tweak, a new footer year, a lastmod that bumped for no real reason. So before spending anything to re-embed it, hash the actual text and compare it to last time. Identical? You're done -- no embedding, no cost. Only genuinely new text gets re-vectorized.
Stack those three and the economics invert. **Keeping a site current stops being proportional to the *size* of the site and becomes proportional to how much of it actually *changed*** -- which, most days, is nothing. A daily refresh of a static site costs one sitemap request and zero embeddings. A busy site costs a handful of pages. Across a whole fleet, "refresh everyone, every day" adds up to cents, because you only ever pay for the diff.
And once freshness is nearly free, the tradeoff disappears. There's no reason to stretch to weekly to save money, no reason to leave the bot answering from a month-old snapshot. It just stays current, quietly, on its own.
The mindset shift is the same one that fixes hallucination: a grounded assistant is a discipline, not a model. Grounding keeps it from inventing answers. Freshness keeps the answers it *isn't* inventing from going out of date. Most tools ship the first and skip the second, because the honest version of the second used to be expensive. It isn't anymore. If a bot is going to cite your pages as the source of truth, the least it can do is read the current ones.
(Disclosure: I build one of these, Curio. This is the discipline it enforces -- WordPress changes push instantly, other sites refresh daily using sitemap timestamps and content hashing, and Hugo only ever re-reads the pages that actually moved. I wrote this because "grounded" and "current" are two different promises, and most tools only make the first.)