Guide 04
A chatbot involves three things at once: the information it answers from, a record of each conversation, and a promise about how that record is handled. If the information is wrong, the bot gives confident but incorrect answers. If the record is poor, you cannot tell that this is happening. If the promise is broken, the problem is no longer only a technical one.
When setting up a chatbot's knowledge base (the business information it answers from), many people want to include everything: the website text, the old FAQ, the contents of policy documents, the internal wiki and the price list from the shared drive. It feels thorough, but it is the most reliable way to end up with a bot that tells customers things that used to be true.
The bot does not know which piece of your information is the official version. When two sources disagree, it does not decide between them; it answers from whichever looks most relevant to the question. An old policy left in the material is not harmless, because it will produce a confident wrong answer as soon as someone asks the matching question.
The careful approach is to remove material rather than add it. A small set of current information works better than a large set that is partly out of date. Three practical rules help:
What belongs there instead is the practical material described in the build guide: the replies you already send repeatedly, the specifics your marketing copy avoids, and the awkward questions with an answer you have decided on in advance rather than one the bot improvises.
Before you decide what to include, read how a chatbot knowledge base actually retrieves an answer, which explains why the bot picks information by how relevant it is rather than how recent it is.
When a chatbot says something false, people usually assume the AI invented it. That is occasionally true. Far more often, the bot accurately repeated something you gave it, and that information was out of date.
The difference matters because the fixes are different. Invented answers are prevented by limiting what the bot can draw from and by explicitly allowing it to say it does not know. A bot with no way to admit that it does not know will always produce some answer, because that is the only thing it can do. "I am not sure, let me get you to somebody who is" has to be a fully planned response, not a fallback nobody set up.
Only two routine habits reliably prevent outdated answers. When a price, policy, service or opening hour changes, update the chatbot's information at the same time. And once a quarter, someone should ask the bot the five questions that matter most to your sales and read the answers carefully. This takes fifteen minutes and catches nearly everything.
There is one more kind of poor answer, and it is the one people miss: the answer that is confident but unhelpful. "Our pricing depends on your requirements" is technically accurate, and the visitor leaves. These answers never appear as errors, and the only way to find them is to read transcripts, the written records of each conversation.
Answering in more than one language is one of the few easy improvements here, as long as you understand where the real difficulty is. Answering in Spanish or Polish is close to free. The hard part comes afterward. If a Spanish-speaking customer has a conversation the bot cannot finish, and the person who picks up the escalation (the conversation the bot hands off to a person) speaks only English, the customer is no better served than before.
So plan the handoff before you turn on other languages. If your staff can serve those customers when the conversation gets difficult, turn the languages on. If they cannot, it is more honest to say which languages you support than to offer a bot that chats warmly until a person is needed.
On the language question specifically, running a chatbot in more than one language covers what changes in daily operations rather than in the technology.
Chatbot reports show many numbers, and the ones they display most prominently often matter least. Conversations started is the clearest example. It goes up when you make the widget (the chat window on your site) more prominent, which reflects a design choice rather than a result.
Four numbers tell you almost everything useful.
Satisfaction scores need a caution. Collect them, but do not rely on them heavily. The people who rate a chat are mostly those with strong feelings, and the many who got what they wanted and left do not appear in that number. A thumbs-down with a comment attached is worth more than fifty ratings without one.
One number few reports show is how many people opened the widget, read the opening message and closed it without typing. That number tells you something about your opening message, and it is usually large.
For more on reading chatbot reports, what chatbot analytics tell you about your customers goes beyond the four numbers above.
The numbers above tell you where people struggled. Only the transcripts tell you why, and the reason is usually more ordinary than you would expect. It might be a question worded in a way that seems intrusive, an answer that assumed knowledge the visitor did not have, or a term your industry uses that your customers do not.
The routine that works is short enough that people actually do it. Read twenty conversations a month, chosen at random rather than by rating, because conversations chosen by rating are either the failures or the successes, and the useful material is in between. List every poor answer, fix the three most important, and stop there.
Reading at random matters more than it sounds. Sorting by satisfaction shows you only the best and worst conversations. Most of your visitors are in the ordinary middle, where somebody got a technically adequate answer and quietly decided you were not worth the trouble.
If the numbers show the bot attempting work it should hand over, the AI customer support guide explains where to draw that line, including what a bot should never handle and how the handoff should be worded.
Estimates of a chatbot's return on investment (ROI), meaning what it saves or earns compared with what it costs, are often built from vendor calculations and should be treated with caution, including when the vendor is us. Two numbers can be calculated honestly.
Time saved. Multiply the number of resolved conversations by the minutes each would have taken a person, then by a loaded hourly rate (the wage plus the other costs of employing someone). Be conservative about the minutes, because a question answered in chat would often have been answered in a two-line email rather than a ten-minute call. This number is usually smaller than marketing claims suggest, but it is easier to defend than anything else on the page.
Revenue influenced. Multiply the leads captured through chat by your close rate (the share of leads that become paying customers), then by the average value of a sale. Then reduce the result substantially, because some of those people would have contacted you through the form anyway. It is genuinely hard to know which sales the chatbot caused, and anyone claiming precision is selling something. Cutting the figure in half is a reasonable starting point.
A third effect is real but hard to measure: customers who got an answer at 10pm and did not go to a competitor. You cannot count them. It is reasonable to believe the effect exists while leaving it out of your calculations.
We have worked through a fuller version of this calculation, including the assumptions most open to question, in measuring the return on AI customer service.
Once you have written down real answers to real questions, you have something more useful than a chatbot. You have the content of a help center, a set of pages where customers find answers themselves, and most businesses never notice.
The two overlap almost completely. A carefully chosen chatbot knowledge base is a set of specific questions with plain answers. That is exactly what a self-service portal is, and exactly the kind of content search engines list well. The same forty answers can sit behind the widget, appear as pages people can browse, and show up in search results when somebody searches the question at midnight without ever visiting your site.
The third use deserves the most attention, because it changes the purpose of the writing. An answer written only for a chat window can be brief and can assume the visitor is already on your pricing page. An answer that also has to work as a page needs enough context to make sense on its own. Writing to that second standard takes a little more effort and serves both purposes.
Two cautions apply. Publishing every answer word for word will expose material you meant only for private conversations, so decide one answer at a time. A published help page also carries the risk of outdated information to a larger audience: a wrong chat answer reaches one person, while a wrong page reaches everybody who searches for it, for as long as it stays up.
Taken far enough, this becomes a help center rather than a chat feature, which is the subject of building a self-service portal customers actually use.
Every recommendation on this page assumes somebody is doing it, and the most common reason none of it happens is that the chatbot belongs to whoever set it up, and after a few months that person has often moved on to other work.
The ownership needed is smaller than it sounds. One named person spends thirty minutes a month on three regular tasks: reading twenty random transcripts, updating the knowledge whenever something about the business changes, and confirming that escalations still reach a person. That is the entire job, and the owner need not be the person who built the chatbot.
It does require that the owner hears about changes. The problem is rarely neglect. More often, pricing changed in a meeting the chatbot owner did not attend. Whatever process you use to announce a price change, a new service or a change in hours, add the knowledge base to that list, next to the website and the phone greeting.
A chat window feels casual to the person typing into it, and that is why it collects more than people intend. Visitors put things in chat that they would never put in a form: medical details, financial circumstances, complaints about a named employee and occasionally a password. Your obligations depend on how sensitive the information actually is, not on what you intended the chat to collect.
This short list covers most of it.
None of this is a reason to avoid running a chatbot. It is a reason to decide these things deliberately during setup, when they take an hour, rather than during an incident, when they take much longer and someone else decides the terms.
One more thing should be settled in advance, and it is the only item here with a deadline: what you do when the bot gives a customer a wrong answer. It will happen, and whether it becomes a small problem or a lasting one depends on what you do in the first hour.
Decide now who can turn the chatbot off, and make sure that is more than one person and does not require the vendor. Decide what you will say, which should be the plain truth: the answer was wrong, here is the correct one, and here is what we are doing about it. Also decide what you will honor. If the bot quoted a price it should not have, honoring that price once usually costs less than arguing about it, and much less than a bad review.
Then fix the cause rather than the single answer. The natural reaction is to add a rule that blocks that exact answer. It is better to find the outdated information that produced it, because material that has gone unchecked long enough to cause an incident rarely contains only one wrong fact.
Security and privacy are covered in more detail in what a business is responsible for when a chatbot handles customer data, which goes further into the regulated cases than this guide can.
Include the replies you already type repeatedly, plus the specifics your website avoids: real price ranges, turnaround times, service areas and what you do not do. Choose carefully rather than adding everything. Outdated information does not make the bot go quiet; it makes the bot give wrong answers that sound convincing.
Limit what the bot can draw from, explicitly allow it to say it does not know, and remove outdated material instead of leaving it in place. Then read transcripts. Most wrong answers come from old information rather than from the bot inventing something.
The numbers that matter are resolution rate, escalation rate with the reasons behind it, the outcome you built the bot for, and how quickly staff reply to escalations. Conversations started rises when you make the widget more prominent, which reflects a design choice rather than a result.
Add the time saved, estimated conservatively, to the revenue influenced, reduced substantially because some of those people would have contacted you anyway. Anyone claiming to know exactly which sales came from the chatbot is selling something.
Tell them that it is a bot, what is stored, for how long and who can see it, in the opening message rather than in a policy. Do not collect sensitive information in chat unless you can handle it safely.
Knowledge and reporting
This includes the business knowledge you provide, the email notifications that tell your team about new and hot leads, and the analytics (on the Growth plan and up) that show conversations, messages and the completion rate of each flow.