5 min read

We asked AI for a daily update about clients, and it showed us errors in our data

When the AI behind our daily client update got things wrong, the fix was usually in our data, not the prompt. What we learned, and why the next version is an update for each person.

At Growth Advisory, one client often works with several of us, say a consultant for the weekly check-in and someone building their marketing funnel. Before any of us joins a call, we need to know what the client told the others. To keep everyone informed about every client, we have a team meeting on Mondays, and everyone can read the Slack channels we have with each client. That effort grows with every new client, and we’re close to the point where we are stretching ourselves thin.

So we built a daily update, composed by AI. Every night, Potwin, the software we’re building, compares what it knows about each client with the day before: call transcripts, Slack messages, tasks in our project management tool, and activities in our CRM. In the morning, Claude, Anthropic’s AI model, turns the changes into a Slack post in a shared channel for the team. Each client gets a traffic light (red for needing attention, yellow for work in progress, and green for good news). One section in this post names the clients we haven’t heard from in two weeks, because a client who goes quiet is easy to miss.

We expected to fix the AI

When we started, I assumed the work ahead was on the prompt, the instructions we give a generative AI: telling it more precisely what to write or where to set the focus. We started sharing the daily updates with the team, knowing they would have errors, and asked the team to reply whenever something was off. We collected feedback and improved the system for about four weeks.

One of the most important aspects of the daily client update is the list of tasks we have and who on the team needs to perform them. This is where we had most of the errors. For example, when someone made a suggestion during a call, the AI picked it up as a task, even though nobody had agreed to do it. Another problem was something that our client asked us to do and we completed during the call, like booking the next meeting on the calendar, and the AI still listed it as an open task. Each time we fixed one of these problems, we added rules to the prompt for the AI. Effectively, we trained our AI system.

We had to fix the data

Alongside these, we saw a different kind of error, one we had to fix outside the AI. The AI had reported our data faithfully, and the data was wrong:

  • A colleague joined a call through a shared Zoom account, and the transcript credited what they said, including their to-dos, to the account’s owner.
  • A client contact had moved to another company, and our client list still assigned what was discussed with them to their old company, which is no longer our client.
  • One client’s name was spelled differently in two of our systems, so the update couldn’t match them.

Those were errors we had to fix either in how we collected the data or in how the data was generated in the first place.

The hardest case was data that was correct and still misleading. For example, our account manager holds a weekly accountability call, where we check in with the client on their progress with the tasks we agreed on for their roadmap. The call is a task in our CRM, and the account manager ticks it off when the call took place, when the client already spoke with them that week, or when the client has left. In our CRM, all three look the same. However, Potwin’s daily update read every ticked task as a call held, and reported calls that never happened. For the account manager, a ticked task only means that nobody needs to do it anymore. The improved daily update now ignores ticked tasks and reads the recorded call instead.

Many of the data errors were old. They had been in our records long before the daily update messages were created, and were irrelevant because we humans understand the nuances and filter information intuitively. The AI sees only what is in the data, not the context in our heads. The daily update effectively became an audit of our data that we had not anticipated.

The lesson is that when the AI gets something wrong, look at what it was given before tweaking the prompt. Speeding up and improving automations on bad data makes us faster at producing bad outcomes, which goes against our philosophy of how we want to work with our clients.

Correct, but too long

Accuracy had to come first. After four weeks, the update was mostly right. Now we can address the information overload before we grow, because with more clients and a bigger team, sending every update to everyone stops working. Already today, on a busy morning, the update reports on more than twenty clients with a thread of twenty-odd replies, and every person on the team skims all of it to find the few lines that concern them. That is why we are starting to build this system now at our current company size.

In conversation with our head of operations, we discussed how to take what we built and get ready to take on more clients. Different roles need different parts of the update. The leadership team needs to know only what is going very badly so that they can help course-correct. The details about daily work that goes as planned matter mainly to whoever is actively working with a client and needs to take the next action.

An update for each person

The next version gives each person their own update, based on their role at each client. We’ve added the roles to our client list, and the rules are simple. Account managers and project managers get everything about their clients. Marketing and sales work across many clients, so they hear about one only when they have a task for it due within a week, or when someone names them in a call. Management gets every client whose light is red.

Each update gets shorter, and what’s left is what the reader has to act on. We’ll write about how it works once the team has lived with it for a few weeks.