What to build before you add AI to your product
A demo of an AI feature takes an afternoon. A feature that real users can rely on takes a lot longer. This is the list I’d work through before and during the build.
I build web apps with AI features, so a lot of this comes from the product side, not the model side.
1. Choose one narrow job
“Add an AI assistant” is not a job. “Draft a reply to a customer email using the order history” is. You can test a narrow job, price it and improve it. A broad one is often wrong.
Write the job in one sentence, and write down what the AI should not do.
2. Decide how you’ll know it works
Before building, start with 20 to 50 representative examples: the input, and the output you would accept. Include ambiguous requests and failures as well as the easy cases. This is a starting test set, not a reliability guarantee. Run it each time you change the prompt, the model or the code, and expand it using real failures.
Without a test set, every change is a guess, and a fix for one case quietly breaks another.
Use real data where you can. When I built a Google Keyword Planner tool for ScriptBee’s agent, Google Ads test accounts returned no real keyword data, which made the tool hard to judge until it ran against a real account.
3. Start with the simplest thing
Often one model call with a clear prompt does the job. Reach for tools and multi-step agents only when a single call can’t. Each extra step adds cost, waiting time and a new way to fail.
When you do need an agent, give it a small set of tools with clear names, and limit what each can do. A tool that can read is safer than one that can write, and a tool that can write should ask for confirmation.
Be strict about each tool’s inputs. The model reads the tool’s description and its parameter list, and when those were loose, the agent called my tools with vague inputs. Tight parameter types and plain descriptions fixed more than prompt changes did.
4. Keep a person in the loop for anything that matters
Models make confident mistakes. For anything that sends a message, changes data or spends money, let a person review it first. You can loosen this later, for the cases where the test set shows the feature is reliable.
5. Plan for slow and long tasks
Model calls can take seconds, and agent tasks can take minutes. Don’t make the user stare at a frozen page.
- Show output as it arrives where you can (streaming).
- Run long tasks in the background, with a progress indicator and a way to come back.
- Handle failure: a retry, a clear message and a way to continue.
On ScriptBee, an agent run took two to ten minutes. When the chat ran inside a normal request, a user who refreshed the page lost the answer. I moved chat work into background tasks on Trigger.dev and added a Tasks page, so a run kept going after the tab closed and the user could come back to the result. Most of my work there was starting the run, saving its messages and showing each run on the Tasks page.
6. Set limits and watch the cost
Model usage is billed per use, and a bug or a heavy user can run up a large bill quickly.
- Cap the size of the input.
- Limit how often one user can call the feature.
- Use an enforced spending limit where the provider supports one. A budget alert is not a hard cap, so add limits in your own app as well.
- Log how much each call costs, so you can see the cost per user and per feature.
7. Log what was asked and answered
When something goes wrong, you need enough context to understand the request, the output and the error. Redact sensitive data, restrict access and choose a retention period. Logs help you find cases your test set missed and add them to the next test run.
8. Think about the data you’re sending
What you send to a model provider leaves your system. Decide what is allowed to go: customer names, emails, documents. Read the provider’s data terms. Remove what the task doesn’t need.
9. Make it easy to turn off
Put the feature behind a switch. If the provider has an outage, or the feature misbehaves, you can turn it off without a deploy, and the rest of the product keeps working.
10. Tell users what it is
Say clearly that a feature is AI-generated, and give users a way to correct or report a bad answer.
A sensible order of work
- Write the one-sentence job and the “must not” list.
- Collect the test set.
- Build the simplest version behind a sign-in and a switch.
- Add logging and limits.
- Try it with a few real users.
- Improve using their examples, and add them to the test set.
- Only then widen the scope.
If you’re planning one
Most of this list is product work, and that is the part I build. I build the interface, the model calls, the tools, the background tasks and the data around them. The AI page shows how I’d start, and the form below is the place to describe the job and the data it needs.
Questions people ask
Should I start with an agent or a simple LLM call?
Start with the simplest thing that does the job, usually one model call with a clear prompt. Add tools and multiple steps only when a single call can't do it, because each step adds cost, delay and ways to fail.
How do I know whether the AI feature is good enough?
Start with 20 to 50 representative examples and the answers you would accept, including failure cases. This is a starting point, not proof of reliability. Expand the set as you learn how people use the feature.
Do I need to train my own model?
Almost never for a first version. Hosted models through an API are enough to find out whether people want the feature. Training is a later decision, with a real reason.
Related services
Build AI features and agents
Chat interfaces, agent tools and background tasks that call a model, built into a real product with accounts, data and deployment.