“We need to add AI”. I’m sure you’ve heard that at least once in the last 12 months - someone on the board, a customer or maybe just in your own head has said it. However as ever when developing new products and services you need to ask do we really need it? And often the answer will be no. What you’re really needing is a specific job or function in your product and on a model doing something.
AI is undoubtedly powerful and having a profound impact on how and what we make but it’s not always the answer. I’m trying here to share a practical guide or process to decide how and when to add AI to your product.
Start with the job, not the model
Write the functionality you want down in one sentence. If you can’t, stop there. AI won’t clarify a fuzzy problem for you.
Good job statements look like this:
- “Help a support agent draft a reply from our help centre articles.”
- “Suggest three likely invoice categories from a free-text description.”
- “Flag messages that look like phishing so a human can review them.”
Weak ones look like this:
- “Make the app smarter.”
- “Add generative AI for engagement.”
- “Use LLMs somewhere in onboarding.”
- “Show some impact”
If the job is really “summarise this document” or “classify this ticket”, say so. Specific jobs are easier to test, easier to evaluate and easier to cost. These principles and ways of thinking aren’t unique to assessing the need for AI. They’re just good fundamentals for product development like thinking market first, code later. Define the what and the why before you pick a fancy way of doing it i.e. the how.
Ask whether simpler software already wins
Before you reach for an LLM, ask whether rules, search, forms or a plain old if statement would do a better job.
Rules-based options often win when:
- The inputs and outputs are known and finite (status codes, categories you already maintain, fixed templates).
- Users need the same answer every time for the same input (pricing, eligibility, compliance wording you control).
- You can express the logic in tests you already trust.
Here’s an example. A “smart” scheduling assistant that only needs to block out weekends and bank holidays is a calendar rule. You don’t need a language model for that. Likewise, a decent search box over your own docs can beat a chat interface that makes up answers when it can’t find the right one.
AI earns its place when the input is messy (free text, images, audio), when the range of possible outputs is huge, or when the useful answer depends on patterns you can’t write down as a short set of rules. Even then good AI products wrap the model in boring software like evals, validation, permissions, audit logs and a clear fallback.
Check the data you actually have
Models need something to work with. That sounds obvious but can be easily overlooked so consider:
- Availability. Do you have examples of good and bad outcomes for this job? Enough to judge quality properly, not just to demo once?
- Sensitivity. Is the data personal, health-related, financial or commercially confidential? Who’s allowed to send it to a third-party API? What needs to stay in the UK or EU?
- Rights. Can you use customer content for prompts, fine-tuning or evaluation? What do your contracts and privacy notice already allow?
- Freshness. Does the answer need today’s data (stock, bookings, policy changes), or is a static knowledge base enough?
If you don’t own and have right to use the data then stop. If you can’t send the data to an external service without a proper review you need to fix data access and governance first, then come back to it.
Be honest about failure and who reviews it
AI models can make mistakes. That means every AI feature can get things wrong sometimes. The product question is what happens next.
Walk through:
- User impact. Is a wrong answer mildly annoying, or could it cause harm, legal risk or lost money?
- Detectability. Will the user (or your team) notice a bad output quickly?
- Human review. Who checks high-risk outputs? Is that person in the loop by design, or as an afterthought?
- Logging. Can you replay what was sent and returned when something goes wrong? Keep sensitive content out of logs where you can, and decide how long you keep them and who can see them before you ship.
High-stakes areas need a much higher bar. What does high stakes mean? Think health advice, financial guidance, access control, critical business processes or anything that looks like a professional opinion. Putting a human reviewer in the loop helps, but on its own it isn’t always enough. You might also need to constrain what the model can say, or hold off on a generative feature entirely until you can defend what happens when it fails. Low-stakes drafting with “edit before you send” is a very different level of risk.
Insulate yourself from change
Whatever you do evaluate it. Create a repeatable process that the model produces results you can trust. This also then means you can become model agnostic and move between models and providers. Models get deprecated and new models launch. Providers change pricing and capability, how does that affect your product?
What helps with this is thinking:
- Task. One job, one type of user, one definition of success.
- Quality bar. How you’ll score the outputs (a human rubric, exact matches where possible, spot checks). Avoid vanity metrics you can’t explain.
- Cost limit. A rough cost per successful task you’re willing to pay at the volume you expect. Work it out from the pricing page for the model you actually plan to use, rather than an industry average from someone’s slide deck.
- Response time. The longest the user can reasonably wait for this step.
- Fallback. What the product does when the model is down, slow or below your quality bar (hide the feature, fall back to search, queue it for a human, show a plain form).
Checklist: yes, not yet, or no
I always like to provide practical advice so I’ve knocked up this as a quick scorecard to help in the decision making process. Mark each item yes, no or not yet.
Consider a small trial when all of these are true
- The task/feature is specific and valuable even without the AI branding.
- A rules, search or UI approach is clearly weaker for that job.
- You can access suitable data with permissions and safeguards you trust.
- Failure is tolerable for this use, or you have safeguards that match the risk.
- You can run a small test with a quality bar, cost limit, response-time limit and fallback.
- Someone owns monitoring after launch (not “we’ll see”).
Lean towards not yet when
- The job is fuzzy (“make it smarter”, “create impact”).
- The data is sensitive and the governance isn’t sorted.
- You’ve no way to score quality beyond a demo that impressed one meeting.
- Nobody owns review, support or on-call when the model misbehaves.
Lean towards no (for now) when
- Simpler software already meets the need.
- Wrong answers could cause serious harm and you can’t make that risk acceptable.
- You’re mainly chasing a buzzword for a pitch deck rather than solving a user’s problem.
- We have no clear way to manage costs
- We can’t mitigate the risk of prompt injection
What to do next
If you landed on yes, or a clear not yet, the next step is pinning down scope. Which job? What data? What does success look like? What are you leaving out of the MVP?
If you’d like help turning that into a draft specification, wireframes and a high-level plan, book an App Planning Day.
