In this article

  1. The problem with AI vendor evaluation
  2. Before you talk to any vendor
  3. Five questions every vendor must answer
  4. Demo traps and how to avoid them
  5. Data readiness: the question vendors won't ask
  6. Governance and risk
  7. An evaluation scorecard

The problem with AI vendor evaluation

The AI vendor market has a structural problem: the people doing the selling are significantly better informed than the people doing the buying. That asymmetry produces bad decisions - not because buyers are unsophisticated, but because the incentive for vendors to obscure complexity is high, and the standard questions buyers ask can be answered convincingly without revealing the things that actually matter.

The result is a generation of mid-market businesses that have signed contracts for AI tools that they cannot fully use, that require data they do not have, or that solve problems adjacent to the ones they actually face.

"Every AI demo looks transformative. The question is whether the demo represents the problem you actually have, using data that looks like yours, at the scale you operate at."

Before you talk to any vendor

The biggest evaluation mistake is starting with vendors. By the time you have seen three impressive demos, your framing of the problem has been shaped by what the vendors have shown you - which is what they are best at, not necessarily what you need.

Before any vendor conversation, your organisation needs to have answered:

  • What specific operational or business problem are we trying to solve? Not "we want to use AI" - a specific, bounded problem with a measurable current cost or friction.
  • What data do we have that is relevant to this problem? Where does it live, who owns it, how clean is it, and how current is it?
  • What does success look like in twelve months? In numbers. Not "improved efficiency" - a specific metric that will have moved.
  • Who in the business will own this? AI tools that do not have a named owner with the authority and time to drive adoption fail consistently.

If you cannot answer all four of these before vendor conversations start, you are not ready to evaluate vendors. You are ready to be sold to.

Five questions every vendor must answer

1. Show me a reference customer in a similar business, with similar data, solving the same problem. Can I speak to them?
Why it matters: Any vendor can produce a reference customer. The test is whether that customer is genuinely comparable to your situation - same data maturity, similar operational context, similar scale. Vendors with strong track records in your specific use case can produce this. Vendors with superficially relevant case studies often cannot.
2. What does the implementation look like? Who does what, over what timeline, at what cost?
Why it matters: AI vendor pricing typically covers the licence or subscription. It rarely covers the implementation, data preparation, change management, and ongoing maintenance cost. The total cost of ownership is often two to four times the headline contract value. Ask for a detailed implementation scope in writing before signing anything.
3. What happens when the output is wrong? How do we know, and what do we do?
Why it matters: AI models produce incorrect outputs. The question is not whether your system will be wrong - it will be. The question is whether the system is designed to make errors visible, catchable, and correctable before they cause downstream harm. Vendors who cannot answer this clearly are asking you to trust a black box.
4. What are the data requirements? What format, volume, frequency, and quality do you need, and what does our data look like against that requirement?
Why it matters: Most AI vendor evaluations collapse at the data stage - after the contract is signed. The vendor's model requires clean, structured, labelled data. Your data is messy, inconsistent, and spread across three systems. Ask the vendor to assess your actual data before the contract, not after.
5. What is your exit strategy if we want to leave in 24 months? Who owns the model, the data, and the outputs?
Why it matters: AI vendor contracts create data dependencies that can be difficult to unwind. If the vendor trains a model on your data, who owns that model? If you switch vendors, can you take your data with you in a usable format? Vendors who are confident in their product answer this easily. Those who are not can be evasive.

Demo traps and how to avoid them

The AI demo is a highly rehearsed performance. It uses the best-case data, the smoothest workflow, and the most impressive output the vendor can find. It is designed to create a sense of inevitability - the feeling that this is clearly the right solution and the only question is timing.

The trap: beautiful interface, opaque model

Many AI tools for mid-market businesses are wrappers around commodity AI models with a well-designed interface on top. The interface is not the product. Ask what model is being used, whether it is proprietary or a foundation model, and what the vendor's actual differentiation is beyond the UX.

The trap: demo data that looks nothing like yours

Ask to run the demo on a sample of your own data. If the vendor cannot or will not accommodate this, that tells you something important about either the product's robustness or their confidence in it with real-world inputs.

The trap: ROI claims with no methodology

Every AI vendor has an ROI calculator. Most of them work by assuming optimistic adoption rates, generous productivity multipliers, and no implementation cost. Ask for the assumptions behind the number. If the vendor cannot show their workings, treat the number as marketing, not analysis.

Data readiness: the question vendors won't ask

AI tools are only as good as the data they run on. This is not a cliche - it is the primary reason AI implementations in mid-market businesses fail. The vendor's model may be excellent. If your data is incomplete, inconsistent, or siloed across disconnected systems, the model will produce unreliable outputs.

Before committing to any AI vendor, commission an honest data readiness assessment. Not from the vendor - from someone independent. The questions that need answering:

  • Where does the relevant data live, and who controls it?
  • How clean is it? What is the error rate, and what causes errors?
  • How consistent is it over time? Is historical data structured the same way as current data?
  • What is missing, and what would it cost to fill the gaps?

If the data readiness work reveals problems - and it usually does - you have two options: fix the data before you deploy the AI, or choose a simpler use case that works with the data you already have. The third option, deploying anyway and hoping, is how expensive AI implementations fail.

Governance and risk

Mid-market businesses deploying AI need a governance framework before they need a contract. The framework does not need to be complex - it needs to address three questions:

  • Who is accountable for AI-generated outputs? If the AI produces a wrong recommendation that affects a customer, a supplier, or a regulatory obligation - who is responsible?
  • How do we audit what the AI is doing? Can we explain, after the fact, why a particular output was produced? For regulated industries, this is not optional.
  • What decisions cannot be delegated to AI? Some decisions require human accountability by regulation or by basic prudence. These need to be identified before deployment, not discovered after an incident.

An evaluation scorecard

Use this to compare vendors after initial conversations. Score each vendor 1-5 on each dimension.

  • Problem fit - how closely does the vendor's solution address your specific, defined problem?
  • Data compatibility - what is the gap between the vendor's data requirements and your current data reality?
  • Reference credibility - are reference customers genuinely comparable? Can you speak to them?
  • Implementation clarity - is the full implementation scope, timeline, and cost defined in writing?
  • Error handling - is the approach to model errors visible, catchable, and correctable?
  • Exit terms - are data ownership and portability clearly defined and acceptable?
  • Total cost of ownership - have you modelled the full cost including implementation, data prep, and internal resource?

A vendor who scores well on the headline demo but poorly on data compatibility, implementation clarity, and exit terms is a vendor who will cost you significantly more than the contract value suggests.

Evaluating AI vendors and want an independent view?

We advise mid-market leadership teams on AI use case prioritisation, data readiness, and vendor evaluation - without a vendor relationship or referral fee.

Book a Scoping Call