Most AI vendor relationships do not fail in the contract. They fail in the pilot, when a polished demonstration meets the organization's actual data, people, and constraints. By then the company has already spent political capital, calendar time, and a quiet assumption that the tool would work. The diligence that should have happened earlier gets compressed into a scramble of Slack threads and a hurried security questionnaire.
Executives who have lived through one of these pilots know the pattern. The vendor arrives with a confident narrative. The internal champion is already emotionally invested. The pilot is framed as low risk because it is temporary. Six weeks later, the organization discovers that the product cannot use its document store without expensive custom work, that the pricing model changes once volume appears, or that the only person who can explain a wrong answer works at the vendor and is currently in a different time zone.
Diligence before the pilot is not about being difficult. It is about deciding, while the relationship is still optional, whether the product can survive contact with the business.
What the vendor actually owns versus what you inherit
Start with a question that sounds almost too basic: what, precisely, is the product doing, and what does the vendor still expect your team to supply? Many AI tools present as turnkey while quietly depending on clean data, labeled examples, prompt maintenance, evaluation harnesses, or a human review queue that the buyer has not staffed.
Ask the vendor to walk through a full production path for one of your real workflows, not a sanitized demo case. Who prepares the inputs? Who decides when the output is good enough to act on? Who monitors quality after week four, when the novelty has worn off and the model is simply another system in the stack? If the answer leans heavily on "your team will iterate," treat that as a staffing commitment, not a feature.
A useful test is to request the vendor's own failure cases. Serious vendors can describe where the product breaks: long documents, multilingual inputs, sparse records, adversarial users, or edge cases that look rare until they show up in your highest-value accounts. Vague reassurances about continuous improvement are not answers. Specific failure modes, with concrete mitigations, are.
Also ask who owns the model under the hood. If the product wraps a foundation model from a third party, you need to know which one, how often it changes, and what happens when the underlying provider alters pricing, rate limits, or behavior. Boards and risk committees increasingly want that chain of custody written down. An executive who cannot name the underlying model in a diligence call will struggle later when something goes wrong and counsel asks the same question under less friendly conditions.
Data, retention, and the quiet terms that outlive the pilot
Data handling is where pilots become permanent decisions. Ask where customer data is processed, whether it is used to train or improve any model, how long prompts and outputs are retained, and who can access them inside the vendor's organization. Push for written answers, not a slide titled "enterprise grade security."
If the vendor offers a "zero retention" or "no training" mode, ask what that mode disables. Sometimes the feature that made the demo compelling depends on logging that the enterprise clause removes. Sometimes the safer mode is available only on a higher tier that was not mentioned in the pilot pricing. Either way, you want the tradeoff visible before anyone loads a customer file into the sandbox.
Ask for a plain description of subprocessors. Many AI products route content through multiple services: embedding stores, OCR, speech-to-text, retrieval indexes, logging platforms. Your legal and security teams will eventually need that map. Getting it early prevents the awkward moment when a pilot is already running and someone discovers a subprocessor that your industry cannot use.
For regulated businesses, ask how the vendor supports audit requests. Can they produce a record of what a model saw and produced for a specific case? Can they support a data subject request without a week of engineering archaeology? If the product cannot reconstruct a decision path, assume that limitation will eventually land on your desk, not theirs.
Evaluation, support, and the economics after the honeymoon
Before the pilot starts, agree on how success will be measured in language the business already uses. Accuracy on a vendor-supplied test set is almost never enough. Prefer measures tied to the operating outcome: reduced handle time that survives after the first month, fewer escalations, higher completion rates on a real queue, or a documented reduction in review hours for a named process. Ask the vendor what they consider a failed pilot. If they cannot define failure, they are selling momentum, not a product.
Ask who will support the pilot day to day. Named technical contacts matter more than a branded success program. Clarify response times for incorrect outputs that affect customers, and whether those incidents are treated as support tickets or as model issues that require research. An executive should know, before go-live, whether a bad answer at 4 p.m. on a Friday produces a workaround or a shrug.
Pricing deserves the same bluntness. Ask what the bill looks like at three times the pilot volume, and at ten times. Ask which costs are usage-based, which are seat-based, and which arrive as professional services once the pilot "succeeds." Many organizations approve a modest pilot fee and then discover that production requires a different SKU, a minimum commit, or custom integration work that was politely deferred during the sales process.
Finally, ask what exit looks like. Can you export prompts, evaluation sets, configuration, and historical outputs in a usable form? How long does deprovisioning take? If the relationship ends, does any residual data remain in vendor systems, and under what terms? Exit planning feels premature at the start of a courtship. It is exactly when it should happen, because that is when the vendor is most willing to be clear.
A short diligence memo before the pilot often saves a longer postmortem after it. The memo does not need to be legalistic. It needs to record ownership, data terms, evaluation criteria, support expectations, pricing at scale, and exit mechanics in language a busy executive can defend in a later meeting. The vendors worth working with will answer these questions without flinching. The ones who treat them as hostility have already told you something useful about the partnership ahead.