Enterprise AI procurement has changed. A demo and a price quote no longer settle a deal. Buyers now weigh model behavior, data flows, legal exposure under the EU AI Act, and exit costs hidden in contracts no one reads twice. This guide gives you a working frame for that decision, with trade-offs we have seen first-hand.
Key Takeaways
- Score vendors across six governance dimensions, then let red flags override the average. A high mean score can hide a single failure that breaks the deal.
- Standard platform SLAs rarely cover failures of the underlying model. Negotiate an AI-specific addendum or accept the risk in writing.
- Fine-tuning a third-party model can reclassify you as a provider under the EU AI Act. Check Article 25 before any customization work starts.
- Data preparation eats 50-65% of any real AI project. A vendor who skips this in the proposal is selling a demo, not a deployment.
- Exit terms matter more than headline price. Most enterprises only discover lock-in at renewal, when leverage is already gone.
Governance Categories That Matter When Evaluating AI Vendors
Six categories cover the risk surface for almost any enterprise AI purchase: Transparency, Bias and Fairness, Privacy and Security, Accountability, Regulatory Compliance, and Vendor Lock-in. Each one probes a distinct failure mode. A single average score across all six is dangerous. It lets a strong Transparency rating mask a fatal gap in Privacy and Security.
The practical rule we apply on advisory work is simple. Treat red flags as overrides, not as inputs to a mean. If a vendor stores your prompts and uses them to retrain a shared model, no score on documentation quality compensates. The finding has to be resolved in the contract, accepted as a documented risk, or the vendor drops out.
There is a counter-argument worth respecting. Scorecards can turn into theater. Procurement collects answers no one reads, and the technical team picks the vendor they already wanted. The fix is not to drop the scorecard. The fix is to give it teeth: one named owner per category, evidence required for every claim, and a written escalation path when a red flag fires.
| Category | Core question | Red flag that should stop the deal |
|---|---|---|
| Transparency | Can the vendor produce a model card and limitations note? | No documented behavior, no audit trail |
| Bias and Fairness | Has the model been tested on data that resembles yours? | No third-party bias testing for your use case |
| Privacy and Security | Where does your data go, and who else sees it? | Customer prompts feed shared retraining |
| Accountability | Who owns an incident when the model fails? | No named incident owner on either side |
| Regulatory Compliance | Does the platform support EU AI Act deployer duties? | No published conformity roadmap |
| Vendor Lock-in | Can you export your data and embeddings in open formats? | Proprietary export only, no transition period |
One concrete test we recommend: ask for a completed model card before the first technical demo. Vendors who produce one in 48 hours have a mature engineering culture. Vendors who promise to send one "after the call" rarely deliver something usable, even after signature.
Six questions every AI vendor should answer before shortlist:
- Where does our data live during inference, during logging, and after account termination?
- What is your model version release cadence, and what notification do we get before behavior changes?
- Can you produce a model card and limitations note within 48 hours of a request?
- What happens to our fine-tuned weights, embeddings, and prompt history when we exit the contract?
- Which of our compliance obligations under EU AI Act Article 26 or GDPR Article 22 does your platform technically support today?
- When your underlying foundation model provider changes terms, which of those changes flow through to us, and with what notice?
A vendor who cannot answer these in writing has not built the operational discipline that enterprise procurement now requires.
How Operational and Contractual Terms Shape the Choice
The governance scorecard tells you which vendors deserve a contract. The contract decides whether you survive the next two years. The two documents that matter most are the SLA and the exit clause. Both are usually drafted for a SaaS world that does not match how AI systems actually fail.
Most platform SLAs measure uptime of the wrapper, not the model. If a third-party LLM degrades, hallucinates, or goes offline, the vendor's SLA often holds. The platform itself is technically up. That mismatch is where enterprise risk sits. The fix is an AI-specific addendum that names the model failure modes you care about: accuracy drift on your benchmark, latency under load, and notification windows for retraining or model version changes.
The contract terms that move the needle:
- Audit rights on demand. Not once a year. On request, within a defined window, with access to logs of model inputs and outputs for your tenant.
- Incident SLAs for model failures. Separate clocks for platform outage and model-quality incidents. Define what counts as each.
- Change notifications. Written notice before model version swaps, retraining events, or upstream provider changes. Ideally with the right to test before rollout.
- Exit terms. Data, embeddings, fine-tuned weights where applicable, and prompt history in open formats. A transition assistance period of 30 to 90 days.
- Liability split for harmful outputs. Who pays when the model gives a wrong answer that costs the business money.
One trap catches legal teams late. If you fine-tune or substantially modify a third-party AI system, EU AI Act Article 25 can shift provider duties onto you. That changes your conformity obligations, your documentation burden, and your liability profile. We have seen this missed in two procurement reviews this year alone. Get a written legal read before any fine-tuning contract is signed, and weigh that cost against the accuracy lift you expect to gain.
If build-versus-buy is still open on your side, our guide on custom AI development covers the trade-offs including where fine-tuning a vendor model creates provider-reclassification exposure and where building from scratch avoids it entirely.
For teams who want help pressure-testing the contract before signature, our AI consulting practice runs build-vs-buy reviews that include SLA and exit-clause analysis.
Where the EU AI Act and NIST AI RMF Actually Bite
Direct answer: EU AI Act Article 26 sets binding duties on deployers of high-risk systems in the EU. NIST AI RMF gives US-anchored teams a voluntary operating model. They are not interchangeable. Applying both blindly creates more paperwork than protection.
If your AI system touches the EU market and falls in a high-risk category (credit scoring, employment screening, certain medical uses, critical infrastructure), Article 26 obligations apply: human oversight, operational monitoring, incident reporting to authorities, and log retention. The official EU AI Act text is the source you cite to legal, not a third-party summary. Your vendor's platform either supports these duties technically, or it does not. There is no middle ground.
One 2026 timeline update matters here. In May 2026, the Digital Omnibus on AI agreement deferred Annex III high-risk obligations to December 2027. That widens the runway for compliance readiness, but does not change what vendors must eventually support. If you are evaluating for a high-risk workload today, ask vendors for their compliance roadmap to the extended deadline, not their current state alone. The gap between "we can support Article 26" and "we will support Article 26 in production by 2027" decides whether the platform is a real option.
NIST AI RMF is a different tool. It organizes risk work around four functions: Govern, Map, Measure, Manage. It does not impose legal duties. Its value is operational. It gives a security architect a vocabulary and a checklist for continuous monitoring, which most procurement processes lack.
The pragmatic answer is to pick the framework that matches the jurisdiction and the workload, and use the other only where it adds something. EU regulated deployment: Article 26 first. US-only internal tool: NIST AI RMF is enough. Personal data anywhere in the EU or UK: GDPR sits on top of both, with the ICO's AI guidance as a practical reference for UK contexts.
| Framework | Jurisdiction / scope | Legal force | When to lead with it |
|---|---|---|---|
| EU AI Act, Art. 26 | EU market, high-risk AI deployers | Binding | Any deployment that reaches EU users in a high-risk category |
| NIST AI RMF | Any sector, US-anchored | Voluntary | Continuous risk monitoring and internal governance design |
| GDPR | EU / EEA personal data | Binding | Whenever personal data flows into the model |
| FERPA | US student records | Binding | EdTech and education data only |
One implementation note from real projects. Tamper-evident logging of model inputs and outputs sounds simple in a slide. In production it costs storage, schema work, and a retention policy that legal signs off on. Build that into the cost case before you sign, not after.
How to Run the Evaluation Without Burning a Quarter
Treat the evaluation as a 90-day operating discipline, not a one-off questionnaire. The phasing we use on AI advisory engagements maps to four blocks, with one named owner per block.
Team composition that works:
- A procurement lead who controls the timeline and the contract
- Legal counsel with AI experience to read SLA exclusions, exit clauses, and provider-vs-deployer reclassification risk
- A CISO or security architect to assess data flow, access controls, and incident response
- An AI governance lead who runs the scorecard and tracks red flags across the vendor portfolio
- A subject matter expert from the business unit that will actually use the system
That last role gets skipped most often. It is the most expensive omission. Without a business-side owner, the model has no one to define success, no one to accept its limits, and no one to escalate when it drifts. Our engineering rule is simple: do not start without an SME on the client side.
The 90-day sequence:
- Days 1-30, inventory. List every AI system in use or in pipeline. Classify each against the EU AI Act risk tiers. Flag systems that touch personal data or regulated decisions.
- Days 31-60, score. Run the six-category scorecard against every shortlisted vendor. Document red flags with evidence, not adjectives.
- Days 61-90, negotiate. Open contract terms for vendors who passed scoring. Escalate unresolved red flags to a written risk acceptance signed by a named executive, or drop the vendor.
- Ongoing, sustain. Tie reviews to renewal dates. Set triggers for model version changes, regulatory updates, and performance drift.
The honest trade-off. 90 days is fast for due diligence and slow for a project sponsor who wanted to go live last quarter. Compress it further and the failure modes show up later, after the contract is signed and the exit cost is high. We have walked into enough remediation projects to know which option costs more.
Patterns We See When Enterprises Get AI Vendor Selection Wrong
Three failure modes repeat across industries.
The first is treating evaluation as a procurement formality. The scorecard gets filled in to clear a gate, not to inform a decision. Twelve to eighteen months later, the same team is managing the consequences of a contract with no audit rights and an SLA that excludes the one failure mode that actually shows up in production. Procurement won the deadline. The business unit pays the bill.
The second is hard negotiation on price and silence on data portability. Six months in, the vendor changes its pricing or pivots its product. The exit cost is prohibitive because nobody asked for open export formats up front. The lock-in was always written into the contract. It just was not read closely.
The third is technical naivety about data preparation. Vendor proposals usually allocate 10 to 20% of the budget to data work. Real projects spend 50 to 65% there. When a buyer accepts the vendor's optimistic split, the gap shows up as overruns, scope cuts, or a model that quietly underperforms on the buyer's actual data.
"The enterprises that get this right share one trait. They treat the AI governance lead as a peer of the CISO, not a compliance afterthought. When governance sits at the negotiation table from day one, the contract reflects how the system will really fail." - Yuliya Marazenko, Head of AI Implementation, Silk Data
A complementary point from our advisory side, on whether to buy a vendor platform or build:
"Buy when the problem is generic and the data is not sensitive. Build, or run on-prem, when the data is the moat or the regulator is watching. Most enterprises decide this by habit, not by analysis. That is where money leaks." - Polina Volodina, AI Advisor, Silk Data
Where Silk Data Fits in This Picture
Two situations send enterprises our way during vendor evaluations. The first is when shortlisted vendors all require sending data to a shared cloud, and the regulator, the CISO, or the client contract says no. We have built on-prem LLM deployments for a marketing agency and for a German publishing client where the archive could never leave the perimeter. The second is when the off-the-shelf product covers 70% of the use case and the remaining 30% is the part that actually matters. Contract analysis, semantic search over a corporate archive, news clustering with opinion mining: these are the cases where a vendor demo looks good and the production deployment never quite lands.
If you are comparing build, buy, and hybrid options, our NLP and LLM services page lists the building blocks we deliver. The case studies hub shows where they have been used.
