I wrote about AI vendor evaluation back in 2024. A lot has changed since then. The market has matured, platform capabilities have converged, and new considerations have emerged. Here’s an updated framework.

The New Evaluation Context

Several shifts require updating how we evaluate AI vendors:

Capability convergence. The major platforms (Azure AI, Vertex AI, Bedrock) now offer similar core capabilities. Evaluation must focus on differentiating factors, not basic features.

Governance requirements. Regulatory pressure means vendor compliance capabilities now carry more weight.

Integration complexity. As AI becomes embedded in more processes, integration quality matters more than standalone capability.

Total cost realities. Hidden costs have become clearer. Evaluation must account for full lifecycle costs.

The Updated Framework

Category 1: Core Capability (20% weight)

Model access and quality. Which foundation models are available? What are their relative strengths? How current are the models?

Questions to ask:

• Which models can I access and under what terms?
• How does model performance compare on my specific use cases?
• How frequently are models updated, and what’s the update process?

Specialised capabilities. Beyond general-purpose AI, what domain-specific features exist?

Questions to ask:

• Are there pre-built solutions for my industry or function?
• Can I fine-tune models on my data? How complex is that process?
• What retrieval augmentation (RAG) capabilities are available?

Category 2: Enterprise Readiness (25% weight)

This category has increased in importance since 2024.

Security controls. Data protection, access management, encryption, and security certifications.

Questions to ask:

• Where does my data go during processing?
• What security certifications does the platform hold?
• Can I bring my own encryption keys?
• What data retention and deletion policies apply?

Governance features. Audit logging, content filtering, usage monitoring, and compliance tooling.

Questions to ask:

• What audit trails are available for AI interactions?
• How can I filter or control AI outputs?
• What monitoring and alerting capabilities exist?
• How does the platform support responsible AI requirements?

Data residency. Particularly important for Australian enterprises.

Questions to ask:

• Can data be processed entirely within Australian borders?
• What happens to data during model inference?
• Are Australian-specific compliance requirements understood and supported?

Category 3: Integration Quality (20% weight)

Native integrations. Pre-built connections to common enterprise systems.

Questions to ask:

• Which systems have native integrations?
• How deep are those integrations? (API access vs. embedded functionality)
• What’s the maintenance burden when integrated systems change?

API quality. For custom integrations, the API experience matters.

Questions to ask:

• Is the API well-documented and stable?
• What SDKs and development tools are available?
• How responsive is API performance?
• What happens when services are unavailable?

Existing ecosystem. Connection to your current technology stack.

Questions to ask:

• How does this integrate with my cloud platform?
• What about my productivity suite, CRM, ERP?
• Can I access AI capabilities from existing developer tools?

Category 4: Total Cost (20% weight)

Visible costs. License fees, usage-based pricing, infrastructure costs.

Questions to ask:

• What’s the pricing model? (Per user, per token, per request?)
• How do costs scale with usage?
• Are there minimum commitments or volume discounts?

Hidden costs. Implementation, integration, training, and ongoing support.

Questions to ask:

• What implementation support is included vs. extra?
• What’s the typical implementation timeline and effort?
• What ongoing support is provided?
• What skills are needed to operate the platform?

Optimisation opportunities. Ways to reduce costs as you scale.

Questions to ask:

• Can I choose lower-cost models for appropriate use cases?
• What caching and efficiency features reduce token usage?
• Are there committed use discounts?

Category 5: Vendor Viability (15% weight)

Financial stability. Will the vendor be around and investing in the product?

Questions to ask:

• What’s the vendor’s financial position?
• What’s the investment trajectory in AI capabilities?
• How many enterprise customers are using this at scale?

Strategic direction. Where is the vendor heading?

Questions to ask:

• What’s on the product roadmap?
• How does AI fit into the vendor’s broader strategy?
• What’s the relationship with underlying model providers?

Ecosystem health. Partners, community, and third-party support.

Questions to ask:

• What partner ecosystem exists for implementation support?
• How active is the developer community?
• What third-party tools and extensions are available?

Applying the Framework

Step 1: Customise Weights

The percentages above are starting points. Adjust based on your priorities:

• High regulatory requirements? Increase Enterprise Readiness weight.
• Complex existing environment? Increase Integration Quality weight.
• Tight budget? Increase Total Cost weight.
• Long-term strategic investment? Increase Vendor Viability weight.

Step 2: Define Evaluation Criteria

For each category, specify what “good” looks like for your organisation. Generic criteria lead to generic evaluations.

Step 3: Gather Evidence

Don’t rely on vendor claims. Test platforms with your data and use cases. Talk to reference customers. Review independent analyses.

Step 4: Score and Compare

Create a structured comparison. Avoid letting enthusiasm for one vendor bias the evaluation.

Step 5: Document Rationale

Record why you made the decision you made. You’ll need this when the decision is questioned later.

Common Evaluation Mistakes

Focusing only on model capability. Models are converging. Over-weighting capability ignores other critical factors.

Ignoring hidden costs. The sticker price is rarely the actual price. Account for full lifecycle costs.

Underweighting integration. AI value comes from integration with business processes. Integration difficulty undermines ROI.

Not testing with real data. Demo performance doesn’t indicate production performance. Test before committing.

Deciding based on one factor. No vendor wins on everything. Decisions require trade-off analysis.

Final Recommendation

The framework above is comprehensive, which makes it time-consuming. For most enterprises:

1. Eliminate non-viable options quickly based on must-have requirements
2. Deep evaluate 2-3 finalists using the full framework
3. Pilot before committing to validate evaluation conclusions
4. Build flexibility into contracts for a market that’s still evolving

The AI vendor landscape will continue changing. Choose based on current needs while maintaining ability to adapt. Flexibility may be the most valuable vendor characteristic of all.