“We need an AI Center of Excellence” has become a common refrain in enterprise AI discussions. The idea is appealing: a dedicated team that builds AI capability, sets standards, and drives adoption across the organisation.

The reality is usually disappointing. Most AI CoEs I’ve observed either become bottlenecks, get ignored, or dissolve within two years.

Here’s how to build one that actually delivers value.

Why Most AI CoEs Fail

They become gatekeepers, not enablers. The CoE positions itself as approving or rejecting AI initiatives. Business units route around them.

They’re disconnected from business. Technically excellent teams that don’t understand how the business actually works produce solutions nobody wants.

They build instead of enabling. Central teams that try to build all AI end up overwhelmed and blocking progress.

They lack authority to drive adoption. CoEs with responsibility but no authority can recommend but not require.

They’re under-resourced from day one. A three-person CoE can’t serve a 5,000-person organisation effectively.

The Successful Model

The CoEs that work share common characteristics:

Focus on Enablement, Not Execution

The CoE shouldn’t build AI applications – it should enable business units to build (or acquire) their own.

What this means in practice:

• Defining standards and patterns, not building solutions
• Providing training and coaching, not doing the work
• Reviewing and advising, not approving or blocking
• Building reusable components that others can use

The balance: The CoE might build shared infrastructure (data platforms, model serving) but not business applications.

Embed, Don’t Centralise

The most effective model involves CoE members embedded in business units for periods of time.

How it works:

• CoE members rotate into business units for 6-12 month assignments
• They work on real business problems with business teams
• They bring CoE standards and knowledge into the business context
• They bring business understanding back to the CoE

This prevents the ivory tower problem while spreading capability.

Clear, Valuable Services

Successful CoEs offer specific services that business units actually want:

Training programs. Structured learning paths for different roles and skill levels.

Advisory services. Expert consultation on AI strategy, architecture, and implementation.

Reusable components. Shared tools, templates, and infrastructure that accelerate projects.

Community facilitation. Connecting practitioners across the organisation, sharing learnings.

Vendor management. Consolidated vendor relationships and procurement.

Governance support. Helping teams meet governance requirements, not blocking them.

If business units wouldn’t voluntarily use CoE services, the services aren’t valuable enough.

Right-Sized Team

CoE sizing matters:

Too small: Can’t provide meaningful services. Becomes a talking shop.

Too large: Overhead exceeds value. Becomes a bureaucracy.

Right-sized: Typically 1 CoE person per 500-1000 employees, depending on AI maturity and ambition.

A 5,000-person organisation might have a 5-10 person CoE core team, supplemented by embedded members and part-time contributors.

Clear Metrics

CoEs should measure value delivered, not activity:

Good metrics:

• Business unit AI project success rate
• Time from AI concept to production
• Organisation-wide AI capability levels
• Business value delivered by AI initiatives

Bad metrics:

• Number of training sessions delivered
• Number of standards documents produced
• Number of projects reviewed

Activity isn’t value. Measure outcomes.

Getting Started

Phase 1: Foundation (Months 1-3)

Staffing: Hire the CoE leader and 2-3 initial team members. Focus on people who combine technical capability with business orientation.

Baseline assessment: Understand current AI capability, projects, and needs across the organisation. Don’t assume you know.

Service definition: Define the initial services the CoE will offer. Start focused – 3-4 services maximum.

Stakeholder engagement: Build relationships with business unit leaders. Understand what they need, not what you want to provide.

Phase 2: Proof of Value (Months 4-9)

Pilot services: Deliver initial services to willing business units. Prove value before scaling.

Quick wins: Identify 2-3 high-visibility projects where CoE support can make a difference. Demonstrate impact.

Capability building: Start training programs. Build internal AI capability broadly.

Iteration: Adjust services based on what’s working and what isn’t.

Phase 3: Scale (Months 10-18)

Expand services: Add services that have proven valuable. Retire those that haven’t.

Embed resources: Place CoE members in business units for extended assignments.

Community building: Establish practitioner communities that extend CoE reach.

Governance integration: Ensure CoE standards and governance requirements are aligned.

Phase 4: Maturity (Ongoing)

Self-service focus: Shift CoE toward providing platforms and tools that business units use independently.

Advanced capability: As basics are covered, focus on advanced capabilities and emerging technologies.

Continuous improvement: Regular assessment of CoE value and adjustment of approach.

Common Questions

Who should the CoE report to?

Options: CIO, CDO, CTO, or dedicated Chief AI Officer. The key is ensuring business connection (not just technical) and sufficient authority. Avoid burying CoE deep in an IT hierarchy.

How do we fund it?

Options: Central funding, chargeback to business units, or hybrid. Central funding is simpler initially but can disconnect CoE from business value. Chargeback creates accountability but adds overhead.

What skills do we need?

Mix of: AI/ML technical expertise, business analysis capability, change management skills, training and facilitation ability. Pure technical teams struggle with business alignment.

How do we handle resistance?

Some business units will resist CoE involvement. Don’t force it. Prove value with willing partners first. Success attracts others.

Final Thought

An AI Center of Excellence can be a powerful accelerator for enterprise AI adoption. It can also be an expensive bureaucracy that slows everything down.

The difference is in design and execution: focus on enablement over control, measure value not activity, stay connected to business needs, and right-size for your organisation.

Get these things right and a CoE delivers real value. Get them wrong and you’ve created overhead without benefit.

Be deliberate about which outcome you’re building toward.