Many organisations have established AI Centers of Excellence (CoE) over the past two years. Some have delivered genuine value. Many have become expensive overhead with little impact. What separates success from failure?

What a CoE Should Do

Before discussing structure, clarify purpose. A well-functioning AI CoE provides:

Capability development. Building AI skills across the organisation, not just within the CoE.

Governance and standards. Establishing and enforcing consistent AI practices.

Delivery support. Helping business units deliver AI initiatives, either directly or through guidance.

Knowledge sharing. Capturing and disseminating learnings from AI work across the organisation.

Strategic guidance. Advising leadership on AI trends, opportunities, and priorities.

Not every CoE needs to do all of these equally. Prioritise based on organisational needs.

Three CoE Models

The Centralized Delivery Model

The CoE builds and operates AI solutions for the entire organisation.

Advantages:

• Concentrated expertise delivers quality
• Consistent standards and practices
• Efficient resource utilisation
• Clear accountability

Disadvantages:

• Becomes bottleneck
• Disconnected from business context
• Can feel like ivory tower
• Limits organisation-wide capability building

Best for: Organisations early in AI maturity with limited distributed capability.

The Federated Enablement Model

Business units own AI delivery; the CoE provides guidance, tools, and governance.

Advantages:

• Scales better than centralized
• Keeps AI close to business context
• Builds broader capability
• More responsive to business needs

Disadvantages:

• Quality variation across units
• Requires mature business unit capability
• Governance harder to enforce
• Potential for duplication

Best for: Organisations with distributed technology capability and maturing AI experience.

The Hybrid Model

CoE delivers complex/strategic initiatives centrally while enabling simpler work in business units.

Advantages:

• Matches approach to complexity
• Balances control and empowerment
• Develops capability progressively
• Flexible to organisational needs

Disadvantages:

• Boundary confusion (what goes where)
• Coordination overhead
• Mixed incentives
• Requires mature governance

Best for: Mid-maturity organisations balancing quality and scale.

Most organisations evolve through these models – starting centralized, moving toward federated as capability builds.

Common Failure Patterns

The Ivory Tower

CoE becomes isolated from business reality. Builds impressive capabilities nobody uses. Viewed as academic rather than practical.

Root cause: Insufficient business involvement. Technology-first rather than problem-first thinking. Success measured by technical achievement, not business value.

How to avoid: Require business sponsorship for initiatives. Measure business outcomes. Rotate CoE staff through business units. Include business representatives in CoE governance.

The Bottleneck

Every AI initiative requires CoE involvement. Queue grows faster than delivery capacity. Business units frustrated by delays. Shadow AI emerges.

Root cause: Centralized model without adequate capacity. Failure to build distributed capability. Scope too broad.

How to avoid: Invest in enablement, not just delivery. Build tools and templates others can use. Establish clear criteria for what needs CoE involvement. Add capacity or narrow scope.

The Service Bureau

CoE becomes order-taker, building whatever business units request without strategic filtering. Resources scattered across low-value initiatives.

Root cause: Insufficient strategic authority. Funding model that rewards utilisation over impact. No portfolio management.

How to avoid: Establish intake and prioritisation process. Fund based on strategic value, not demand. Say no to low-value requests.

The Governance Police

CoE focuses on rules and compliance rather than enabling good work. Viewed as obstacle rather than partner. People avoid engaging.

Root cause: Governance without enablement. Risk-first culture. Success measured by compliance, not value.

How to avoid: Balance governance with support. Celebrate successes, not just catch failures. Make governance efficient and proportionate. Frame as enablement, not control.

What Success Looks Like

Successful AI CoEs share characteristics:

Business outcomes focus. Measured primarily on business value delivered, not activity metrics.

Respected expertise. Business units want CoE involvement because it improves outcomes.

Appropriate governance. Standards that enable rather than obstruct. Proportionate to risk.

Capability building. Measurable improvement in organisation-wide AI capability over time.

Strategic influence. Voice in AI investment decisions. Trusted advisory role to leadership.

Sustainable model. Not dependent on heroic effort. Processes that work at scale.

Building a CoE

If you’re establishing or restructuring an AI CoE:

Step 1: Clarify Purpose

What problem is the CoE solving? What would success look like? Get explicit agreement from sponsors.

Step 2: Define Scope

What’s in scope and what’s not? Which AI activities belong to the CoE and which to business units? Draw clear boundaries.

Step 3: Choose Model

Centralized, federated, or hybrid? Match to your organisational maturity and strategic needs.

Step 4: Staff Appropriately

Mix of technical depth and business acumen. Avoid purely technical teams that can’t translate to business value.

Step 5: Establish Governance

Intake processes, prioritisation criteria, standards, review processes. Keep proportionate to organisational scale.

Step 6: Build Connections

Relationships with business units are as important as technical capability. Invest in partnerships, not just delivery.

Step 7: Measure and Adapt

Define metrics that matter. Review regularly. Be willing to evolve the model as needs change.

The Talent Question

CoE staffing is critical and challenging:

Technical skills. Need genuine AI expertise, not just general technology capability.

Business translation. Need people who can connect AI to business problems.

Consulting skills. Need people who can influence without authority.

Teaching ability. Need people who can build capability in others.

This combination is rare. Consider mixing specialist technical staff with broader consulting/business staff rather than seeking unicorns who have everything.

Retention matters. Competitive market makes keeping good AI talent challenging. Interesting work, development opportunities, and competitive compensation all matter.

Final Thought

An AI Center of Excellence is a means, not an end. The goal isn’t a great CoE – it’s an organisation that uses AI effectively.

Build CoE structures that serve that goal. Avoid the trap of optimising the CoE at the expense of broader organisational capability.

The best CoE is one that progressively reduces the organisation’s dependency on it while maintaining appropriate governance. Build capability, not empire.