How to think about the “proof” of your AI proof of concept (POC)

AI and Automation

Brian Yeh

JUMP TO SECTION

You’re taking the right steps to stay ahead in the world of AI. Your company has an AI strategy, you’ve done the homework, and you’re ready to run a proof of concept (POC) for an AI solution. But if your POC isn’t set up properly, the results could be disappointing — think “garbage in, garbage out.” To help you make your AI POC a success, we’ve compiled lessons from real-world examples and distilled them into actionable advice.

Where will AI be working during the POC?

AI can transform many areas of support, from automating responses to empowering agents and uncovering actionable insights. Because you can’t evaluate everything at once, it’s essential to focus your POC on a specific use case that aligns with your CX goals.

The deflection-centric POC

Best for: High-volume, low-complexity inquiries, like FAQs or order tracking

‍Watch outs: Balance internal testing and low-stakes customer-facing deployment

‍Metrics: First response time, automation rates

If your #1 priority is reducing repetitive work and increasing efficiency, a deflection-focused POC might be the best choice. At their best, AI agents can automate certain customer interactions by delivering direct answers via chatbots or voice tools, leveraging resources like your knowledge base, help center, and previous tickets. Since the outcome of a deflection-centric POC is an application that will directly touch customers, think about how you plan to test this with the right guardrails.

The copilot-centric POC

Best for: Complex tickets that require human oversight or for upskilling agents with varying levels of expertise

‍Watch outs: Pay attention to change management and agent adoption of new workflows

‍Metrics: Average handle time, resolution rates, or ease of collaboration between agents and AI

If your company’s support strategy emphasizes human expertise, an agent-centric POC may be the right focus. By focusing on agent empowerment rather than full automation, an agent-centric POC allows you to validate whether AI can elevate the human-driven support experience your company values most. Copilots act as behind-the-scenes partners, drafting responses, providing relevant knowledge, or automating routine tasks so agents can focus on more complex or emotionally sensitive cases. Pilot the tool with a select cohort of agents (skill-level, seniority, focus area) to understand how it supports varying skill levels and work styles.

The data-centric POC

Best for: Teams looking to extract data to inform future strategy, investment areas, etc

‍Watch outs: Ensure leadership buy-in to evaluate and act on the insights generated

‍Metrics: TBD depending on project objectives

If your goal is to surface actionable insights from support data, consider focusing the POC on AI’s ability to analyze interactions and uncover trends, patterns, and themes that might otherwise go unnoticed. It’s a great way to get value from the support data that already exists in your company. An insights-centric POC might only involve a small group of people, but it’s likely to engage higher levels of leadership to evaluate the impact of the insights that are surfaced.

Measuring success during an AI POC

The best way to measure the success of an AI POC is to start by identifying the metrics that matter most to your team. While the full benefits of AI will only become clear after a full implementation, a well-structured POC should still deliver noticeable improvements. Keep in mind that the scope of these changes will depend on the duration of your pilot. A 90-day POC provides more time to evaluate and impact key metrics compared to a shorter 30-day POC.

A note on accuracy

It’s very tempting to look at “accuracy” and “quality” as the only benchmarks for an AI POC. After all, what matters most is getting the right answer to every customer, every time. Be wary of focusing exclusively on quality/accuracy when evaluating an AI solution in a POC. Not only is high quality table stakes, but it’s also changing quickly, as underlying LLMs improve almost daily. This means that any random sample of responses to the same ticket can have slight differences that might make them appear a bit “better” or “worse.” Over-scrutinizing here can detract from the bigger picture: operational impact and scalability.

Success metrics for a 30-day AI POC

30 days should give your team enough time for a technical evaluation of the AI tool. You should get a sense of how easy it is to configure, whether or not you’ll have the control you need, and an idea of how to improve quality over time.

In a short POC, focus on evaluating the technical capabilities of the AI tool and its ease of adoption with a small group of participants. Testing too broadly over a short timeframe may actually have a negative impact on support as many people will be trying to learn something new, all at the same time. Remember to align your success metrics to the kind of AI POC (copilot, deflection, etc) you’re running.

Here are some ideas for success metrics during a 30-day AI POC:

Ease of use: Do agents find the copilot useful? Are all the features intuitive? Are managers and admins able to customize and adjust the copilot? A simple survey where you ask agents to rate the tool will give you the information you need to make sense of feedback. If you’re testing out an admin feature like automations, you’ll want to make sure the team members find it easy to set up these workflows.
Improved compliance with internal policy: Many support teams, like the team at Tithely, have a policy where they require an agent to send a helpful link with every support ticket. Because agent copilots make it easier to find internal documentation, having a copilot should increase agents’ ability to include the right links. Measure whether or not compliance with this type of policy improves among your POC participants.
Control mechanisms: When you get hands-on with an AI tool, you’ll be able to see whether or not that tool has the kinds of features that will give you the level of control you need. If you’re evaluating a copilot or automated tool, will you be able to customize your brand voice? Does the reporting give you the visibility you need? Are you able to see everything from the zoomed-out view down to the impact of AI on individual agents and tickets?
Partnership: The world of AI for support is ever-evolving, so you’ll want to be sure you’re aligned with your AI provider’s approach during the POC. Use the time to get a sense of how they support their product and their areas of future investment. AI technology is developing fast, and you want to make sure your partner can keep up.

Success metrics for a 90-day AI POC

A 90-day AI POC requires a greater time investment but offers your evaluation team a deeper understanding of how AI can integrate into your operations. Beyond conducting a basic technical assessment, this extended timeline allows you to configure and test customer-facing automations thoroughly. For customer-facing AI use cases, a 90-day period provides the necessary time to evaluate impact, make adjustments, and simulate a real-world AI implementation process.

Participant selection is just as crucial for a successful 90-day POC. Focus on time-boxing training and change management efforts to streamline the process while gathering meaningful feedback. For automation-focused pilots, select participants who have a foundational understanding of AI to help accelerate setup and testing.

Here are some ideas for success metrics during a 90-day AI POC:

Maintain quality: Is CSAT still high on cases where AI steps in? With AI, you should expect to maintain your CSAT at a minimum. If CSAT starts slipping, diagnose whether there’s a correlation with the use of AI.
Number of successful automations: If you’ve successfully identified clear, consistent processes, you should be able to easily configure, quality-check, and deploy automations. Your AI solution should also be able to help you identify potential use cases for new automations. During a 90-day POC, it’s reasonable to set up and launch at least four fully-automated processes.
Integrations - One of the game-changing aspects of today’s AI products is their ability to integrate with and take action in integrated systems. You may want to consider a way to incorporate integration testing into your AI POC to pressure-test whether you can get value from your most critical systems.

Was my AI POC successful?

At the end of a POC, you should have the information you need to clearly answer the question of whether or not it was successful. With companies running multiple AI POCs at once, it’s even more important to define success metrics early and make sure they’re aligned to company priorities. Taking the time to define success before you start and align with your technology providers will make sure you’re taking the mystery out of AI and setting up your support function for success!