Observations from dogfooding our own AI product

By now you may have heard that Assembled has been working on a new AI tool for customer support teams.

Currently in beta with a handful of Assembled users, this AI tool that helps agents quickly find answers and write replies has been on a high-speed bullet train of product development since April of this year.

And while Team New Products — the name given to the special projects team tasked with leveraging AI in a new customer support tool — has been working fast and furiously to build the product, Assembled’s own support team has been working just as hard to pressure-test it internally.

“I’ve always really liked the idea of having a tool to make it easier for support people to solve problems — as well as actually form the response,” said Assembled Head of Support Jeremy Eckman. “I think those are the big challenges that any support person has.”

Supporting product development

Assembled’s support team provides support to Assembled’s customers’ support teams (say that three times fast). This makes it a particularly potent testing ground for any tool that’s intended to be used by frontline support people.

“I think what most AI companies get wrong when they're talking about AI and support is the idea that AI is going to replace support,” Jeremy said. “But that’s just not realistic.” 

“Support is all about helping people with a really technical challenge or trying to better understand what they're actually trying to do,” he added.

Armed with this understanding of the purpose and challenges of support, Jeremy and his team began experimenting with an early-stage prototype of Assembled’s forthcoming AI product — known internally as Cal — just one week after the formation of Team New Products.

At the time, Cal was a simple Slackbot that would search support documentation to quickly find answers to questions asked over Slack.

Assembled’s support team was the first to point out that having access to the tool in the help desk itself would eliminate the need to context-switch between different platforms. And by the time August rolled around, Cal was accessible directly in Zendesk.

This would be the first of many examples where constructive feedback provided by internal users would influence the direction of the product. Current and future Cal users can also thank Assembled’s support team for what has come to be one of Cal’s most celebrated features — that it cites the sources it uses in its suggested replies.

“I imagine they would have eventually gotten to these things without us,” Jeremy said. “But it was really cool to see those updates come out as we were talking about them.”

Emotional support rotations

Assembled’s support team interacts with Cal daily, but they aren’t the only Assemblers who have had a chance to try Cal on for size. Cal has become incredibly useful for support rotations — an Assembled tradition that gives non-support people an opportunity to jump in the queue and answer support tickets. 

Stu Waters, who leads demand generation at Assembled, has participated in two support rotations this year — one without Cal and one with Cal. 

“Both experiences required a lot of conversation with proper support agents,” Stu said. “But with Cal, I felt more informed when asking the support team for help.”

Support rotations can be particularly anxiety-inducing for non-support people because of the technical nature of the questions that are being asked. In a lot of cases, folks on support rotation are entirely unfamiliar with where to even begin.

“Having Cal there to summarize helps you get to the bottom of the question itself,” Stu said. “It gets you closer to understanding what’s being asked so you can go investigate on your own or ask for assistance with more clarity.”

Turns out this is a benefit that goes both ways.

“Support rotations add more layers of complexity for the support team because we’re essentially supporting non-support people while they’re supporting our customers,” Jeremy said. 

“But with Cal, we’ve seen a big improvement,” he continued. “People can ask Cal for help and it will point them toward documentation, which gets them thinking about how they can learn more before coming to us.”

“Ramping seems like a really solid use case for Cal,” Stu said. “From my experience, I was able to do more with very little prep or training. So it really does feel like a force multiplier early on.”

Onboarding in the fast lane

Zach Winans joined Assembled as a technical support engineer in July of this year — around the time that Cal was making the transition from Slack to Zendesk. As such, he’s seen Cal’s rapid development firsthand while using it to get up to speed in his role.

“I've worked in one other support role, and that had a really long ramp,” Zach said. “I think it was six or seven months. But at Assembled, I was answering all tickets by the second or third month.”

A self-described overthinker, Zach said he really appreciates that Cal let’s him do his own research before having to ask his teammates for help. He credits the tool with allowing him to be more self-sufficient in learning the ins and outs of the product.

“We’re a pretty small support team, and we all have a bunch of tickets,” Zach said. “So I like that Cal gives me a starting point I can use to investigate an issue without having to interrupt my coworkers when they’re in their own workflows.”

“Before Cal, people on the support team learned by going into the product and seeing how things worked,” Jeremy said. “Both are great ways to learn, but I think the benefit of Cal is that it gives you more of a starting point. And as a newer support person, knowing where to start can be one of the hardest parts.”

Zach has used ChatGPT in the past, but he said it’s nothing like having access to a tool that caters to a specific product and associated support documentation.

“The fact that Cal shows you the sources that it pulls information from is the best part, in my opinion,” he said. “It basically helps you create a web of helpful information that you can latch onto and start troubleshooting.”

The not-chatbot that keeps on giving

Faster onboarding and more confidence for non-support people are two big areas where the team has seen improvement. But testing Cal internally has also revealed more incremental benefits too.

For example, one unexpected outcome of using Cal is that it has exposed gaps in Assembled’s support documentation.

“We’ve updated plenty of support articles because, through the process of using Cal to troubleshoot an issue, we’ve identified pertinent missing information,” Zach said.

It’s not as big or splashy as onboarding a new support person 2–3× faster, but it does contribute more efficient and reliable answers over time.

For his part, Jeremy is really excited to see Cal get into the hands of more support people.

“When a support person can rely on an AI-powered research assistant to check the help center, it makes them work so much faster and better,” Jeremy said. “This frees them up to really dig into the issue while creating a good experience for the customer.” 

“That’s much better than hoping the customer will get sent the right documentation via a chatbot,” he added.