Entry 01

APPSKETIERS

Reducing dispute escalations using an AI ticket manager

UX Design

UX Research

AI Tool

Desktop

TL;DR

The abridged version…

The Problem

The slow part of building client apps at Appsketiers wasn’t the actual development part. It was deciding whether each change request was in scope. PMs asked AI tools they didn’t trust, re-checked the docs by hand, and every dispute stalled the project.

My Role

Team: 2 UX Designers, 2 Developers

I co-led the research and owned the design of the ticket creation, triage flows, and the scope analysis redesign after testing.

Two of my key decisions: separate “Is the AI right?” from “What happens next?”, and show the exact evidence passage behind each verdict instead of a section number, so reviewers could check the AI without leaving the screen.

The Outcome

Since launch, every PM and QA engineer has worked from one shared tool instead of their own AI tool. 15 projects have been onboarded, and 6 have launched to clients.

TIME TO SCOPED TICKET

2h 35m → 18m

Escalated to Leadership

12 → 3
short on time? skip the deep dive and…
Jump to the solution
Jump to the solution

PROBLEM

RESEARCH

VERSION 1

TESTING

REDESIGN

SOLUTION

PROBLEM

RESEARCH

VERSION 1

TESTING

REDESIGN

SOLUTION

PROBLEM

RESEARCH

VERSION 1

TESTING

REDESIGN

SOLUTION

01 — PROBLEM

Every change request needs a scope decision

PROJECT MANAGER

A client just asked us to add a feature that wasn't in scope. I had to spend the whole afternoon proving why.

One request took the whole afternoon? That can't be how it usually goes… right?

PROJECT MANAGER

Pretty much. The build is actually the easy part. It's what happens when the change requests roll in that takes forever.

Appsketiers builds custom B2C apps from the ground up. From the first idea to the final shipped product, the team manages design, development, and delivery across months of active collaboration with each client.

Every piece of client feedback between Development and Testing needs a decision: is it in the Business Requirements Specification (BRS) we signed, or is it new work?

While the team debates, the work stops. The request sits blocked, and if the client pushes back hard enough, it escalates to leadership.

A quick answer isn’t enough. It has to be one the team can stand behind.

02 — RESEARCH

PMs were already using AI. They weren't sure they could trust it.

We interviewed our 3 PMs and 2 QA engineers. Going in, I assumed the familiar story in cases like these:

Maybe the team needs AI to move faster…

That's not what we found…

Every PM and QA engineer described the same loop: ask an AI tool, then re-check its answer against lengthy BRS documents and email threads anyway.

The AI wasn’t saving any time. Reviewers couldn’t tell when it was right, so checking its answer took as long as finding it themselves.

QA ENGINEER

I definitely don't want to give the client wrong information. That's why I need to check everything myself.

So how might we…

make the AI’s scope verdict easy to verify so that reviewers can defend it to a client?

Success Meant:

Less time verifying AI output by hand

Fewer disputes reaching leadership

What we decided not to build:

Automated decision-making system: Even when AI is confident in its decision, it never approves or declines a request on its own. Every scope call is signed off by a human reviewer.

More on that in a bit…

03 — VERSION 1

We put the reviewer in charge

…or so we thought

Claude and Figma Make turned our whiteboard sketches into clickable (but less than pixel-perfect) prototypes, which we fixed by designing using the shadcn/ui design system.

But it helped us get a proof of concept to the testing table much faster.

I recommended using the shadcn/ui design system because it’s built on the same Radix UI foundation that Figma Make uses. What we tested matched what developers would ship, with less rework.

Version 1 showed the AI’s verdict, an alignment score, and three buttons after comparing the change request with the approved BRS contract:

Nothing moved forward until a reviewer picked an option, so a human was always making the final option.

But did they understand what their options were?

An evidence section that doesn't really show the evidence and three buttons of equal weight. What could go wrong?

04 — TESTING

Reviewers misinterpreted the buttons and doubted the AI

Our 3 PMs and 2 QA engineers worked through the same 3 change requests. Two problems showed up.

FINDING #1

FINDING #2

The buttons meant different things to different people.

On the dark mode request, the right answer was "Override" since the sales team had promised both versions. 3 of 5 picked the wrong button. One person who got it right seemed to get there for the wrong reason…

QA ENGINEER

I thought Discuss meant discussing it with the client. So I chose Override so that the AI actually creates a ticket for the devs instead of calling it out of scope.

FINDING #1

FINDING #2

The buttons meant different things to different people.

On the dark mode request, the right answer was "Override" since the sales team had promised both versions. 3 of 5 picked the wrong button. One person who got it right seemed to get there for the wrong reason…

QA ENGINEER

I thought Discuss meant discussing it with the client. So I chose Override so that the AI actually creates a ticket for the devs instead of calling it out of scope.

FINDING #1

FINDING #2

The buttons meant different things to different people.

On the dark mode request, the right answer was "Override" since the sales team had promised both versions. 3 of 5 picked the wrong button. One person who got it right seemed to get there for the wrong reason…

QA ENGINEER

I thought Discuss meant discussing it with the client. So I chose Override so that the AI actually creates a ticket for the devs instead of calling it out of scope.

A human was making the final decision, but without the confidence that they were correct: the exact problem we started with.

So back to the drawing board we go…

05 — REDESIGN

Letting reviewers check the AI’s work

Reviewers now judge the AI before deciding, see how sure it is and what it relied on, and can overrule or reopen any decision.

Before
After
Before
After

SOLUTION #1

SOLUTION #2

Ask two distinct questions, not one

Confirm and Override asked: “Is the AI right?”

Discuss asked: “What happens next?”

We discussed and discarded…

IDEA #1 (Discarded)

Let's relabel the buttons to be clearer.

Maybe, but this doesn't solve the problem of having three buttons serve two different purposes.

IDEA #2 (Discarded)

How about we add "In Scope" and "Out of Scope" buttons to let the reviewer decide the verdict?

That would solve part of the problem but it still doesn't define what happens next.

Then, I suggested…

IDEA #3 (KEPT)

Let's split it into two distinct steps. First, judge and validate the AI output, then pick an action as an outcome of the verdict.

BEFORE

AFTER

Confidence % shows how sure the AI is of its verdict.

Why could this work? Chunking: Splitting one overloaded choice into two simple ones lowers cognitive load. Each step asks a single, clear question.

SOLUTION #1

SOLUTION #2

Ask two distinct questions, not one

Confirm and Override asked: “Is the AI right?”

Discuss asked: “What happens next?”

We discussed and discarded…

IDEA #1 (Discarded)

Let's relabel the buttons to be clearer.

Maybe, but this doesn't solve the problem of having three buttons serve two different purposes.

IDEA #2 (Discarded)

How about we add "In Scope" and "Out of Scope" buttons to let the reviewer decide the verdict?

That would solve part of the problem but it still doesn't define what happens next.

Then, I suggested…

IDEA #3 (KEPT)

Let's split it into two distinct steps. First, judge and validate the AI output, then pick an action as an outcome of the verdict.

BEFORE

AFTER

Confidence % shows how sure the AI is of its verdict.

Why could this work? Chunking: Splitting one overloaded choice into two simple ones lowers cognitive load. Each step asks a single, clear question.

SOLUTION #1

SOLUTION #2

Ask two distinct questions, not one

Confirm and Override asked: “Is the AI right?”

Discuss asked: “What happens next?”

We discussed and discarded…

IDEA #1 (Discarded)

Let's relabel the buttons to be clearer.

Maybe, but this doesn't solve the problem of having three buttons serve two different purposes.

IDEA #2 (Discarded)

How about we add "In Scope" and "Out of Scope" buttons to let the reviewer decide the verdict?

That would solve part of the problem but it still doesn't define what happens next.

Then, I suggested…

IDEA #3 (KEPT)

Let's split it into two distinct steps. First, judge and validate the AI output, then pick an action as an outcome of the verdict.

BEFORE

AFTER

Confidence % shows how sure the AI is of its verdict.

Why could this work? Chunking: Splitting one overloaded choice into two simple ones lowers cognitive load. Each step asks a single, clear question.

RETEST

Every tester chose the right next step on all requests

After the redesign, the same 5 testers worked through 10 real requests in a sandbox.

What's the catch?: Since this is an internal tool, we had to test with the same user group. Some of the improvement may be attributed to familiarity.

06 — SOLUTION

AI-drafted tickets, human-made decisions

01 — From client call to clean tickets

A call transcript becomes scope-checked draft tickets, and the reviewer sorts them before anything is created.

02 — The AI makes the case, the PM makes the call

The AI explains its scope verdict with evidence from the BRS. The reviewer decides and can change their mind with every step on record.

IMPACT

Faster scope calls and fewer escalations

We tracked 30 change requests before the tool was created and the first 30 requests after launch, reviewed by the same 5 PMs and QA engineers.

TIME TO SCOPED TICKET

2h 35m → 18m

on average across 30 requests

ESCALATED To LEADERSHIP

12 → 3

client disputes out of 30 requests

REFLECTION

Some thoughts and takeaways

What's a journal entry without some introspective musings?

What did I learn?

Showing the work mattered more than being right. Once reviewers could see why the AI decided something, they could trust it, or catch it when it was wrong.

What could I have done differently?

Follow the request upstream. The scope decision happens with PMs, but many requests start with what sales promised or what clients assumed. I’d trace that path earlier, so the tool accounts for it from day one.

What happens next?

As adoption grows, I’d watch for over-trust: if reviewers stop saying "No" altogether, that’s a warning to investigate, not a win.

thanks for reading!
Go back to the journal
Go back to the journal

Create a free website with Framer, the website builder loved by startups, designers and agencies.