Notes from building an AI design collaborator · part 7
We ran our design process on itself, and it broke in useful ways
· 3 min read
I'd built this whole AI design workflow, nine steps, gates between them, the works. And I'd tested it the way you test something you're proud of, which is to say gently, on friendly little examples that were never going to embarrass it.
Then I did the thing I'd been avoiding. I picked a real feature, a settings redesign we actually needed, and ran the whole workflow on it start to finish. No shortcuts, no stepping in to save it when it wobbled. Just me and the process, in public, for real.
It worked. It also broke in about a dozen places. The breaking is the part worth writing about.
What only shows up when it's real
Every failure was something a friendly test would never have surfaced.
The very first step quietly skipped a question it should always ask, because on this run there was no PRD, and the question had been hiding inside the PRD path. Nobody would have caught that on an example that happened to have a PRD. Later, when it pushed the work into Figma, it captured one screen per flow and dropped the other states, the empty ones, the error ones, the exact states a real feature has and a demo doesn't. The demos had all been one-screen-happy-path, so the gap had never had a chance to show.
None of these were dramatic. That's the point. They were small, specific, and completely invisible until a real feature dragged them into the light.
The scoreboard
By the end I had a list. Thirteen findings about the workflow itself, and about the same number of new rules about my own taste that the feedback loop caught along the way. Every one of them was a thing I would have sworn was already handled, right up until it wasn't.
I fixed them in batches over the next few days. The workflow is meaningfully better now, and none of the improvements came from me sitting around imagining how it might fail. They came from watching it fail at something I actually cared about getting right.
You cannot design-review your way to this. A review looks at the thing and asks if it seems right. Dogfooding uses the thing and finds out. The gap between "seems right" and "survived contact with a real task" is where all the interesting bugs live, and the only way across it is to trust your own tool with real work and then write down everything that made you wince.
It's uncomfortable, because you're basically running an experiment designed to make your favourite thing look bad. Do it anyway. The wince list is the most useful document you'll produce all quarter.
Part of a series on building an AI collaborator for our design team at Xflow. Each post stands on its own.
- ai
- design-process
The rest of Notes from building an AI design collaborator
- 1. I stopped describing my product to AI. I just gave it the HTML.
- 2. I gave the AI my design process, not my design decisions
- 3. I turned my design process into a dozen small skills, not one giant prompt
- 4. I built a free tool that gives your AI your real product
- 5. I taught the AI to push designs into Figma. Then it quietly stopped.
- 6. We rebuilt our Figma design system out of the code, not the other way round
- 8. How do you teach an AI your taste?
- 9. It worked great on the smart model. Then I ran it on a cheaper one.
- 10. The AI told me our brand colour with total confidence, and it was wrong
- 11. A screen list lies. I found a whole feature I didn't know we shipped.
- 12. Our documentation updates itself, because updating it is the AI's job
- 13. I sat down to turn my workflow into a swarm of agents. Most of it refused.
- 14. Every modal in our app shared one URL. Our analytics couldn't tell them apart.
- 15. My AI's rules only worked because one tool bothered to read them
- 16. Six designers, one GitHub account, and a script that saves us from ourselves
- 17. "Out of scope" usually means "we'll repaint this in six months"