Notes from building an AI design collaborator · part 1
I stopped describing my product to AI. I just gave it the HTML.
· 6 min read
A couple of years ago I wrote about coding my own portfolio site, back when I was a designer who could barely tell a flex div from a margin. I'm still mostly that designer. I've just picked up a few opinions since.
Here's one. The reason your AI keeps handing you generic design help isn't the model. It's that the model has never actually seen your product.
The one everyone's reading
There's a great piece going around by Thariq, an engineer at Anthropic, called "the unreasonable effectiveness of HTML." If you build anything with AI, go read it. His argument is that HTML beats Markdown as the thing an AI writes for you. Specs, prototypes, a whole living design system in one file, all of it in HTML because HTML is dense, it's visual, and it keeps you in the loop instead of handing everything off to the machine.
I agree with all of it. I want to talk about the other direction, the one I don't see anyone talking about. Not the HTML your AI writes for you. The HTML you give it.
The average of the internet
Ask an AI to redesign one of your screens and you get something clean, sensible, textbook, and not at all yours. It isn't wrong. It's just average.
I hit this every single day at Xflow. I'd ask for a small change to a page in our dashboard and get back a gorgeous redesign of a page we'd never built. It moved a toggle that didn't exist. It tidied up a section that was never there. Very confident. Completely made up.
Of course it was. The model has seen thousands of settings pages, so it hands you the average of all of them. And in a cross-border payments dashboard full of compliance states and payout flows, an average isn't a starting point. It's a well-dressed guess.
That's the word I keep landing on. Honest. A design is honest to your product when it comes from what the product actually is, not from the AI's idea of what a product like yours probably looks like.
I tried describing it. I tried screenshots.
My first fix was to just describe the thing better. "Sidebar with these five items, header with the account switcher," and so on. This turns you into a slow, lossy, human screenshotter. It doesn't scale past two screens, it goes stale the second the product changes, and the AI still builds its own version of your sentence anyway.
Screenshots were the next idea. Better at looks, useless for work, because the AI can't edit a picture. It looks at it, forms an impression, and redraws the whole thing from memory. A screenshot is something to point at, not something to change.
Then the obvious finally landed, and it's really just Thariq's point flipped around. If HTML is the best thing an AI can write, it's also the best thing it can read. My product was already HTML sitting in a browser. I'd been reading it out to the AI like it was over the phone, when I could have just sent the file.
So I gave it the HTML
At Xflow we built a small pipeline that grabs our real pages as self-contained HTML. The real DOM, the styles baked in, the page able to open on its own. I'm a designer and not an engineer, so version one was held together with code that would've made our developers wince. It worked, and we cleaned it up later.
Now the AI opens the actual page and edits the actual page. I mention a toggle and it touches the real toggle, in the section it really lives in, with our real styles. The same one-line request that used to come back as fiction now comes back as an edit.
None of this is an Xflow thing, which is the point. Whatever you work on is already HTML in a browser, and that HTML is a more honest brief than anything else you can hand an AI. More honest than a Figma file, which is a drawing of the product and not the product. More honest than your own description, which is really just your memory of it on a good day.
But won't that cost a fortune in tokens?
Fair question, and yes, if you're careless. Raw product HTML is massive. Framework wrappers, the same inline styles a thousand times over, tracking scripts, images stuffed straight into the markup. Send all of that and you've burned a small fortune before the AI has done a thing.
So you don't send the raw page. You clean it first. Rip out the tracking and the dead scripts, collapse the repeated styling, swap the giant encoded images for placeholders, and the same screen shrinks to a fraction of its size. Cheaper to send, and honestly easier for the model to read too, because what's left is mostly structure and words instead of junk.
Which is the whole loop, really. HTML goes in, HTML comes back. You hand the model the real page, it hands you an edit, and the same format carries both trips. Thariq's half and mine are just the two ends of it.
Try this in ten minutes
You don't need any of our tooling to feel the difference. Open a screen in your product, hit "Save Page, Complete" in your browser, and there's your HTML. Give that file to your AI and ask for your next real change. Then ask for the exact same change with nothing attached, and look at the two.
One of them edits your product. The other invents a cousin of it.
These days I run one test on anything an AI designs for me. Could this ship tomorrow without inventing a component we don't have? If yes, it's honest. If it needs three controls that don't exist, the model drew me the internet, not our product.
One honest catch, though. A brand new screen has no HTML to hand over, so for genuinely new work you're back to describing, or better, giving the AI your design system's real components to build from. You're still grounding it in something real. Just the system this time instead of a page.
What I built
Doing that capture by hand got old fast. Save the page, clean it up, do it again next week when the product moved. So I built a tool that does it for any product. You point it at a URL and it turns the real thing into a context library your AI can read: the pages, the structure, even the product's own colours and type, all pulled in as facts, cleaned down so it's cheap to feed in.
It's called the Design Context Kit, it's free and open source, and it's up now. If this honest-to-your-product idea clicks for you, that's the fastest way to try it on your own product. The rest of the series gets into how the pieces actually work.
This is the first post in a series about building an AI collaborator for our design team at Xflow, one that works from the real product, follows a real design process, and actually remembers why we made the calls we made. Each post stands on its own. I'll link the rest as they go up.
Next: why I gave the AI my design process, and none of my design decisions.
- ai
- design-process
- html
The rest of Notes from building an AI design collaborator
- 2. I gave the AI my design process, not my design decisions
- 3. I turned my design process into a dozen small skills, not one giant prompt
- 4. I built a free tool that gives your AI your real product
- 5. I taught the AI to push designs into Figma. Then it quietly stopped.
- 6. We rebuilt our Figma design system out of the code, not the other way round
- 7. We ran our design process on itself, and it broke in useful ways
- 8. How do you teach an AI your taste?
- 9. It worked great on the smart model. Then I ran it on a cheaper one.
- 10. The AI told me our brand colour with total confidence, and it was wrong
- 11. A screen list lies. I found a whole feature I didn't know we shipped.
- 12. Our documentation updates itself, because updating it is the AI's job
- 13. I sat down to turn my workflow into a swarm of agents. Most of it refused.
- 14. Every modal in our app shared one URL. Our analytics couldn't tell them apart.
- 15. My AI's rules only worked because one tool bothered to read them
- 16. Six designers, one GitHub account, and a script that saves us from ourselves
- 17. "Out of scope" usually means "we'll repaint this in six months"