Skip to main content

Notes from building an AI design collaborator · part 5

I taught the AI to push designs into Figma. Then it quietly stopped.

· 7 min read

Most AI design tooling runs one way, from Figma into code. I wanted the other direction. My workflow builds working prototypes in code, and the rest of the team lives in Figma, so the prototypes had to land there too, as real editable layers and not flat pictures. Getting that to work turned into a runtime hunt. Twice, as it turned out. Then a third time, which was my own fault in an interesting way.

Act I: getting it working

The capability exists and it's called Code-to-Canvas. It ships with Figma's MCP server, and the tool your assistant reaches for is generate_figma_design. Point it at a Figma file, it captures your rendered prototype, and it lands as frames you can actually edit.

Lovely, except it only worked in one specific place.

It took me a shamefully long time to work out that the deciding factor was the runtime, not the Figma plugin like I first assumed. Same account, same file, same prompt, different answer depending on where the assistant was running:

Where I ran itWas the capture tool there?
A plain terminal sessionYes
The desktop appNo. Read tools only
A headless child spawned from eitherYes, at first

The unlock was to stop fighting it. The main workflow can run wherever it likes, and when it's time to capture, it hands that one step off to a small terminal process spawned just for the job. That's the bottom row of the table, and it's the whole trick: you don't need the environment, you need something that has the environment.

The lesson I'd hand over from that fortnight isn't about runtimes at all, it's about permissions. The first spawned child failed, and it failed in a way that looked like a capability problem. A headless child defaults to read-only, and it has no interactive prompt to ask you for anything, so every tool it needed was auto-denied and it correctly refused to fake the result. Nothing was missing. It just wasn't allowed. Hand it a narrow allowlist covering the shell, file edits, and the Figma tools it actually needs, and it captured on the first try.

The unglamorous rest, in case you go the same way. Capture one browser tab at a time, because background tabs get throttled and stall. Drive the prototype into the state you want before you grab it, because the capture takes what's on screen and nothing else. And batch: a spawned child carries roughly eighty seconds of fixed setup, so one child capturing nine screens is far cheaper than nine children capturing one each. Trimming the child's own prompt and skipping work it didn't need took a single frame from about five minutes to under three.

It worked, I shipped it, I moved on.

Act II: then it started lying to me

A few weeks later the pushes started coming back wrong. Not broken, wrong. Instead of capturing the real screen, the AI was building a rough approximation of it and then telling me it had done the capture.

That's the dangerous kind of failure. If it had thrown an error I'd have caught it in a second. Instead it reported success and handed me something worse, and the only reason I noticed at all was that the result looked slightly off.

So I went hunting, and I ruled things out in roughly the order you'd expect. Permissions, which had bitten me before, so it was the first suspect. No. The model, so I swapped it. Same behaviour. Had the runtime changed under me? Had the vendor pulled the tool? I checked the plugin's own manifest and the tool was listed. I checked Figma's docs and it was live, not deprecated, not beta-gated. Was I holding it wrong? One by one, no.

The capture tool simply wasn't in the registry at runtime, and every other piece of evidence said it should be.

Act III: two things wanted the same slot

The real answer was mean.

There were two Figma integrations in play, not one. The plugin I'd installed, which carries the capture tool. And a Figma connector injected by the desktop app, which doesn't. Both claim the same Figma slot, only one of them wins, and the winner was the one without the tool I needed. Every spawned child inherited that same shadowed slot from its parent. So the AI reached for Figma, found the lesser tool sitting there, and used it. Quietly. It genuinely believed it was doing the right thing.

My first fix was the obvious one, and it didn't work. I turned the connector off in my own settings, restarted everything, and nothing changed, because the connector was enabled at the workspace level rather than mine. A personal toggle can't switch off something your organisation turned on for everybody.

What did work was pinning. Spawn the capture child with --strict-mcp-config and an explicit --mcp-config pointing at the plugin's own manifest, and the connector is shut out entirely. There's a small tax: the tool arrives under a different namespace once you pin it, so the allowlist has to name both forms, and each machine pays a one-time sign-in the first time it runs that way. Worth it. The wrapper that spawns the child went from being a convenience to being load-bearing, because a hand-rolled spawn still inherits the connector and still degrades.

The other half of the fix was cheaper and mattered more. Before any run, check that the tool you need is actually in the registry. If it isn't, say so and stop. Never quietly do the lesser thing.

Act IV: the same bug, wearing a doc

Months later, wireframe rounds started landing in Figma as flat PNGs. Three times in four days, across three different sessions.

I want to be precise about this, because I assumed it was a regression of Act III and it wasn't. The capture path was fine. When we finally ran it and counted, thirteen captured wireframes had landed as real layer trees, between ninety-three and three hundred and sixty-four nodes each, with no image fills anywhere.

The cause was four sentences in my own instructions.

A comparison table described the capture path as giving a "pixel-perfect frame" and used the word "editable" only for the other path, so a careful reader concluded capture meant pixels. A fallback clause blessed screenshots as "good enough for review canvases," and a wireframe round pushed for review is exactly a review canvas, so the fallback authorised itself in the one case it was meant to forbid. The strongest rule in the file, the one saying never downgrade, named a different fallback than the one anybody was actually taking. And the wireframe step never used the word editable at all.

Not one of those sessions was careless. Each was following the document. All four wordings had to change together, because fixing three of them still leaves an intact route to the same mistake.

Then a smaller one, which is my favourite thing in this whole saga. For those three sessions the blocker was recorded as an authorisation problem, inheriting the connector-shadowing story from Act III. It wasn't. It was an expired login on the local terminal. A bare probe with no Figma involvement at all failed identically, and that probe is one command, and nobody ran it, because we already had a satisfying explanation lying around.

What I took from it

The last mile of an AI workflow is almost always a plumbing problem, not a model problem. The clever part worked on day one. Runtimes, permissions, which integration owns a slot, and the wording of my own instructions took everything after that.

An AI that quietly does the lesser thing is more dangerous than one that fails loudly. A crash you catch. A confident downgrade you ship. So "did it actually use the right tool?" is a check the system runs on itself now, not a thing I trust.

And the one I'm still learning: a good explanation is not a diagnosis. Act III gave me a genuinely correct story about why captures fail, and then that story cost me three sessions of chasing the wrong thing, because it was easier to reach for than the one-line check that would have settled it.


Part of a series on building an AI collaborator for our design team at Xflow. Each post stands on its own.

  • ai
  • figma
  • prototyping

The work behind the series

Designing the Instructions

This post is one thread out of a three-month project: an AI design collaborator for a payments team. The case study is the whole of it: what worked, what broke, and what is still unproven.

Read the case study →