In a companion piece we showed that AI agent skills can make front-end output measurably more beautiful in a blind test. This article is the other half of the story — the one most "AI made our code better" posts skip: where the skills did not help, what they cost, and how we know the difference is real. If you're deciding whether to invest in agent skills, the gaps matter as much as the gains.
We built eight skills and tested them two different ways. The most important result is that the eight skills are not one thing — they split cleanly into two kinds that behave in opposite ways.
Two Kinds of Skills
It's tempting to treat "agent skills" as a single lever. They aren't.
Knowledge skills (seven of them)
These encode what good front-end is: CSS architecture, a design system, responsive layout, semantic HTML and accessibility, animation, modern CSS and browser support, and performance. They are reference material the agent loads on demand.
One process skill
The eighth, "Visual Polish," encodes how a designer works: build, render it in a real browser, screenshot it, critique it against a rubric, fix the weakest things, and repeat — plus a vetted design-system asset library and a strict "definition of done." It changes the agent's behavior, not just its knowledge.
This distinction turned out to explain every result below.
How We Measured It
Two evaluations, each comparing the same task built with skills versus unaided (no skills):
- Correctness eval — seven build tasks, graded by objective checks: does it use relative units, semantic HTML, accessible focus states, em-based breakpoints, native lazy-loading, and so on.
- Quality eval — two fresh tasks (a pricing section, a finance landing page), judged blind: an independent reviewer scored only the rendered screenshots, did not know which was which, and the labels were randomized so it couldn't pattern-match.
The Numbers
| Measure | With skills | Unaided | Verdict |
|---|---|---|---|
| Correctness pass rate | ~90% | ~95% | Effectively a tie |
| Blind quality — Task 1 | Won (high confidence) | Lost | Distinctiveness 9 vs 3 |
| Blind quality — Task 2 | Won (high confidence) | Lost | Distinctiveness 8 vs 3 |
| Time per task | +~33% | baseline | Skills cost more |
| Tokens per task | +~24% | baseline | Skills cost more |
Read those two blocks together and the whole story appears: on correctness, skills and no-skills tie; on craft, skills win decisively — but cost more.
Where Skills Win (the gains)
1. Craft and aesthetics — the decisive win
With the process skill, output went from the recognizable "generic AI default" to genuinely art-directed — and won a blind comparison on both tasks at high confidence. The reviewer, with no idea which design was which, independently called out the unaided versions' tells: the dark purple gradient and gradient headline text.
"One is art-directed and clearly custom-designed — inspiration-board material. The other leans on the two biggest generic-AI tells at once. The gap is clear, not marginal."
2. It catches real defects
Because the process skill forces the agent to render and look, it caught two production-grade bugs a code review missed: content that was invisible without JavaScript, and a layout that overflowed horizontally on mobile. A one-shot generation shipped neither — because nothing made it look at the result.
3. It enforces your conventions, every time
Skills reliably inject project-specific decisions a model won't choose by default: a consistent sizing strategy, em-based media queries, modern CSS gated by real browser-support data, reduced-motion safety, and layered (not flat) shadows. That consistency across hundreds of runs is itself a deliverable.
Where They Don't (the gaps)
The honest part. These are real and worth planning around.
Knowledge skills barely beat a strong model
A frontier model already knows relative units, semantic HTML, and accessibility basics. So on the correctness eval, the seven knowledge skills essentially tied the unaided model — and on one task the grader even scored them slightly lower. Their value is consistency and helping weaker or cheaper models, not raw lift on a top-tier one.
They cost more
Skilled runs used roughly 24% more tokens and 33% more time on the knowledge tasks; the full render-and-iterate loop is heavier still (dozens of build-and-check steps versus a handful). You are buying quality with compute — worth it when quality matters, wasteful when the job is "correct code, fast."
The best lever needs a browser
The process skill's superpower is rendering in a real browser. In an environment without one, it degrades to a knowledge skill and the biggest advantage disappears. Plan the toolchain accordingly.
Quality is hard to measure cheaply
Our objective correctness checks were poor at judging quality — they mostly passed both versions, and produced false negatives on the design task. Genuine quality assessment needed a human-style blind judge. And our quality eval was small: two tasks, one reviewer. Convincing, but not yet statistically robust.
The Insight That Ties It Together
One sentence captures it:
"Correctness is solved. Craft is not. The gap between competent and exceptional front-end isn't knowledge the model lacks — it's taste applied through iteration, and a refusal to ship the generic default."
That is why adding more knowledge barely moved the needle, while adding a process — render, critique, refine, against shipped-in taste — moved it a lot. A text instruction can't make a one-shot pixel-perfect. A skill that forces the agent to look at its own work and fix it can.
What This Means for Teams
- Don't expect a correctness lift from a strong model. If your evaluation only checks correctness, skills will look pointless — because you're measuring the thing the model already does.
- Invest in process, not just prompts. The durable gains come from verification loops the agent must run, not from longer instruction files.
- Ship taste, not principles. Vetted, opinionated defaults beat abstract advice every time.
- Measure quality the way users experience it — render it, look at it, compare blind. Boolean checks can't see beauty.
- Budget for it. Quality output costs more tokens and time; spend it where quality is the point.
Recommendations: Closing the Gaps
The findings point to a concrete to-do list — for our own skill suite and for any team building one:
- Scale the blind evaluation. Two tasks and one reviewer prove direction, not magnitude. Expand to ten-plus tasks across page types, add a second AI judge, and include at least one human reviewer. Track win rate and confidence, not a single verdict.
- Tune triggering, not just content. A skill that doesn't fire is worthless, and one that fires on the wrong task is worse. Run an automated description-optimization pass so each skill activates when it should and stays quiet when it shouldn't.
- Make the process skill degrade gracefully. When no browser is available, it should detect that and fall back to its knowledge and checklist instead of assuming it can render — so it never silently loses its biggest lever.
- Consolidate overlapping knowledge. Eight skills carry some duplicated surface — modern CSS, responsiveness, and performance touch the same ground. Sharper boundaries mean less to maintain as the platform moves.
- Route by intent. Use the heavy render-and-iterate loop when quality is the goal; skip it for "produce correct code, fast." The biggest waste is paying craft-level compute for a throwaway internal tool.
- Re-measure on a weaker model. Knowledge skills should help more where the base model is less capable; confirming that quantifies their real value for cost-sensitive deployments.
How SplatDev Applies This
This is the work we do for clients: not just pointing an AI at a codebase, but engineering the skills, guardrails, and verification loops that make its output reliably excellent — and then proving the result with honest evaluation rather than assuming it. We measure quality, not just speed, and we tell you where the gains are and where they aren't.
Want AI that ships work you'd put in a portfolio — and an honest account of its limits?
Contact SplatDev — Build Better With AI
Or email us at contact@splatdev.com