{"id":26109,"date":"2026-09-02T03:14:00","date_gmt":"2026-09-02T10:14:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/?p=26109"},"modified":"2026-09-02T03:14:00","modified_gmt":"2026-09-02T10:14:00","slug":"behind-spfx-dev-skills-testing-what-agents-know-and-fixing-what-they-miss","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/microsoft365dev\/behind-spfx-dev-skills-testing-what-agents-know-and-fixing-what-they-miss\/","title":{"rendered":"Behind SPFx Dev Skills: testing what agents know and fixing what they miss"},"content":{"rendered":"

AI coding agents can build SharePoint Framework solutions. But should\nyou trust them to choose the right toolchain and package versions by\nthemselves, then apply every upgrade step correctly?<\/p>\n

We wanted evidence. Before publishing SPFx Dev\nSkills<\/a>, the SharePoint Framework team and Developer Relations worked\ntogether to understand what agents already know and where they struggle.\nWe then measured which guidance improves the outcome.<\/p>\n

It\u2019s easy to package our expertise in a skill and tell developers to\ntrust it. Proving where the skill adds value takes more work, but\nproduces a far more useful result.<\/p>\n

We started without the skill<\/h2>\n

The collaboration brought together Vesa Juvonen and Bert Jansen from\nthe OneDrive and SharePoint SPFx team, and Garry Trinder and Waldek\nMastykarz from Developer Relations. Collectively, we defined what a\ncorrect upgrade should look like and designed the evaluations, then\ntraced failures through the agents\u2019 complete sequence of actions.\nTogether, we turned those observations into changes across the surfaces\nthat agents use.<\/p>\n

The evaluations also built on years of work by the CLI for Microsoft 365<\/a> maintainers.\nThey have encoded deterministic, version-by-version SPFx upgrades in the\nspfx project upgrade<\/code> command. We used that work as an input\nto our evaluations and, later, as the upgrade path we wanted agents to\ndiscover.<\/p>\n

Our first scenario focused on upgrading an SPFx 1.21.1 project to\n1.22.2. This requires more than changing package versions because SPFx\n1.22 introduced the move from gulp to Heft, along with related changes\nto configuration and the project structure.<\/p>\n

We asked GitHub Copilot Chat in Visual Studio Code on Windows, using\nClaude Sonnet 4.6 to do just that:<\/p>\n

Upgrade the project to 1.22.2<\/code><\/pre>\n
Notice that we purposefully didn\u2019t mention SPFx or point to\ndocumentation, and we ran the scenario without a skill. This gave us the\nbaseline: what can the agent do with the project context and its\nexisting knowledge?<\/p>\n
Because language models are non-deterministic, we ran the same\nscenario 5 times. We checked whether the project still built and ran,\nbut we also evaluated the details that distinguish a partial upgrade\nfrom a correct one.<\/p>\n\n\n\n\n\n\n\n\n
Classification<\/th>\nPassed<\/th>\nTotal<\/th>\n<\/tr>\n<\/thead>\n
Prerequisite<\/td>\n5<\/td>\n5<\/td>\n<\/tr>\n
Idiomatic use<\/td>\n7<\/td>\n10<\/td>\n<\/tr>\n
Dependency currency<\/td>\n34<\/td>\n50<\/td>\n<\/tr>\n
Configuration correctness<\/td>\n38<\/td>\n85<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Every run produced a project that passed our execution gates. At\nfirst glance, that looks like success. A closer look showed that the\nagent updated the headline SPFx version but missed dependencies and\nmigration details. Starting with a baseline kept us from writing a large\nskill that repeated knowledge the agent already had while missing the\nproblems that actually affected developers.<\/p>\n
Copilot already knew a lot about SPFx<\/h2>\n
Before invoking a tool, the agent recognized the project as SPFx\nsimply from its file tree! It knew that Microsoft packages needed\nupdating and expected changes to the TypeScript toolchain. When it\nneeded the release notes, it didn\u2019t search for them. It constructed the\nfull Microsoft Learn URL directly, following the URL pattern established\nacross years of previous SPFx releases. Its weakness was combining that\nknowledge with the information it found.<\/p>\n
For example, an agent asked to upgrade to 1.22.2 fetched the release\nnotes for 1.22. It noticed that the documented version didn\u2019t exactly\nmatch the requested version, but didn\u2019t fetch every intermediate release\npage. That\u2019s risky because SPFx upgrades are incremental, so changes in\n1.22.0 and 1.22.1 still matter when the target is 1.22.2.<\/p>\n
The agent also formed its plan before reading the documentation, then\nused the docs to confirm it. A tip suggesting CLI for Microsoft 365\ndidn\u2019t change its approach because the page also offered detailed manual\nmigration steps that looked actionable.<\/p>\n
So, what should a skill add when the agent already recognizes SPFx\nand can find its docs? Precise decision rules for the places where\ngeneric knowledge is unreliable.<\/p>\n
We tested the obvious answer<\/h2>\n
The SPFx team had already created a separate anti-hallucination\nskill. It instructed the agent to verify SPFx facts using authoritative\ndocumentation. We compared the baseline with this skill, then added the\ncontext7 MCP server to see whether another documentation source helped\nfurther.<\/p>\n\n\n\n\n\n<\/colgroup>\n\n\n\n\n\n\n\n
 
Classification<\/th>\nBaseline<\/th>\n+ Anti-hallucination skill<\/th>\n+ Anti-hallucination skill and\ncontext7<\/th>\n<\/tr>\n<\/thead>\n
Prerequisite<\/td>\n5 \/ 5<\/td>\n5 \/ 5<\/td>\n5 \/ 5<\/td>\n<\/tr>\n
Idiomatic use<\/td>\n7 \/ 10<\/td>\n10 \/ 10<\/td>\n8 \/ 10<\/td>\n<\/tr>\n
Dependency currency<\/td>\n34 \/ 50<\/td>\n40 \/ 50<\/td>\n39 \/ 50<\/td>\n<\/tr>\n
Configuration correctness<\/td>\n38 \/ 85<\/td>\n46 \/ 85<\/td>\n47 \/ 85<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The anti-hallucination skill improved the results and reduced average\ntoken use by roughly 9%, but directing the agent to authoritative\ndocumentation still left the project only partially upgraded.<\/p>\n
Adding context7 didn\u2019t provide meaningful additional lift. In 3 of 5\nruns, its tools were available but never invoked. Availability alone\ndidn\u2019t make the tools relevant to the agent\u2019s plan.<\/p>\n
We had better numbers, but not enough to call the upgrade problem\nsolved. Treating the anti-hallucination skill as the answer would have\nbeen the agent-development equivalent of saying, trust us, this\nhelps<\/em>, so we kept looking.<\/p>\n
The best upgrade guidance already existed<\/h2>\n
CLI for Microsoft 365<\/a> includes\nthe spfx project upgrade<\/code> command. It analyzes the source\nand target versions, then generates the version-specific changes\nrequired for an incremental upgrade.<\/p>\n
When we explicitly told the agent to use the CLI, dependency currency\njumped from 34\/50 to 50\/50. Configuration correctness rose from 38\/85 to\n83\/85.<\/p>\n\n\n\n\n\n\n\n\n
Classification<\/th>\nBaseline<\/th>\n+ CLI for Microsoft 365<\/th>\n<\/tr>\n<\/thead>\n
Prerequisite<\/td>\n5 \/ 5<\/td>\n5 \/ 5<\/td>\n<\/tr>\n
Idiomatic use<\/td>\n7 \/ 10<\/td>\n4 \/ 10<\/td>\n<\/tr>\n
Dependency currency<\/td>\n34 \/ 50<\/td>\n50 \/ 50<\/td>\n<\/tr>\n
Configuration correctness<\/td>\n38 \/ 85<\/td>\n83 \/ 85<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
An existing product tool produced the strongest results because it\nalready encoded the upgrade logic. The agent didn\u2019t discover it by\nitself, so we focused SPFx Dev Skills on guiding the agent to the CLI\nwhen an upgrade required it.<\/p>\n
Documentation changed the agent\u2019s plan<\/h2>\n
The SPFx release notes already included a tip about CLI for Microsoft\n365. Even when we added the exact command, agents continued with their\nmanual plan.<\/p>\n
Why? Because a tip offers an alternative without telling the agent\nthat its current approach is wrong. Meanwhile, the linked migration\nguide contained detailed manual steps supported by npm commands and JSON\nsnippets. Agents consistently chose that actionable content.<\/p>\n
We used Dev Proxy<\/a> to test\ndocumentation changes without publishing unproven ideas to Microsoft\nLearn. Several attempts didn\u2019t work. Moving the tip made no difference,\nand removing the migration guide hurt manual runs without reliably\nincreasing CLI adoption.<\/p>\n
Then we added a warning immediately before the tip, directly\nchallenging the agent\u2019s manual approach:<\/p>\n
Upgrading from a previous minor version requires changes across\npackage versions, build configuration files, and toolchain settings that\nvary depending on your source version. Manually updating package.json\nalone will result in build failures.<\/p><\/blockquote>\n
With the detailed migration link removed, CLI adoption moved from 0\nof 5 runs to 5 of 5. The warning worked because it challenged the plan\nthe agent had already formed.<\/p>\n
We couldn\u2019t simply remove useful guidance for human readers, though.\nWhen we restored the step-by-step migration guide, CLI adoption fell\nback to 0 of 5. So we rewrote the guide to explain what changes during\nthe gulp-to-Heft migration while directing readers to CLI for Microsoft\n365 to apply those changes. With the conceptual guide supporting the\nwarning, agents used the CLI in 5 of 5 runs, and configuration\ncorrectness reached 85\/85.<\/p>\n
We submitted these improvements to the SPFx documentation. The\nrelease-note version changes shipped in PR\n#10855<\/a>, followed by the warning and rewritten migration guidance in\nPR\n#10921<\/a>. Putting these fixes at the source made them available to all\ndevelopers and agents, including those without SPFx Dev Skills.<\/p>\n
The evaluation shaped SPFx Dev Skills<\/h2>\n
Our evaluation covered the upgrade scenario and compared several ways\nof helping the agent, including the separate anti-hallucination skill.\nThe results showed what SPFx Dev Skills should add and what it should\nleave to authoritative documentation or existing tools. The SPFx Dev Skills<\/a>\npreview packages the SPFx-specific decision rules in 1 focused\nspfx<\/code> skill and routes each request to a self-contained\nplaybook.<\/p>\n
Today, the skill covers these areas:<\/p>\n