Behind SPFx Dev Skills: testing what agents know and fixing what they miss

AI coding agents can build SharePoint Framework solutions. But should you trust them to choose the right toolchain and package versions by themselves, then apply every upgrade step correctly?

We wanted evidence. Before publishing SPFx Dev Skills, the SharePoint Framework team and Developer Relations worked together to understand what agents already know and where they struggle. We then measured which guidance improves the outcome.

It’s easy to package our expertise in a skill and tell developers to trust it. Proving where the skill adds value takes more work, but produces a far more useful result.

We started without the skill

The collaboration brought together Vesa Juvonen and Bert Jansen from the OneDrive and SharePoint SPFx team, and Garry Trinder and Waldek Mastykarz from Developer Relations. Collectively, we defined what a correct upgrade should look like and designed the evaluations, then traced failures through the agents’ complete sequence of actions. Together, we turned those observations into changes across the surfaces that agents use.

The evaluations also built on years of work by the CLI for Microsoft 365 maintainers. They have encoded deterministic, version-by-version SPFx upgrades in the spfx project upgrade command. We used that work as an input to our evaluations and, later, as the upgrade path we wanted agents to discover.

Our first scenario focused on upgrading an SPFx 1.21.1 project to 1.22.2. This requires more than changing package versions because SPFx 1.22 introduced the move from gulp to Heft, along with related changes to configuration and the project structure.

We asked GitHub Copilot Chat in Visual Studio Code on Windows, using Claude Sonnet 4.6 to do just that:

Upgrade the project to 1.22.2

Notice that we purposefully didn’t mention SPFx or point to documentation, and we ran the scenario without a skill. This gave us the baseline: what can the agent do with the project context and its existing knowledge?

Because language models are non-deterministic, we ran the same scenario 5 times. We checked whether the project still built and ran, but we also evaluated the details that distinguish a partial upgrade from a correct one.

Every run produced a project that passed our execution gates. At first glance, that looks like success. A closer look showed that the agent updated the headline SPFx version but missed dependencies and migration details. Starting with a baseline kept us from writing a large skill that repeated knowledge the agent already had while missing the problems that actually affected developers.

Copilot already knew a lot about SPFx

Before invoking a tool, the agent recognized the project as SPFx simply from its file tree! It knew that Microsoft packages needed updating and expected changes to the TypeScript toolchain. When it needed the release notes, it didn’t search for them. It constructed the full Microsoft Learn URL directly, following the URL pattern established across years of previous SPFx releases. Its weakness was combining that knowledge with the information it found.

For example, an agent asked to upgrade to 1.22.2 fetched the release notes for 1.22. It noticed that the documented version didn’t exactly match the requested version, but didn’t fetch every intermediate release page. That’s risky because SPFx upgrades are incremental, so changes in 1.22.0 and 1.22.1 still matter when the target is 1.22.2.

The agent also formed its plan before reading the documentation, then used the docs to confirm it. A tip suggesting CLI for Microsoft 365 didn’t change its approach because the page also offered detailed manual migration steps that looked actionable.

So, what should a skill add when the agent already recognizes SPFx and can find its docs? Precise decision rules for the places where generic knowledge is unreliable.

We tested the obvious answer

The SPFx team had already created a separate anti-hallucination skill. It instructed the agent to verify SPFx facts using authoritative documentation. We compared the baseline with this skill, then added the context7 MCP server to see whether another documentation source helped further.

The anti-hallucination skill improved the results and reduced average token use by roughly 9%, but directing the agent to authoritative documentation still left the project only partially upgraded.

Adding context7 didn’t provide meaningful additional lift. In 3 of 5 runs, its tools were available but never invoked. Availability alone didn’t make the tools relevant to the agent’s plan.

We had better numbers, but not enough to call the upgrade problem solved. Treating the anti-hallucination skill as the answer would have been the agent-development equivalent of saying, trust us, this helps, so we kept looking.

The best upgrade guidance already existed

CLI for Microsoft 365 includes the spfx project upgrade command. It analyzes the source and target versions, then generates the version-specific changes required for an incremental upgrade.

When we explicitly told the agent to use the CLI, dependency currency jumped from 34/50 to 50/50. Configuration correctness rose from 38/85 to 83/85.

An existing product tool produced the strongest results because it already encoded the upgrade logic. The agent didn’t discover it by itself, so we focused SPFx Dev Skills on guiding the agent to the CLI when an upgrade required it.

Documentation changed the agent’s plan

The SPFx release notes already included a tip about CLI for Microsoft 365. Even when we added the exact command, agents continued with their manual plan.

Why? Because a tip offers an alternative without telling the agent that its current approach is wrong. Meanwhile, the linked migration guide contained detailed manual steps supported by npm commands and JSON snippets. Agents consistently chose that actionable content.

We used Dev Proxy to test documentation changes without publishing unproven ideas to Microsoft Learn. Several attempts didn’t work. Moving the tip made no difference, and removing the migration guide hurt manual runs without reliably increasing CLI adoption.

Then we added a warning immediately before the tip, directly challenging the agent’s manual approach:

Upgrading from a previous minor version requires changes across package versions, build configuration files, and toolchain settings that vary depending on your source version. Manually updating package.json alone will result in build failures.

With the detailed migration link removed, CLI adoption moved from 0 of 5 runs to 5 of 5. The warning worked because it challenged the plan the agent had already formed.

We couldn’t simply remove useful guidance for human readers, though. When we restored the step-by-step migration guide, CLI adoption fell back to 0 of 5. So we rewrote the guide to explain what changes during the gulp-to-Heft migration while directing readers to CLI for Microsoft 365 to apply those changes. With the conceptual guide supporting the warning, agents used the CLI in 5 of 5 runs, and configuration correctness reached 85/85.

We submitted these improvements to the SPFx documentation. The release-note version changes shipped in PR #10855, followed by the warning and rewritten migration guidance in PR #10921. Putting these fixes at the source made them available to all developers and agents, including those without SPFx Dev Skills.

The evaluation shaped SPFx Dev Skills

Our evaluation covered the upgrade scenario and compared several ways of helping the agent, including the separate anti-hallucination skill. The results showed what SPFx Dev Skills should add and what it should leave to authoritative documentation or existing tools. The SPFx Dev Skills preview packages the SPFx-specific decision rules in 1 focused spfx skill and routes each request to a self-contained playbook.

Today, the skill covers these areas:

• creating SPFx web parts, extensions, libraries, and Adaptive Card Extensions;
• upgrading projects with CLI for Microsoft 365;
• choosing Heft for SPFx 1.22.0 and newer, or gulp for earlier versions;
• building React interfaces with Fluent UI v9; and
• using PnPjs for SharePoint and Microsoft Graph data access.

The upgrade guidance reflects what we observed during evaluation. It tells the agent to detect the installed version and check Node and TypeScript compatibility before acting. The agent then uses CLI for Microsoft 365 to apply every generated change in order, and it must finish with a clean build. The global guidance also tells agents to run long SPFx dependency installations synchronously, avoiding the repeated terminal polling we saw consume time and tokens during testing.

The skill complements the documentation and CLI by helping the agent choose the right source and use it correctly.

SPFx Dev Skills is currently available in preview. Its playbooks for creation and design extend beyond what we evaluated here, as do its toolchain and data playbooks. We’ll keep validating them through real usage, and developers should review the output before committing or shipping.

The collaboration improved more than the skill

The investigation also surfaced an opportunity beyond SPFx. Microsoft Learn supports returning pages directly as Markdown, giving agents cleaner content than converting the full HTML page. We shared this finding with the GitHub Copilot teams, who added Accept: text/markdown support to GitHub Copilot Chat and GitHub Copilot CLI.

Agent experience problems can originate outside a skill. In this case, some fixes belonged in documentation or the CLI, while another sat in the platform that retrieves content. Establishing the baseline showed us where to work.

Try the preview and help us improve it

You can explore SPFx Dev Skills on GitHub. The repository contains the portable spfx skill and its task-specific references, along with manual installation instructions for compatible AI coding agents. Try the preview with your SPFx work and review what the agent produces. When the result is wrong or surprising, open an issue. That evidence will help us decide what the skill should teach next, and what we should fix somewhere else.