Early Lessons From GPT-4: The Schillace Laws

John Maeda

March 23rd, 20231 4

What if you could use natural language to create software? What if you could leverage the power of a large-scale language model that can generate code, data, and text from simple prompts? What if you could balance the trade-offs between leverage and precision, uncertainty and interaction, complexity and simplicity? These are some of the questions that Sam Schillace, a software engineer and entrepreneur, explored when he had early access to GPT-4, the latest version of OpenAI’s generative pre-trained transformer model.

Based on his experience with GPT-4, Microsoft’s Deputy CTO Sam Schillace developed nine principles for using LLMs to create software. We call them the “Schillace Laws”:

Don’t write code if the model can do it; the model will get better, but the code won’t.
Trade leverage for precision; use interaction to mitigate.
Code is for syntax and process; models are for semantics and intent.
The system will be as brittle as its most brittle part.
Ask Smart to Get Smart.
Uncertainty is an exception throw.
Text is the universal wire protocol.
Hard for you is hard for the model.
Beware pareidolia of consciousness; the model can be used against itself.

These principles capture some of the best practices and challenges of using LLMs to create software, especially in domains where natural language is involved or desired. They also reflect some of the strengths and weaknesses of LLMs as a new partner in software development.

A Quick Look At Principle Five From LinkedIn Learning

LinkedIn Learning has a free course available on Semantic Kernel that includes an overview of the Schillace Laws. Here’s a sneak peek:

Upon completion of the course, a certificate is awarded that you can share on your LinkedIn profile.

The Schillace Laws In Practice

One of the practical questions that I asked Sam Schillace was how to use GPT-4 to generate valid JSON expressions from natural language prompts. JSON is a popular data format that is widely used for web applications and APIs. It is also a format that can be easily parsed and validated by conventional computer code.

However, generating JSON expressions from natural language prompts can be tricky and error-prone, especially when using LLMs with their non-deterministic nature. How can we ensure that LLMs produces accurate and relevant results? How can we deal with the uncertainty and variability of LLM outputs? How can we determine the boundaries between syntax and semantics in our programs?

Sam Schillace elegantly suggested three different strategies to guide LLMs to produce valid JSON expressions from natural language prompts:

Lowering the temperature to 0 so that there’s greater determinism in what comes back.
At the end of the prompt, start the JSON expression exactly the way you want it so that LLM can start from that point forward.
Use a conventional JSON parser to feed what comes out of the LLM. And if an error is generated by the parser, feed that error back into the LLM with what it came back with, and keep doing so until the parser gives the JSON a clean bill of health.

These strategies illustrate some of the Schillace Laws in action, such as:

Don’t write code if the LLM can do it. Lowering the temperature to 0 reduces the need for filtering the output.
Trade leverage for precision; use interaction to mitigate. Providing a template for the LLM’s output trades some leverage for more precision, but also allows for interaction with the LLM to refine or modify its output.
Uncertainty is an exception throw. Using a deterministic parser as a validator handles the uncertainty of an LLM’s output gracefully and intelligently, and sending it back to the model gives it a second, third, or more chances to get it more correct.

Using LLMs to create software requires a different mindset and approach than using conventional computer code alone. This example shows how LLMs can complement code by providing semantic and intent-based capabilities that are hard or impossible to achieve with code alone.

Schillace Laws In A Nutshell

The Schillace Laws fall into roughly three buckets:

The relationship between code and models, and how to use them effectively and efficiently for different aspects of software development.
The trade-offs and challenges of using LLMs to create software, and how to balance leverage and precision, uncertainty and interaction, complexity and simplicity.
The limitations and opportunities of using LLMs to create software, and how to avoid false assumptions or illusions about the model’s capabilities, while also exploiting its weaknesses or blind spots.

Relationships Between Code And Models

Don’t write code if the model can do it;the model will get better, but the code won’t.
Code is for syntax and process; models are for semantics and intent.
Text is the universal wire protocol.