Today, we’re going to start the next multi-million-dollar pizza chain. You might think that the first thing we need to make pizza is a pizza oven and the ability to make toast without burning it, but who ever became a millionaire by following the “conventional” approach? Instead, we’re going to start with the ordering system.

The days when you could just shout “supreme” and receive your pizza are gone. Everyone has very specific preferences, either for health reasons (“no cheese, I’m lactose intolerant”), taste reasons (“no anchovies”) or lack-of-taste reasons (“extra anchovies”), and a regular menu simply cannot handle all of the possible combinations. So we’re going to use a voice recognition system to take our orders.

Project Oxford is a bundle of cloud-based machine-intelligence tools. You can find everything from face detection and emotion recognition through to spell checking (try the demos from your browser). We are going to be using the speech APIs and the Language Understanding Intelligent Service (LUIS) to create our pizza store. All of these services are in preview right now and don’t cost anything.

We will use Python 3.5 and the projectoxford library to access the APIs. You can use Visual Studio Community 2015 to write the Python code, or any other editor. However, the audio APIs in the projectoxford library currently only work on Windows (you can help us fix this).

Step 1. Take the Order

Before integrating speech, we will start by making a pizza ordering script that works with the keyboard.

Start with a blank Python file, write a welcome message, and then ask the customer for their order. Feel free to change the name to whatever you like.

print("Hello, welcome to Fabrikam Pizza Shack")
order = input("How may I help you?")

It’s always polite to repeat the order back to the customer, so they can correct you if you misheard.

print("You said:", order)

We’ll come back to actually understanding the order soon, but right now we need to thank our customer so they come back again. Poor customer loyalty is a great way to make sure we never become millionaire pizza tycoons. (Make sure you use the same name here as when you welcomed your customer, or else they will be very confused.)

print("Thank you for visiting Fabrikam Pizza Shack")

That’s the basic structure! You can now test your pizza ordering system by pressing F5 (in Visual Studio) or running the script from PowerShell or the command prompt.

Step 2. Talk to your Customer

While our text-based interface is functional, it isn’t really as fluent as our customers expect or deserve. Luckily, we can very easily add both text-to-speech and speech-to-text functionality and interact directly with them.

First, we need to install the projectoxford library from PyPI. The easiest way to do this is using pip, or through Visual Studio.

C:\Projects\FabrikamPizzaShack> pip install projectoxford
Installing collected packages: projectoxford
Running setup.py install for projectoxford ... done
Successfully installed projectoxford-0.3.1

Next, visit www.projectoxford.ai/speech and subscribe to the speech API service. That will give you a “Primary Key” that you can save for later.

Finally, at the very top of our script, we need to import the speech API client, and then replace the normal print and input functions with ones that will use the microphone and speakers. You’ll need to paste your API key from above. You can also change the locale to match your own, which may improve speech recognition, but be aware that not all combinations of locale and gender will work. (Changing locale changes the language, so if you choose “es-MX” you’ll need to speak Spanish and not English.)

from projectoxford.speech import SpeechClient
sc = SpeechClient("PASTE-YOUR-KEY-HERE", gender='Male', locale='en-US')
print = sc.print
input = sc.input

Make sure you have your volume up, microphone ready, and run the script again. Instead of typing your responses, you can just speak after the beep to make an order. Try saying “I’d like pizza with some ham and tomato”, “how about pizza with cheese and bacon and cheese and bacon and cheese and bacon”, or if you’re ordering for someone else, “cheese pizza with anchovies, anchovies and anchovies”.

Step 3. Understand the Order

As fun as it is to have our pizza sales assistant read back exactly what you said, it doesn’t feel like you have been understood. You can order anything you like and it sounds like we’ll give it to you. (Go ahead, try ordering a golf club, a sports car, or world peace.) What will really make our pizza shop a money-maker is to understand what the customer has asked for.

To perform this processing, we will turn again to Project Oxford. LUIS takes simple examples of phrases and uses machine learning to create an service that extrapolates to whatever a user asks. For example, if we teach LUIS that “I’d like pizza with cheese and bacon” is a pizza order, LUIS can also identify other statements that look like pizza orders, and can extract information like the list of toppings. The more examples and more usage, the better it gets.

For our pizza shop, you can download a predefined set of examples from here. This is a JSON file with statements like “can I have pizza with pineapple and bacon” and “sausage and pepperoni on my pizza please”, some non-pizza requests such as “I would like a sandwich with bacon, lettuce and tomato”, and it has the classifications already included. We will import this into LUIS to save creating a new application, but you can watch this video to see how to start from scratch.

Go to luis.ai/ and sign in with your Microsoft Account. Once you are in, click the “New App” button and select “Import Existing Application”. Download the Pizza.json file and then select the downloaded file in LUIS, then click Import. Feel free to enter an utterance of your own here, or simply click “Train” in the lower-left corner. When training is complete, click the “Publish” button in the upper-left and then “Publish web service” to get your URL.

Back in our Python script, we want to import the LuisClient class from projectoxford and pass in our URL (make sure that you include the “&q=” at the end of the URL).

from projectoxford.luis import LuisClient
lc = LuisClient("YOUR-URL-GOES-HERE")

Now, after obtaining the order from the customer, we can call into LUIS to see what the customer is asking for. The three parameters returned from query are the intent, a list of entity names (in this case, the toppings), and a list of entity types (here we only have “Topping”, but you can support different types).

print("You said: ", order)
intent, toppings, _ = lc.query(order)

If the intent is “Pizza” and we have at least one topping, we know what the customer has asked for. If a customer asks for something else, such as a sandwich or golf clubs, the intent will not be “Pizza” and so we will apologize. The speech module has a helpful join_and function that will make the list of toppings into something that can be read out loud.

from projectoxford.speech import join_and

if intent == "Pizza" and toppings:
    print("I will send you a pizza with", join_and(toppings))
else:
    print("Sorry, we only sell pizza here.")

print("Thank you for visiting Fabrikam Pizza Shack")

Summary

With our functional, automated pizza ordering system, we are sure to very quickly succeed in this market. If you want to keep extending the system, here are a few ideas you could try.

• Determine how much the pizza will cost, based on the toppings requested
• Expand your empire by training LUIS to recognize sandwich orders
• Change the voice in the speech client (personally, I like gender="Female", locale="en-AU")

With Project Oxford and Python, you can very quickly create highly interactive applications that hear what you say, know what you mean, and can talk back to you. Let us know in the comments what you create, and visit the Visual Studio blog to find ten other new things to try.