“We’re not just creating AI for AI’s sake, but we’re creating AI that serves a purpose, that solves a problem, that empowers people, that respects people.” —Haiyan Zhang

Haiyan Zhang is a creative technologist, designer, and maker who has worked on projects ranging from wearable devices for people with Parkinson’s to immersive experiences for social change. She is currently GM of Gaming AI for Xbox where she leads a team of researchers and engineers working on cutting-edge AI projects. She is also a passionate gamer and a former IDEO designer. Haiyan has a fascinating background that spans from China to Australia to the UK, and from web development to product design to social impact. In this interview, she shares how her early exposure to computers, her passion for the liberal arts, and her curiosity for human behavior influenced her career path and her vision for the future of gaming.

Haiyan’s Quick Backstory

“I wanted to challenge the reason why we were doing things.”

Haiyan started her career as a software engineer, working on biomedical and data warehousing applications. She was always curious about the purpose and value of the features she was coding, and wanted to understand the users and their needs better. She decided to pursue a master’s degree in interaction design in Italy, where she learned from pioneers like Gillian Crampton Smith and Bill Moggridge.

“I uncovered a whole host of other disciplines and learnings.”

At the Interaction Design Institute Ivrea, Haiyan was exposed to a variety of topics and perspectives that broadened her view of design and technology. She learned about the social and ethical implications of her work, and how to create solutions that benefit not only users, but also society at large. She also developed her skills in prototyping, storytelling, and user research.

“I think AI is a very powerful tool for creating new kinds of experiences.”

After graduating, Haiyan worked as a design researcher, a creative director, and a product innovation director at various organizations, including Microsoft Research, BBC, and IDEO. Along the way, she developed a passion for AI and its potential to create new kinds of experiences….

Featured Interview With Haiyan

“As a child, I was sneaking onto the Internet without my parents knowing about — back when the “tech space” was Usenet and the web was just all text based.”

John Maeda: I’m also from an engineering background myself, but my parents didn’t really have a interest in complex liberal arts topics. How did you bump into the world of people and society as an engineer? Did you always have that in you?

Haiyan Zhang: Definitely. Because I was born in China and grew up there right after the Cultural Revolution. I had a very early Communist-style upbringing. But when I was a kid, my parents moved to Australia when I was eight years old.

JM: Really? Woah that was a big shift!

Haiyan: Absolutely. And so I went from this kind of very Communist mental model where we were taught that Christmas was the ultimate kind of, like, bourgeois consumers thing, you know? We didn’t give gifts! I mean this is back in, I think the early 80s. China was just very, very strict about these things. So when we moved to Australia, my Dad was an Electrical Engineering professor, we had the first computers he had access to — especially with the laptop that he brought home. So as a child, I was sneaking onto the Internet without my parents knowing about — back when the “tech space” was Usenet and the web was just all text based.

JM: Usenet! That’s so nostalgic.

Haiyan: <laughter> Recently somebody on my team said, “I can’t believe Haiyan was a web developer before the web was visual!”

“My master’s thesis was on creating game controllers that clamped on to everyday objects and turned them into video game controllers.”

JM: So computers were a big part of your early life, and in your later lives as well? When did gaming come into play?

Haiyan: Well, gaming was something I was always interested in — but from a player’s perspective. When I was doing my graduate work, my master’s thesis was on creating game controllers that clamped on to everyday objects and turned them into video game controllers.

JM: Ah. That is very much of that era of research — very ITP!

Haiyan: Yeah. So I was very interested in play, and then when I then I went to work at IDEO for a number of years I did projects with some video game companies. And then I when I moved into Microsoft Research, I focused on play and physical digital interactions for a number of years. We invented a lot of hardware sensing technologies like Project Zanzibar.

JM: You’re making me think of Colin Burns and Durrell Bishop’s work.

Haiyan: Oh yeah, Colin Burns — that was slightly before me. I remember working with him at the British Design Council.

“AI was first imagined as part of a play scenario or a competitive game scenario — like the first chess algorithms to play a human.”

JM: Alright, so then why did you start combining AI in gaming? What was the motivation?

Haiyan: Oh, so … AI. I think if you work in in gaming and research space, AI feels like a natural area where even since the beginnings of AI as an academic research discipline, it was imagined as part of a play scenario or a competitive game scenario like the first chess algorithms to play a human.

JM: You’re right!.

Haiyan: So in hindsight, when AI was conceived it was about an AI that would play chess or play checkers. And this is in effect how it was benchmarked, and also how some of the tenets of what AI is today involve imagining it as a player in a game.

JM: Looking forward, what do we need to think when considering the tenets of AI and gaming in the future?

Haiyan: Well, I think there is no easy or definitive answer to them, but I think there are some principles and practices that can help us. For example, I think we need to be more human-centered and user-oriented in our AI design and development. We need to understand the needs, preferences, emotions, and contexts of the users and the games, and we need to involve them in the co-creation and evaluation of the AI. We need to be more collaborative and interdisciplinary in our AI research and innovation work. We need to work with people from different backgrounds, domains, and perspectives, and we need to learn from each other and share our insights and experiences. We need to be more experimental and creative in our AI exploration and expression. We need to try new things, test new ideas, and challenge ourselves and the AI. We need to be more critical and reflective in our AI impact and implication. We need to think about the consequences, the risks, and the opportunities of the AI. Because we’re not just creating AI for AI’s sake, but we’re creating AI that serves a purpose, that solves a problem, that empowers people, that respects people

“It was not until GPT where we saw a model that can generally go in and bridge the gap between AI research and how it can work in actual people’s hands on their machines.”

JM: Regarding all the pre-training foundation model work, in what year did you feel it changing?

Haiyan: Well until last year, I was primarily working more in reinforcement learning for game playing. We in Xbox have also used other language models and NLP for services like customer support and community moderation. However with the generative capabilities of GPT, we are able to imagine a lot more application in the games themselves.

JM: Yeah. That was state-of-the-art. <laughter>

Haiyan: Really, for the last five or six years I’ve seen reinforcement learning do amazing things. But I think the challenge has been the brittleness of those AI research models. It was not until GPT where we saw a model that can generally go in and bridge the gap between AI research and how it can work in actual people’s hands on their machines, or in the Cloud. And it’s actually able to do general thing across many different use cases without breaking. Where the programming, or prompting, of the model is democratized to empower everyone to create useful tools. That’s been a really revolutionary step.

JM: It’s mind boggling, indeed. Last question. You’re using Semantic Kernel (SK) to play with these emerging Large Language Models (LLMs). What were your impressions of SK? You’ve seen it come out of the ground, and you know it’s always changing. What are your general hopes for it?

Haiyan: Well, I had early access to the raw GPT APIs. Someone described it early on as we had entered a world where every house in the street is digging up the street to put plumbing in. You don’t as a designer/software engineer want to be putting the plumbing into every house in a completely different way. It’s nice to use Semantic Kernel as a paved path for how to bring AI into an app. Just layer your ideas on top of SK and its toolset so you can focus on your application instead of the plumbing. I think SK is really valuable and I loved seeing it emerge from an incubation into its open-source form in such a short amount of time. Beyond yourself, I know many of the folks on the team — and their passion as a team to bring SK to life has been an inspiration. So I’m a big, big fan. I’ve built with it. And, honestly, I found one bug in it, but I think you guys fixed it! <laughter>.

Haiyan Zhang is one of the many talented and innovative people who are using Semantic Kernel to create explore amazing new AI-powered applications. . If you want to learn more about Semantic Kernel and how you can use it to unleash your creativity, check out aka.ms/semantic-kernel.