Deploy Into Azure Using Pulumi and GitHub Actions
On this week’s episode of The DevOps Lab we have Kat Cosgrove and Matty Stratton from Pulumi joining us!
In this week’s episode we take our Infrastructure as Software a step further and deploy into Azure, using GitHub Actions. Check out the episode below:
02:45 – Why do tests and pipelines matter for your infrastructure as code?
3:57 – Review previous project and tests from the past two episodes
08:15 – Add our unit tests to our GitHub pipeline
13:13 – Adding the Pulumi GitHub Action to perform a Pulumi Preview
April: Welcome to this week’s episode of the DevOps lab with episode three, with Matty Stratton, back from Pulumi. We also have Kat Cosgrove joining us as we talk about infrastructure as software and using Pulumi and GitHub Actions. Welcome Matty and welcome Kat.
Kat: Hello.
April: Really good to have you both back. Kat, can you take a minute and tell everyone out there what you do for Pulumi?
Kat: Yeah. So I’m a staff developer advocate at Pulumi, which ultimately means that my job is to teach people about infrastructure as code in general. But also if they’re interested specifically how to use Pulumi, if I think it is a good fit for them.
April: Awesome. Welcome to the show and we have Matty back with us. So Matty, welcome again. Tell everyone out there that may have missed the last episodes. What is it you do for Pulumi?
Matty: The funny thing is I do exactly what Kat does except, you know, maybe not as, not as amusingly, I don’t know.
Maybe Kat’s funnier than I am. We’ll find out. We’ve got, 20 minutes to find out who can be. I’m also doing a developer advocacy at Pulumi and just try to help people, see how treating their infrastructure as software can make things more awesome. I come on streams like this, and, just hang out in the community.
April: Wicked. Well, we’re going to have a little judge in the audience. Who’s funnier at the end of this video.
Let’s get down to it. Let’s talk about Pulumi. I’m a big fan of infrastructure as code. I’m really excited to have you all. Today we’re going to talk about having infrastructure as software with Pulumi and deploying it with GitHub Actions. Go ahead and take it away.
Matty: Yeah. I’m going to give a little recap and then I’m going to let Kat kind of talk about how we want to do stuff, but just for a catch-up, this is the third DevOps Lab that we’ve been on. Just sort of building, I don’t want to assume that you all have watched them all.
Why would you not have? Because I am so funny. In our first episode, we wrote some code to deploy an app onto a VM. Then the last time we said: ‘Hey, wouldn’t it be great if we put some tests around it?’ And then what we want to do now is send, now let’s put this all in a pipeline.
We’re going to do a review of kind of what we’d had, but Kat, maybe talk a minute about why those three things matter.
Kat: Those three things matter because core pieces of building and actually deploying an application. While you can manually deploy everything every single time from your console without a CI/CD system in place to automate things like running your tests, standing up new infrastructure and actually deploying your application, that’s slow.
Also it kind of sucks, it’s boring, and humans are really, really prone to making mistakes when we do the same thing over and over and over again. Cause we get bored and we get distracted and we have bad days and your CI/CD pipeline does not have those problems. This is also cool because doing it with Pulumi in a pipeline, allows us to test our infrastructure, which is rad and something that.
Now a lot easier to do than it used to be. In some ways it was previously impossible and now it is both possible and pretty easy.
Matty: Definitely. Great. Why don’t we pull up my screen. I’m going to kind of walk through this repo here. Awesome Azure. This is the code that we’ve been working over the last couple episodes.
Not all of the different kinds of files you will see in this repo are necessary for plumbing program because. Never write software the way that I do. But one of the things we didn’t really touch on a little bit when we talked about one of the cool things with is that when you write a Pulumi program to describe and deploy your infrastructure, you’re writing that in a general purpose programming language, like TypeScript or Go or Python or any of the .NET languages.
I wrote this in TypeScript. So this index dot Ts here, this is my program. That is going to go ahead and deploy our VM. And I’m going to go over this kind of quickly, because if you want to really see how it works, you know, go head back to that first DevOps Lab episode, go back to the back catalog, but we do a couple of fun things here.
We’re using these packages that are coming in. One of the things that’s important is this idea of a native package in Pulumi. Because we’re using, what’s called the Azure native package, this package, which in this case is a node package, but it could be a Go library or anything. These are built basically daily, according to the open API spec that’s published from ARM.
As soon as something is available in the Azure API, Pulumi, within 24 hours, you know, pretty damn close. So that’s really powerful. But all we have to do is when we bring these in, we can start creating objects. So we do things like we need a resource group, we need to create a virtual network.
We’re giving it a public IP and our web servers, not super exciting. The way it’s running is just running the Python. Built-in web server just to echo ‘hello world’. We do a little bit more so we can see all of these things that we’re building with our infrastructure as software.
And then we export out what the public IP address so we can use elsewhere. The couple of things I want to call your attention to are because these are objects. So we created an object called resource group. Then we refer back to it later. So we don’t have to, when we’re creating our network interface and we know the resource group we wanted to go into, we don’t have to know the name of it.
That’s because we’re writing this in JavaScript and TypeScript. We can use things like variables and stuff, and we can use loops. Now I want you to notice one thing. We have a tag on here, we’re saying, okay, we’re going to give our VM a name now. So that tells us what our Pulumi program will do. The next thing we’re going to look at is we said we had some tests, right?
We wanted to be able to run tests. In this case, because our Pulumi program is written in JavaScript. We’re going to use MOCHA, which is what testing framework you can use, whatever testing framework, your programming language supports, which is one of the dope things about writing and for code in a general purpose programming language, as you have the whole ecosystem of that programming language.
So what we’re doing here is this is a unit test, which means we’re mocking it. So we’re not actually readily running Pulumi and creating a resource we’re pretending to. And we’re saying what would happen? We want to make sure that every server that we create, it has to have a name tag.
We don’t want us, we don’t want to write it. In for code that will create a servers without name tag, we VMs without name tags. Cause then how do you know what they are? Who they are, who to go yell at when they forget to turn them off? So this is saying if they’re for missing a name tag on the server, then if it doesn’t have this tag, then say, it’s okay.
What we did last time is we ran that locally, right? I could on my local machine where I’m running this, I can run the test before I push it up. Now we want to bring it into into our CI/CD and which means we want to bring it into our pipeline so that every time we push code. We’re going to actually see this test run.
There are two things. I’m going to show you two of the workflows that we’ve built and then we’ll kind of talk them through. The first one is the one that’s the test. Um, This isn’t a GitHub Action. So if you haven’t used GitHub Actions before, they’re pretty easy to use,even I can use them.
You can build them with a template. In this case, I actually built this one using just sort of a default template for a Node application, because that’s fundamentally what this is. Right. So what we’re saying here is we’re saying the first thing is, we’re saying, what do you do? Right. We want this.
We are either pushing to ‘main”. So anytime we’re merging or pushing directly to our trunk, we should run this test, but also pull requests that are against the main branch. Now in this case, you know, we could do a matrix build. If we want to test against lots of versions of Node, we don’t really need to do that here.
And what is happening? As we said, okay, we need to have Mocha installed. We get our NPM install and then we run NPM test. And if you aren’t familiar with how Node works, in our package dot JSON, we have a script, which is test. And we say, when we’d run the word, when we run, if you run NPM test on this package, it should run this.
This is basically, you could also do. Running it at the command line, it’s a little cleaner to run it this way. Cause then, you know, if you’re lazy and don’t want to type, I’m very lazy. So Kat, that was just sort of seeing the tests. I know it’s going to sound a little silly, but like why do we care about like enforcing, testing in our pipeline here?
Kat: Because again, humans are bad at this. And computers are not bad at this. There are all kinds of things that could go wrong that you can’t really anticipate. So it’s important to make sure that core functionality like your infrastructure is actually there and operational, at least as far as the rest of your application code expects to be able to interact with it.
If you. Trusted and ignore that, like sure. It might, it might be fine like most of the time, but you’re not going to know until it’s actually live in production, whether or not it’s going to work. And that’s kind of, um, horrifying at least to me. So you want to make sure before you actually. Merge into Maine.
And your application is out there for your users, that your, your infrastructure is actually there and your application can interact with it the way it’s supposed to be interacting with it. Otherwise, you’re just going to push that update and everything’s going to go down and it’s going to be horrible and embarrassing and stressful.
And you’re going to page all of your engineers at like, God only knows what time, and they’re all gonna be really mad at you. And I don’t want to deal with that.
April: There’s business reasons why as well, if we push something to production, not only do you look back. It adds risk.
It also adds cost to whatever it is you’re doing as an organization. When you’re writing infrastructure as code, if you mess something up and you’re automating something, if you’re not running a test with it, you could absolutely destroy everything. I’ve seen people like deploy something and it disappears or breaks everything, we’ve all done it.
Testing is important. It’s something that, especially with infrastructure as code used to be really hard and we didn’t do a lot of it. We’d be like, ‘oh, let’s see what happens when we push this little button.’ It’s so much better. It’s so much more inclusive so that when we run that CI/CD pipeline, we know exactly what’s going to happen.
When we have a lot of organizations when they migrate to the cloud or say they can’t due to compliance, it gives you full traceability. That’s the other thing like compliance is, is such an easier discussion. If you put in the right testing and procedures for your CI/CD pipelines, you fulfill that compliance and you have that full traceability instead of going through who pushed what button, when.
Kat: logs, you know, so everything is locked.
Which is rad and like offer most engineers are going to take down production at some point, right? It happens. It’s part of what happens and it’s deeply embarrassing the first time you do it, you want the story to at least be good. The first time you take down production, you don’t want it to be because you’ve decided not to test your infrastructure.
That’s not a fun story for Twitter.
Matty: I want to show one more workflow and then we’re going to sort of go through this. So the other thing, so this is what I did so far was just. Same thing you could do. It’s just some, some fun node testing, but Pulumi has a GitHub Action itself which you can use.
What we wanted to do here is say on a pull request, run a Pulumi preview. So if you remember in the past, when you run Pulumi up, before it applies to changes, it just shows you basically. I don’t want to say it’s testing your application. It’s not, but it’s making sure that it’ll run.
It sort of does this compile, does this run, does this execute and it’ll tell you what it would do. That’s a gloomy preview. So we said we want pull requests to run a Pulumi preview. So that means every time there’s a pull request, it’s going to prove it’s not going to touch my Azure infrastructure, but it’s going to tell me what would happen.
I’ve set the secret. This is one of the things you could do with GitHub Actions in the settings, on the repo, you can set an encrypted thing. So I needed to have the access token. It needed to know my Azure credentials to be able to do that. And it’s just going to run Pulumi preview.
We’re going to do two things right now. If we did a new pull request, my tests would pass because. We have a tag, right? We have tagged named Floyd. So we’re going to create a PR, that’s going to have a breaking test. After I start the PR, we’re going to do a little quiz.
Cause I’m curious, we’re going to see if Kat and April can predict what the GitHub Actions will do when we get rid of our tag, what our result is going to be. So we’re going to say break production, like a real engineer. Okay. We’re going to create a new branch. That’s going to give us our pull requests, this breaks stuff, and also here’s a detailed.
Just put your army a good engineer.
April: Cause you’re putting
Matty: in. I know that’s what I was. I was just, I was, I was trying to think about like the starkest way I could say like lorem ipsum or hipster ipsum in here or something. So we’re going to create, there’s also startup Epsom, which I really like.
If we take a look, it’s building up our two checks. It’s doing the preview and it’s running the unit tests. And it’s going to take a second to kind of go through this they’re in progress. If we go take a look here, we look at the details. We can see it’s running, it’s running all through those pieces.
This is how GitHub Actions work. You’ve defined things, but what’s fun is you notice they’re happening in parallel. Which in this case, but we can make that, I didn’t even want to bother running the Pulumi preview if the unit tests failed, which honestly I might want that I could create not a dependency on the other water or build them into one workflow file.
In this case, it just seemed efficient to do them both at the, at the same time. If we go look at our Pulumi preview, it’s going to run NPM install,which takes a few minutes.
We can see that it’s gotten that stuff in there and now it’s actually going to ahead and it’s going to run our Pulumi preview against that. Okay. So before we can see, uh, April and Kat, what do you think is going to be the result of these two checks?
Kat: Surely they will succeed or are going to fail.
It’s going to work.
April: It’s going to fail. You forgot the tag.
Matty: What happens is you’ll see that actually the preview was successful, which is fine. Cause the preview is not running the test. The preview is saying if I was to, if you were to run this, this is what it would do. It would create all those things.
But our unit tests failed. And if we were to go into that, we can see it says error, missing a name tag on the server, and then we got out of there. So one last thing that’s kind of fun. So we’ll see, this is the Pulumi bot leaves this. If I look at the update and I click on this, it takes me into my account for my stack, and I can see the whole run.
There’s no diagnostic the diff right. You know, I can see all the details of what would have happened. If we look at resources, we’ll see, there are no resources, it didn’t create any resources. Cause we just did a preview. And then again, we look into the activity, we can see that I ran this preview.
You can also see that I was testing this right before the stream, but that’s fine. And like I said, if we wanted to, we could go ahead and we could make this actually deploy the infrastructure when we merge this to main. But if we wanted to fix this, we would just go back in and we would just add our name tag.
We know that’s going to pass because we’ve seen it so we should still test it. Let’s go ahead and we’re going to edit the file. And where did the VM go? It goes down.
April: I feel like we need to spin up a GitHub Codespaces for you.
Matty: I should have actually just done this in, I should’ve just done this in there.
And the other thing I’m going to do is I’m going to say, because right now the change I made doesn’t show it’s going to actually show no diff because I put it right back, but we go ahead. We should see, we made our commit. We should see that it changed. Yup. We added our tag back and then if we look back at our tasks, they’re going to run again and then when it’s done, we’ll, we’ll see the result.
April: and this is where the, a good engineer after you failed that test, won’t allow that merge, that pull request to main and it won’t go to production. So you should have those guardrails in place. The folks running the infrastructure ascode to say,it didn’t fail. It failed the test. Didn’t pass all the tests.
We’re not gonna push it, which would be really good practice as an engineer to not push broken code.
Kat: Okay. Correct. I mean, not here, it’s broken all the time, but I don’t merge it into main.
Matty: I think it’s like.
If we could create, I know we’re getting ahead of ourselves on that, but this is, I think, what we’re talking about is requiring status checks to pass, right? We could say match against main and you can’t merge unless the stuff goes and you should also do this check because just because you’re an admin doesn’t mean that you get to do whatever you want. That said do, as I say, not as I do. Cool. So that’s sort of chucking away, Kat, you want to kind of recap the whole situation?
Kat: We’ve taken an application that Matty has built slowly over time, over two episodes of this and taken it from nothing, just an application that we stand up with Pulumi to something that we also test with Pulumi, to something that now exists entirely inside of your CI/CD pipeline.
So that every time you make an application update, your infrastructure is going to be set up part as part of your CI. And it’s going to be tested as part of your CI. So you can be sure that every change you make to your application. Infrastructure is absolutely there and working as intended by the time it makes it to main because you’ve got this safety harness, which is great because for the thousandth time, I think I say this in like every talk I give to humans are super bad at repetitive tasks.
It’s really important that we take these things out of your engineers’ hands and we also know if something does go wrong, we have a record of every step along the way, everything that occurred to lead to some kind of infrastructure failure we know about in one way or another. Because when something goes wrong, you got to go fix it.
You can’t begin to unravel it. If you don’t really know what went wrong, you don’t have an idea of where to start. So this all helps. You make sure that everything is packaged up nicely. Everything knows where it lives. Everything is healthy. And if something does kind of fall over, it’s easier to track down why it fell.
Matty: Absolutely. I want to add one thing that we didn’t touch on, but I think it’s helpful to know is we showed an example of where the only thing in this repo was the infrastructure code. What you absolutely can sidecar your application code in your interest, which is a more common use case.
Again, it depends on how you’re doing. Maybe you’re building the systems for other people. Maybe you’re doing that, but you could have, for example, in your repo that has your Node app or your .NET app or your whatever app, it doesn’t even matter. We’ll have a folder maybe called ‘infra’ that has the Pulumi program in it.
So that whole repo, that can be running the checks on your application. We can also be checking your, infra code. And then also, like we said, on merge to main, it can then actually deploy the infrastructure that your applications.
Kat: Yep. This is how I build my personal website.
I have a separate directory for my infra code and a directory for my actual website code and all of it happens within the same repo, all that happens in the same pipeline. And I don’t have to worry about it. All of it gets tested every time I update anything.
April: Awesome. And there’s also things you could do with a branch policy.
So if you’re not touching the interrupt, but there’s a dependency, it will test the infra and how your code affects it, but not having to run everything.
Kat and Matty, thank you both for being on The DevOps Lab! Really excited to finish off this episode, three of using Pulumi in Azure. If you guys haven’t seen the first two episodes, please go check them out.
Check out episode 1 with Matty Stratton, Infrastructure as Software with Pulumi and Azure:
Check out episode 2 where Matty and myself talk about Testing your infrastructure Code with Pulumi:
Learn about DevOps and subscribe:
The DevOps Lab on YouTube
Azure DevOps YouTube Channel
Recommended resources:
Azure DevOps Hands on Labs
Get started with Azure DevOps learning path
Microsoft Azure: $200 Free Credit
Follow us on Twitter:
April Edwards | @TheAprilEdwards
The DevOps Lab | @TheDevOpsLab
Matty Stratton | @mattstratton
Kat Cosgrove | @Dixie3Flatline
Light
Dark
0 comments