.NET Core and systemd
In preview7 a new package was added to the `Microsoft.Extensions` set of packages that enables integration with systemd. For the Windows focused, systemd allows similar functionality to Windows Services, there is a post on how to do what we discuss here for Windows Services in this post. This work was contributed by Tom Deseyn from Red Hat. In this post we will create a .NET Core app that runs as a systemd service. The integration makes systemd aware when the application has started/is stopping, and configures logs to be sent in a way that journald (the logging system of systemd) understands log priorities.
Create and publish an app
First let’s create the app that we will use. I’m going to use the new worker template, but this would also work well with an ASP.NET Core app. The main restriction is that it needs to be using a Microsoft.Extensions.Hosting based app model.
When using the command line you can run:
dotnet new worker
This command will create you a new worker app the same as the VS UI. Once we have our app we need to add the Microsoft.Extensions.Hosting.Systemd NuGet package, you can do this by editing your csproj, using the CLI, or using VS UI:
Once you’ve added the NuGet package you can add a call to UseSystemd in your program.cs:
At this point you have configured your application to run with systemd. The UseSystemd method will noop when not running as a daemon so you can still run and debug your app normally or use it in production both with and without systemd.
Create unit files
Now that we have an app we need to create the configuration files for systemd that tell it about the service so that it knows how to run it. To do that you create a .service file (there are other types of unit file, the .service file is what we will use since we are deploying a service). You need this file on the Linux machine that you will be registering and running the app on. A basic service file looks like this:
This file needs to exist in the /etc/systemd/system/ directory, /etc/systemd/system/testapp.service in our case. By specifying Type=notify an application can notify systemd when the host has started/is stopping. Once the file exists in the directory run the following for systemd to load the new configuration file using the systemctl command which is how you interact with systemd:
sudo systemctl daemon-reload
After that if you can run the following to see that systemd knows about your service:
sudo systemctl status testapp(replacing testapp with the name of your app if you used a different name)
You should see something like the following:
This shows that the new service you’ve registered is disabled, we can start our service by running:
sudo systemctl start testapp.service
Because we specified Type=notify systemd is aware when the host has started, and the systemctl start will block until then. If you re-run sudo systemctl status testapp you will see something like the following:
If you want your service to start when the machine does then you can use:
sudo systemctl enable testapp.service
You will see that the status message now changes to say enabled instead of disabled when running systemctl status.
If you are having trouble getting your app to start for some reason then you should make sure that you can run the file in the ExecPath yourself in the terminal first, then use systemctl status to see what messages you are getting from the app when it fails to start.
Now that we have an app running with systemd we can look at the logging integration. One of the benefits of using systemd is the centralized logging system that you can access with journalctl.
To start, we can view the logs of our service by using journalctl, a command to access the logs:
sudo journalctl -u testapp
This displays all the logs for the unit (-u) file with testapp in the name. You could be more specific by using testapp.service. If you run journalctl without specifying the service you are interested in then you will see logs from all services interleaved with each other as all logs are seen as one big log stream in this system. You use journalctl to focus that single log stream to what you are interested in at the time.
Running the command would give you output that looks like:
You can see the logging is different than when running from the terminal: each message is on a single line. systemd is also aware of the log priorities. To show this in action I added a few log statements to my testapp and run again:
Then if I run sudo journaltcl -u testapp I see:
In this log output the tool has highlighted the critical log message in red and shows that the lifetime of my app is now the SystemdLifetime proving to me that the integration has worked. The tool can do this because when calling UseSystemd we map Extensions.LogLevel to syslog log levels:
|LogLevel||Syslog level||systemd name|
With this information I can run sudo journalctl -p 3 -u testapp which will filter log messages to only display critical and error logs.
If you’re using .NET Core apps with systemd then we think this package should give you a much better experience, and we hope you try it out and tells us any other features you’d like to see by logging issues on GitHub. If you are going to use this to run a web app then there is some additional guidance on the ASP.NET Docs page.
Hi, I’ve used .net core 3.1 to create a grpc service with the systemd and windowsServices nuget packages, on windows, it was able to run as a service using the sc create/start/stop, so i was trying to achieve the same thing on linux. i’ve transferred the files over to linux. and in the terminal, when I typed “dotnet .dll in the folder itself, it was able to start. log files indicating it was runnning fine.
Now, looking at this article, I am confused.. there is nothing specifying the dll to load when the darmon is supposed to start (similar to the binpath parameter in the sc create command). So how does it know which dll to run, given that there are so many dlls in the folder? and if I put the folder in my home directory, how to I specify that? surely I can’t say ExecStart= ~/ ?
please advice.. as you can see, I am not familiar with linux, so please provide as much help as possible.. thank you very much for your time and patience..
setting ExecStart=/usr/sbin/DIRECTORY NAME/app.dll
still fails.. with error: permission denied..
why? what else is needed? please advice.. this thing is really GETTING ON MY NERVES..
I did this on Ununtu 18.04 in Hyper-V enhaned mode in my PC.. linux is ANNOYINGLY PAINFUL!
the service file DOES NOT work at all! I’ve put the path as /home/john/test/test.dll (my own test worker app), and set the permission of the test.dll to 777 (chmod 777 test.dll), and it’s giving format error! status:203.
kindly advice how to get it to work..
my codes are compiled in Windows.. running it directly inside linux using dotnet test.dll, it can run alright..
Ubuntu 18.04 in HJyperV enhanced mode in windows 10..
what else do I need to do???
Hi John, I think the answer is to select a Linux runtime when building the application. Checkout the –runtime option for dotnet build.
Great post but I’m scratching my head like John Lonewolf. What ExecStart should point to is not clear to me either. I’ve tried the same options as John with similar lack of success. I can run my app using the dotnet command directly so I know it will run. Any help would be appreciated!
OK… I am an idiot. The answer is pretty obviously to use the –runtime option when publishing the application and selecting the appropriate Linux runtime for the target OS. That results in the extensionless Linux executable that you reference from ExecStart in your .service file.
so for instance, my case, that would compile to test instead of test.dll, but what about the rest of the dlls that it depends on? Also, wjat would be the ExecStart? still /home/john/test/test?
did you try compiling the source codes directly in linux and see if it will auto generate the extensionless file?
So what I did was to publish from Visual Studio with the Target Runtime set to linux-arm (there are a number of options available here – check the Microsoft Runtime Catalogue at https://docs.microsoft.com/en-us/dotnet/core/rid-catalog). I also chose to use a Deployment Mode of Self-contained so I didn’t need to install the dotnet core runtime on the host. After publishing with such settings you’ll see an extensionless binary with the same name as the DLL in the published directory. I copied all the published files to a folder on the Linux host and configured ExecStart to be the absolute path to the extensionless file (it must be an absolute path or you’ll get an error). TBH I’m not sure if all the files are required. Setup systemd as described in the body of the post above and you should be golden. You may need to give the files executable permissions. I haven’t tried building directly on a Linux host other than using the dotnet Docker images, sorry.
well, after several days fumbling around, cursing at the penguins.. I’ve managed to get my app to run in linux as daemon. Here’s the missing parts:
1. binaries need to be compiled with the RID of the target platform, that’s the way that does not require installation of the .net core runtime on the target machine. I compiled my project using this command: dotnet publish [proj].csproj -c release -r linux-x64 -o d:\deploy[proj]
whole set of binaries less than 100MB, no runtime folder in it.
use WinSCP to transfer binary to linux, in linux terminal, set the executable flag for the binary (that’s the binary with no extension): chmod +x [binary file]
if you don’t set this, after file is transferred over, you’ll get permission denied error when the systemd tries to run it.
The service file, as described above needs to include the following if you need network to be up to operate properly:
put these 2 lines in the [Unit] section of the service file
rest of the steps are fine.. I’ve written a grpc service and ran as a daemon in ubuntu 19 and it works.. hope this can help others not familiar with linux. 🙂
This article needs to fill some gaps to reduce a lot of time-wasting and headaches. Many of the commentors above me have identified those gaps, but there’s no update to the article for the comprehensive fixes.
You must build/publish your app to a specific linux runtime (such as “linux-x64” that I used for Ubuntu 18.04). For example: “dotnet publish -r linux-x64 -c Release”. The output will have a lot of extra folders. Make sure you get the linux-x64 folder output. The extension less version of the app is what you’ll target with systemd (don’t target the DLL in systemd .service file).
The systemd .service file seems to also need a [WorkingDirectory]= line in the [Service] section or the appsettings.json file will fail to be loaded. Without that line, all references to the appsettings configurations would be null.
You need to make sure systemd has permission to execute the app files. Use chmod to set the proper file permissions in your scenario.
@Justin, I added this line “System.IO.Directory.SetCurrentDirectory(System.AppDomain.CurrentDomain.BaseDirectory);” right in the first line of Main() in Proigram.cs and it works.
One thing I can say about this article is, it’s coming from the perspective of compiling .net core codes in linux. I’m now using Linux Mint and compiling my .net core codes in linux doesn’t require the chmod +x step, neither do I need to specify the -r linux-x64. the output is linux binary. So all the issues we faced is a result of compiling .net core codes in windows, expecting it to work in linux.
Maybe this article can set up a section of the extra steps needed when compiling .net core codes in windows but deploying in Linux.
For those not familiar with Linux, strongly recommend Linux mint if you’re thinking of trying out linux. Main reason is, the back mouse button can work well. Even Ubuntu can’t support back mouse button, let alone forward mouse button.. this is particularly useful when jumping between codes in VS code: CTRL + click to go to definition, then back mouse button to go back to oiriginal position. 🙂