.NET Core and systemd

Avatar

Glenn

In preview7 a new package was added to the `Microsoft.Extensions` set of packages that enables integration with systemd. For the Windows focused, systemd allows similar functionality to Windows Services, there is a post on how to do what we discuss here for Windows Services in this post. This work was contributed by Tom Deseyn from Red Hat. In this post we will create a .NET Core app that runs as a systemd service. The integration makes systemd aware when the application has started/is stopping, and configures logs to be sent in a way that journald (the logging system of systemd) understands log priorities.

Create and publish an app

First let’s create the app that we will use. I’m going to use the new worker template, but this would also work well with an ASP.NET Core app. The main restriction is that it needs to be using a Microsoft.Extensions.Hosting based app model.

In VS:

Visual Studio new project dialog

Command Line:

When using the command line you can run:

dotnet new worker

This command will create you a new worker app the same as the VS UI. Once we have our app we need to add the Microsoft.Extensions.Hosting.Systemd NuGet package, you can do this by editing your csproj, using the CLI, or using VS UI:

Once you’ve added the NuGet package you can add a call to UseSystemd in your program.cs:

At this point you have configured your application to run with systemd. The UseSystemd method will noop when not running as a daemon so you can still run and debug your app normally or use it in production both with and without systemd.

Create unit files

Now that we have an app we need to create the configuration files for systemd that tell it about the service so that it knows how to run it. To do that you create a .service file (there are other types of unit file, the .service file is what we will use since we are deploying a service). You need this file on the Linux machine that you will be registering and running the app on. A basic service file looks like this:

This file needs to exist in the /etc/systemd/system/ directory, /etc/systemd/system/testapp.service in our case. By specifying Type=notify an application can notify systemd when the host has started/is stopping. Once the file exists in the directory run the following for systemd to load the new configuration file using the systemctl command which is how you interact with systemd:

sudo systemctl daemon-reload

After that if you can run the following to see that systemd knows about your service:

sudo systemctl status testapp
(replacing testapp with the name of your app if you used a different name)

You should see something like the following:

null

This shows that the new service you’ve registered is disabled, we can start our service by running:

sudo systemctl start testapp.service

Because we specified Type=notify systemd is aware when the host has started, and the systemctl start will block until then. If you re-run sudo systemctl status testapp you will see something like the following:

If you want your service to start when the machine does then you can use:

sudo systemctl enable testapp.service

You will see that the status message now changes to say enabled instead of disabled when running systemctl status.

If you are having trouble getting your app to start for some reason then you should make sure that you can run the file in the ExecPath yourself in the terminal first, then use systemctl status to see what messages you are getting from the app when it fails to start.

Exploring journalctl

Now that we have an app running with systemd we can look at the logging integration. One of the benefits of using systemd is the centralized logging system that you can access with journalctl.

To start, we can view the logs of our service by using journalctl, a command to access the logs:

sudo journalctl -u testapp

This displays all the logs for the unit (-u) file with testapp in the name. You could be more specific by using testapp.service. If you run journalctl without specifying the service you are interested in then you will see logs from all services interleaved with each other as all logs are seen as one big log stream in this system. You use journalctl to focus that single log stream to what you are interested in at the time.

Running the command would give you output that looks like:

You can see the logging is different than when running from the terminal: each message is on a single line. systemd is also aware of the log priorities. To show this in action I added a few log statements to my testapp and run again:

Then if I run sudo journaltcl -u testapp I see:

Log messages in a console app with red critical logs

In this log output the tool has highlighted the critical log message in red and shows that the lifetime of my app is now the SystemdLifetime proving to me that the integration has worked. The tool can do this because when calling UseSystemd we map Extensions.LogLevel to syslog log levels:

LogLevelSyslog levelsystemd name
Trace/Debug7debug
Information6info
Warning4warning
Error3err
Critical2crit

With this information I can run sudo journalctl -p 3 -u testapp which will filter log messages to only display critical and error logs.

Conclusion

If you’re using .NET Core apps with systemd then we think this package should give you a much better experience, and we hope you try it out and tells us any other features you’d like to see by logging issues on GitHub. If you are going to use this to run a web app then there is some additional guidance on the ASP.NET Docs page.

28 comments

Leave a comment

  • David Strauss
    David Strauss

    Awesome post! Like the other commenter, I love seeing these sorts of cross-platform coordinations. However, under “Create unit files”, the sample file doesn’t specify “Type=notify”, but the text later states “Because we specified Type=notify”. Shouldn’t the sample unit file include the Type= specification? Without specifying, the default is Type=simple given the rest of the sample unit file. I’m also curious how the Type=notify integration is happening here. Does Microsoft.Extensions.Hosting.Systemd look for $NOTIFY_SOCKET and interact with the socket accordingly? Or is it the developer’s responsibility to add that?

  • Avatar
    Malachi Burke

    Great stuff!  We use systemd heavily already without the benefit of this integration, so this is pretty exciting for us.  I’d like to note that running ASP.NET Core “regular style” from a systemd unit (without this enhanced integration) does work well, and includes the journalctl features described in this article, possibly excluding the log level awareness (never tested that).

    • Avatar
      Glenn Condron [MSFT]

      In general I don’t think running systemd inside a container is something you want to do. You can kind of make it work, but it usually isn’t required if your building an app and running it. Instead I would be looking to run my app as the process of the container and have the orchestration components around that take care of keeping it up and sending logs to places.

    • Avatar
      cheong00

      I think you only need to auto-start the docker daemon, then enable your docker instance so it’ll be loaded the next time the system is rebooted.
      docker run –restart unless-stopped –name <your instance name>
      https://docs.docker.com/config/containers/start-containers-automatically/
      If you want to do the changes on file, you can edit /var/lib/docker/containers/<your container id>/hostconfig.json and set “RestartPolicy” to {“Name”:”unless-stopped”,”MaximumRetryCount”:0}
      Where you can get the first part of <your container id> with “docker ps -a”

  • Михаил Снытко
    Михаил Снытко

    This is the best guide to Worker service on linux that I could find. I had to add some nuget packages to make it working on debian 10 and .net core 3.1:

    Microsoft.Extensions.FileSystemGlobbing” Version=”3.1.1″ />
    Microsoft.Extensions.Hosting” Version=”3.1.0″ />
    Microsoft.Extensions.Hosting.Abstractions” Version=”3.1.1″ />
    Microsoft.Extensions.Hosting.Systemd” Version=”3.1.0″ />
    Microsoft.Extensions.Options.ConfigurationExtensions” Version=”3.1.1″ />

    • John Lonewolf
      John Lonewolf

      could you be kind enough to share with me how you specify your service file to get the daemon to run? I’ve tried and failed.. permission denied.. I don’t know what else is needed.. I’m not familiar with linux.. so please advice me.. thank you for your time..

      From the article, ExecStart sis not include specifying the binary file.. how did you do yours? I soecified the binary file and still, permission denied.. :'(

  • John Lonewolf
    John Lonewolf

    Hi, I’ve used .net core 3.1 to create a grpc service with the systemd and windowsServices nuget packages, on windows, it was able to run as a service using the sc create/start/stop, so i was trying to achieve the same thing on linux. i’ve transferred the files over to linux. and in the terminal, when I typed “dotnet .dll in the folder itself, it was able to start. log files indicating it was runnning fine.

    Now, looking at this article, I am confused.. there is nothing specifying the dll to load when the darmon is supposed to start (similar to the binpath parameter in the sc create command). So how does it know which dll to run, given that there are so many dlls in the folder? and if I put the folder in my home directory, how to I specify that? surely I can’t say ExecStart= ~/ ?

    please advice.. as you can see, I am not familiar with linux, so please provide as much help as possible.. thank you very much for your time and patience..

    • John Lonewolf
      John Lonewolf

      setting ExecStart=/usr/sbin/DIRECTORY NAME/app.dll

      still fails.. with error: permission denied..
      why? what else is needed? please advice.. this thing is really GETTING ON MY NERVES..
      I did this on Ununtu 18.04 in Hyper-V enhaned mode in my PC.. linux is ANNOYINGLY PAINFUL!

  • John Lonewolf
    John Lonewolf

    the service file DOES NOT work at all! I’ve put the path as /home/john/test/test.dll (my own test worker app), and set the permission of the test.dll to 777 (chmod 777 test.dll), and it’s giving format error! status:203.

    kindly advice how to get it to work..
    my codes are compiled in Windows.. running it directly inside linux using dotnet test.dll, it can run alright..
    Ubuntu 18.04 in HJyperV enhanced mode in windows 10..

    what else do I need to do???

  • Andy
    Andy

    Great post but I’m scratching my head like John Lonewolf. What ExecStart should point to is not clear to me either. I’ve tried the same options as John with similar lack of success. I can run my app using the dotnet command directly so I know it will run. Any help would be appreciated!

    • Andy
      Andy

      OK… I am an idiot. The answer is pretty obviously to use the –runtime option when publishing the application and selecting the appropriate Linux runtime for the target OS. That results in the extensionless Linux executable that you reference from ExecStart in your .service file.

      • John Lonewolf
        John Lonewolf

        so for instance, my case, that would compile to test instead of test.dll, but what about the rest of the dlls that it depends on? Also, wjat would be the ExecStart? still /home/john/test/test?
        did you try compiling the source codes directly in linux and see if it will auto generate the extensionless file?

        • Andy
          Andy

          So what I did was to publish from Visual Studio with the Target Runtime set to linux-arm (there are a number of options available here – check the Microsoft Runtime Catalogue at https://docs.microsoft.com/en-us/dotnet/core/rid-catalog). I also chose to use a Deployment Mode of Self-contained so I didn’t need to install the dotnet core runtime on the host. After publishing with such settings you’ll see an extensionless binary with the same name as the DLL in the published directory. I copied all the published files to a folder on the Linux host and configured ExecStart to be the absolute path to the extensionless file (it must be an absolute path or you’ll get an error). TBH I’m not sure if all the files are required. Setup systemd as described in the body of the post above and you should be golden. You may need to give the files executable permissions. I haven’t tried building directly on a Linux host other than using the dotnet Docker images, sorry.

  • John Lonewolf
    John Lonewolf

    well, after several days fumbling around, cursing at the penguins.. I’ve managed to get my app to run in linux as daemon. Here’s the missing parts:
    1. binaries need to be compiled with the RID of the target platform, that’s the way that does not require installation of the .net core runtime on the target machine. I compiled my project using this command: dotnet publish [proj].csproj -c release -r linux-x64 -o d:\deploy[proj]
    whole set of binaries less than 100MB, no runtime folder in it.

    use WinSCP to transfer binary to linux, in linux terminal, set the executable flag for the binary (that’s the binary with no extension): chmod +x [binary file]
    if you don’t set this, after file is transferred over, you’ll get permission denied error when the systemd tries to run it.
    The service file, as described above needs to include the following if you need network to be up to operate properly:
    After=network-online.target
    Wants=network-online.target

    put these 2 lines in the [Unit] section of the service file

    rest of the steps are fine.. I’ve written a grpc service and ran as a daemon in ubuntu 19 and it works.. hope this can help others not familiar with linux. 🙂

  • Avatar
    Justin Skiles

    This article needs to fill some gaps to reduce a lot of time-wasting and headaches. Many of the commentors above me have identified those gaps, but there’s no update to the article for the comprehensive fixes.

    You must build/publish your app to a specific linux runtime (such as “linux-x64” that I used for Ubuntu 18.04). For example: “dotnet publish -r linux-x64 -c Release”. The output will have a lot of extra folders. Make sure you get the linux-x64 folder output. The extension less version of the app is what you’ll target with systemd (don’t target the DLL in systemd .service file).

    The systemd .service file seems to also need a [WorkingDirectory]= line in the [Service] section or the appsettings.json file will fail to be loaded. Without that line, all references to the appsettings configurations would be null.

    You need to make sure systemd has permission to execute the app files. Use chmod to set the proper file permissions in your scenario.