V15.0 - systemd inithook.service fails to initialize appliance when running in a container

[Dude4Linux]

@JedMeister - I mentioned earlier that I was having difficulty getting v15.0 appliances to initialize when running systemd in a container. I believe I've tracked down the cause and am looking for advice on how to fix the problem. After starting a container and waiting for initialization, the systemd journalctl shows the following:

# journalctl -xe
May 01 16:09:48 tkldev-test systemd[1]: inithooks.service: Failed to set invocation ID on control group /system.slice/inithooks.service, ignoring
May 01 16:09:48 tkldev-test systemd[523]: inithooks.service: Failed at step STDIN spawning /bin/sh: No such file or directory
-- Subject: Process /bin/sh could not be executed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The process /bin/sh could not be executed and failed.
-- 
-- The error number returned by this process is 2.
May 01 16:09:48 tkldev-test systemd[1]: Starting inithooks: firstboot and everyboot initialization scripts...
-- Subject: Unit inithooks.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit inithooks.service has begun starting up.
May 01 16:09:48 tkldev-test systemd[1]: inithooks.service: Main process exited, code=exited, status=208/STDIN
May 01 16:09:48 tkldev-test systemd[1]: Failed to start inithooks: firstboot and everyboot initialization scripts.
-- Subject: Unit inithooks.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit inithooks.service has failed.
-- 
-- The result is failed.
May 01 16:09:48 tkldev-test systemd[1]: inithooks.service: Unit entered failed state.
May 01 16:09:48 tkldev-test systemd[1]: inithooks.service: Failed with result 'exit-code'.
May 01 16:17:01 tkldev-test CRON[614]: pam_unix(cron:session): session opened for user root by (uid=0)
May 01 16:17:01 tkldev-test CRON[615]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 01 16:17:01 tkldev-test CRON[614]: pam_unix(cron:session): session closed for user root
May 01 17:17:01 tkldev-test CRON[669]: pam_unix(cron:session): session opened for user root by (uid=0)
May 01 17:17:01 tkldev-test CRON[670]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 01 17:17:01 tkldev-test CRON[669]: pam_unix(cron:session): session closed for user root
May 01 18:17:01 tkldev-test CRON[725]: pam_unix(cron:session): session opened for user root by (uid=0)
May 01 18:17:01 tkldev-test CRON[726]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 01 18:17:01 tkldev-test CRON[725]: pam_unix(cron:session): session closed for user root

At first I couldn't figure out why /bin/sh couldn't be found as implied by the error message. Finally found a post where someone pointed out the the error message is misleading. Taking a look at the inithooks.service file shows:

# cat /lib/systemd/system/inithooks.service 
[Unit]
Description=inithooks: firstboot and everyboot initialization scripts
After=getty@tty8.service
ConditionKernelCommandLine=!noinithooks

[Service]
Type=oneshot
StandardInput=tty-force
TTYPath=/dev/tty8
TTYReset=yes
TTYVHangup=yes
TTYVTDisallocate=yes
EnvironmentFile=/etc/default/inithooks
ExecStart=/bin/sh -c '\
    FGCONSOLE=$(fgconsole); \
    openvt -f -c 8 -s -w -- ${INITHOOKS_PATH}/run; \
    chvt $FGCONSOLE'

[Install]
WantedBy=basic.target

After doing some checking, I realized that the TTYPath i.e. /dev/tty8 is what was missing. In doing the updates for the LXC appliance and images for LXD, I tried to cleanup and remove as much cruft as possible. I believe that confconsole and the /dev/tty's had already been removed in v14.2, but that was not an issue because we never ran systemd in the containers. So my question is what do. I see three options:

1. Restore /dev/tty8 so that inithooks.service can run unmodified.
2. Modify inithooks.service in containers to use /dev/console which was retained.
3. Modify inithooks.service in containers so it runs without user input.
   I think 3 is the most desirable mode for containers as we are already pre-seeding inithooks and redirecting the output to /var/log/inithooks.log. Looking for a second opinion.

[JedMeister]

Thanks @Dude4Linux. I started testing lxc containers yesterday myself and hit a somewhat similar issue. Your log doesn't look quite the same as mine, but certainly similar.

So your LXC containers don't have any ttys at all?! Does SSH still work? I would have expected that to fail without at least one tty (/dev/console is the kernel console AFAIK, but perhaps there is some LXC trickery to work around that?) FWIW, in my tests (on Proxmox), I limited the ttys to 1, but then I could only run one SSH session (any additional SSH sessions would fail).

ttys aside, I'm fairly sure that at least part of the issue is with the ExecStart line in the SystemD service file. If you look at the SysvInit script it checks to see if fgconsole works as expected (it won't in a container) and if it doesn't, it skips the trickery to bind inithooks to the foreground tty. In the SystemD script, it doesn't check that, so when it trys to run the command under SystemD in a container, it'll return a non-zero exit code and fail.

As to your suggested fixes, whilst preseeding should obviously be supported and is probably the most desirable, it should also be possible to allow interactive inithooks. Most Proxmox users will just download, boot and log in (without knowing about the pre-seeding). So unless they are confronted with the inithooks, we'll likely be flooded with support requests asking what the default passwords are...

As I said, I was trying to use a single tty but the side effect was only being able to use one SSH session, so I wasn't super keen on that. I'm not sure how it worked under SysvInit, as that was (at least in theory) limited to a single tty, but appears to handle multiple SSH sessions no problem... I haven't dug that deep yet, but will have a bit more of a play today.

[JedMeister]

@Dude4Linux - I was just looking through your lxd-make-image script to see what you do and compare it against what I'm doing. I noticed that you are still writing an inittab file. My understanding is that that file was only used by SysvInit and that SystemD will totally ignore it?! So would be superfluous for Stretch builds. Although perhaps I'm wrong...

Also, I see that you are tweaking getty-static.service. That's an interesting approach, I was doing things a little differently and was using /etc/systemd/logind.conf to limit ttys (but as i said, that had some unwanted SSH session side effects).

Also, not sure if you got that from Ubuntu or not, but on Debian Stretch, by my understanding dbus isn't required. Dbus is a "recommends", not a "depends"; some additional info here.

I might have a go following your steps and see what behaviour I get.

Also, one other consideration is that I'm still running Proxmox v4.x so am probably using a much older version of LXC than you!

[JedMeister]

Oops, just realised that your getty-static.service tweaks are within the Wheezy/Jessie section of your script, so only applied in those cases...

Where abouts are you tweaking the ttys? Or perhaps as I noted above, my understanding of /etc/inittab is wrong?! Can you perhaps double check for me when you get a chance and see how many (if any) ttys you have in your container. FWIW, in my testing, they show up under /dev/pts/.

[Dude4Linux]

@JedMeister - Originally lxd-make-image was written for version 14.2. I pulled most of the code from buildtasks/bt-container, the lxc-turnkey template, and Stéphane's lxc-to-lxd script. The ttys were removed by Alon in buildtasks/bt-container/patches/container/conf but leaving /dev/console for remote ssh access. I can't recall ever having a problem with multiple ssh connections to a container. I just checked and I have /dev/console and /dev/tty which was never removed. Like you observed, as connections are made they show up under /dev/pts/. I only checked two connections so I don't know if there is a hard limit.

When I started working with stretch, I had to revise the code and adapt it where necessary. I tried to maintain backward compatibility with 14.x whenever possible. Since we're using systemd-sysv, I assumed that /etc/inittab was still required, but if not, it should be moved to the conditional section.

The tweak to getty-static.service comes from the lxc-turnkey template which took it from the lxc-debian template. You'd have to ask Alon to explain why it was needed.

I discovered another issue related to pre-seeding containers. bt-container sets REDIRECT_OUTPUT=true in /etc/default/inithooks. If any of the firstboot.d scripts prompt for user input, the process hangs. I'm not sure if it will ever timeout as I don't have the patience to wait. I got bit be the recent addition of /usr/lib/inithooks/firstboot.d/85secalerts which prompts the user for an email address. I had to add export SEC_ALERTS=SKIP to the pre-seeded /etc/inithooks.conf.

[JedMeister]

Thanks for the background and additional info @Dude4Linux.

I can't recall ever having a problem with multiple ssh connections to a container.

No, I've never had issues either.

I just checked and I have /dev/console and /dev/tty which was never removed. Like you observed, as connections are made they show up under /dev/pts/. I only checked two connections so I don't know if there is a hard limit.

Cool thanks for checking that. 2 is better than I managed when I limited ttys to 1. FWIW /dev/tty should just be something of an alias for whichever tty is active, so in this case, /dev/pts/N

I'm thinking that we don't actually need any of that stuff now. I've done a fair more more testing today and FWIW, it seems that SystemD is completely aware that it's running in a container.

With no modifications made, instead of any of the normal (for other TurnKey v15.0 VMs) getty-static.service, I have container-getty@0.service - inactive (dead); start condition failed & container-getty@1.service - active (running). I'm not sure, but I think it may be getty.target - active both in a container and a VM - that takes care of that...?

I've been playing with the inithooks.service file and with a tweaked one applied via the container patch overlay (i.e. overlay/lib/systemd/system/inithooks.service) I can sort of get it to work (still not quite right, but close). However, that's not ideal as it would be overwritten by a package update.

I've been trying to add tweaks via an overlayed /etc/systemd/system/inithooks.service.d/override.conf which is a preferable measure as that wouldn't be overwritten by package updates (perhaps a doulbe edged sword?!). FWIW, removing the service file and using the one that SystemD generates from the init.d script appears to work, but isn't ideal either.

@OnGle did suggest that worst case scenario we could create a separate inithooks-container package (made from the same source code as inithooks, but with a different service file). That should be fairly straight forward I would imagine?! So could be a good option. I still need to get it working reliably though. Unfortunately I don't know enough about SystemD so have been doing lots of reading today...

I hope you don't mind, I moved conversation of your other inithooks point to a separate issue: #1073

[Dude4Linux]

Note: I hadn't refreshed my page when I started composing this, so I hadn't seen your latest reply above. Thanks for opening #1073. I was just about to open an issue and you saved me the trouble.

@JedMeister - Because of the way bt-container redirects output to /var/log/inithooks.log, I don't think it's possible to have interactive inithooks in a container. Keep in mind that managing containers is very different than managing hosts. Typically you would login to the host machine using ssh, and from there use lxc-console or lxc-attach on LXC v1; or lxc exec on LXD v2 to connect to the containers. Seldom if ever, do you want to login to the container using ssh. This is not new behavior, but follows what Alon setup in the first LXC appliance.

In developing lxd-make-image and the 15.0 updates for LXC, I've tried to follow the best-practices from running Debian on LXD v2. They are even more radical and do things like removing /dev/console, disabling root login, and disabling ssh altogether.

I've tested the following change to inithooks.service and verified that it runs the firstboot.d scripts when the container is started for the first time.

# cat /lib/systemd/system/inithooks.service
[Unit]
Description=inithooks: firstboot and everyboot initialization scripts
After=console-getty.service
ConditionKernelCommandLine=!noinithooks

[Service]
Type=oneshot
EnvironmentFile=/etc/default/inithooks
ExecStart=/bin/sh -c '${INITHOOKS_PATH}/run'

[Install]
WantedBy=basic.target

I had to guess on the After= line, but it seems to work. I propose using this only in containers.

[Dude4Linux]

@JedMeister - FWIW, I verified that there are no inittab's in containers running Debian9 or Ubuntu Xenial. My bad for not noticing. I'll modify lxd-make-image on the next update so that /etc/inittab is created only for Wheezy/Jessie. BTW, Wheezy is included because when I started this project over a year ago, Wheezy was still supported.

[JedMeister]

@Dude4Linux

Apolgies on the length of this...

Because of the way bt-container redirects output to /var/log/inithooks.log, I don't think it's possible to have interactive inithooks in a container.

v14.x interactive inithooks on first login (via SSH or directly) within an LXC container work really well. I'm aiming to recreate that experience with v15.x.

As I noted over on #1073: "[assuming no user preseeding] the inithooks run twice, once self-preseeded (which is what brings up the [init]fence). Then they re-run interactively, and sit there waiting for the user to log in."

Keep in mind that managing containers is very different than managing hosts. Typically you would login to the host machine using ssh, and from there use lxc-console or lxc-attach on LXC v1; or lxc exec on LXD v2 to connect to the containers.

Whilst that is clearly a legitimate use case scenario, and may be the way you usually work, I wouldn't say that's always the case. IMO in a multi-tenant type arrangement, it is much better to only allow user's access to their own LXC container rather than the host machine as well.

Whilst I'm aware that LXC still doesn't (and may never) provide the same level of separation and security that OpenVZ did, SSH still serves that purpose fairly nicely. E.g. we have a user who is using it in an educational setting where each student has their own LXC container(s) - initially launched by the teacher. Giving all individual students access to the host is highly undesirable IMO.

Seldom if ever, do you want to login to the container using ssh.

Personally, my host machine is a purpose built headless server and I almost always access the individual container I wish to work with, directly via SSH. I rarely log into the host machine (other than to create the initial container). Having to first log into the host, then enter the machine requires additional commands which are essentially redundant in my use case. I would hate to see that added as a requirement and I doubt that I'm the only one.

FWIW, even in their marketing hype, Ubuntu note: "[...] Install SSH and log into them remotely, they [LXD containers] behave just like real machines."

This is not new behavior, but follows what Alon setup in the first LXC appliance.

Not that it matters, but circa 2010, with a little help from Adrian, I did the initial PoC container (OpenVZ) implementation. Alon then partnered with Proxmox and implemented the "official" TurnKey (v12.0 IIRC) OpenVZ implementation (based on the work that Adrian and I had done).

For v14.0, Anton and I did the transition from OpenVZ to LXC. Interactive inithooks within a container via first login (SSH or otherwise) have always worked since Alon's initial OpenVZ implementation. I'm hoping it won't require as much work to get that working this time as it did with the v14.0 transition from OpenVZ, but we'll have to see.

Considering that the current TurnKey "container" build is first and foremost (at least at this point) a Proxmox integration, support for interactive inithooks is a requirement. Otherwise, we'll be flooded with support requests on what the default passwords are (IME that's what people expect unless they are presented with the inithooks, or read the docs).

They are even more radical and do things like removing /dev/console, disabling root login, and disabling ssh altogether.

FWIW, if you wish to disable root, then inithooks provides an easy way to do that via turnkey-sudoadmin. It's set to false by default, but can easily be enabled for firstboot (or at buildtime).

I've tested the following change to inithooks.service and verified that it runs the firstboot.d scripts when the container is started for the first time.

Thanks for sharing this John. That's really handy. I probably should have shared exactly what I'd tried yesterday, but as I hadn't got to the point of reliability (and was trying out a few different things and doing lots of reading) I wasn't sure of the value. I will endeavour to post back how I go later today.

I propose using this only in containers.

As I noted yesterday, that is one way to go, but to do it properly, would require us to create a new inithooks package that we install in containers (otherwise /lib/systemd/system/inithooks.service would be overwritten by future inithooks package updates). I have no fundamental aversion to going that path and it may be the best path. Perhaps it could be called inithooks-container, inithooks-lxc or similar.

One of the other alternatives that I've been looking at, is to use a local systemd override.conf file that is included as part of the overlay. As I say though, that is the double edged sword. On one hand, we wouldn't need to provide an additional/alternative inithooks package. On the other, if it's only included as an overlay, we'd likely need to rebuild the LXC containers if we discover a bug at some point in the future...

BTW, Wheezy is included because when I started this project over a year ago, Wheezy was still supported.

FWIW Debian LTS support for Wheezy ends end of this month: https://wiki.debian.org/LTS

[Dude4Linux]

@JedMeister - You've made some excellent points. I guess I've been viewing LXC and LXD through the lens of how I intended to use it as a testing environment when paired with Ansible. I agree that for the multi-user environment that you describe, having ssh access control on a per-container basis is a must. That can be setup using either LXC or LXD using pre-seeding. It would also be easy to use Ansible to create a complete set of containers with individual user names and passwords.

After thinking about it for awhile and doing some checking, it seems to me that we need inithooks.service aware of when it running in a container. Currently, both LXC v1 and LXD v2 set an environment variable in each container, i.e. container=lxc. This should also work for Proxmox, but you probably have to test for Docker as well. After a little googling I came up with the following:

#!/bin/bash
if grep -qa 'docker' /proc/1/cgroup; then
  echo "I'm running on docker."
elif grep -qa 'container=lxc' /proc/1/environ; then
  echo "I'm running in container."
fi 

Of course you will have to adjust the logic accordingly.

[JedMeister]

@Dude4Linux - As you would have noticed, I didn't post back yesterday... Mostly because I didn't really have any real progress... I tried your systemd service file, and whilst it works beautifully when pre-seeding, it doesn't work so well when not pre-seeding. It only runs the first time and brings the fence up, then exits. Trying to restart it, does nothing. The only way to work around is manually invoke turnkey-init which is not ideal.

So I'll need to do some more reading and perhaps even reach out to someone who knows more than me for assistance (there is so much to read about SystemD!). One thing I am going to try shortly, is to see what happens if I just remove the default service file altogether (and let SystemD generate it's own one from the old init.d script).

One thing that I did turn up in my digging through SystemD docs though, was that SystemD can already tell what environment it's running in! The command systemd-detect-virt returns a string matching the virtualisation/containterisation. It's pretty cool!

That functionality can also be leveraged in SystemD service scripts, either by specific virtualized environment, or simply container or vm. E.g. via adding a ConditionVirtualization=lxc to the [Service] section of the service file will only run the service when running as an LXC container.

[JedMeister]

@Dude4Linux - Ok had a fair bit of progress today! 😄

I've pretty much got it working as intended. As I had other inithook bugs to troubleshoot which I thought may have also been to do with the service file. So I did (temporarily) revert to using the autogenerated SystemD service file (i.e. removing the default one supplied with the package and allowing systemD to generate one from /etc/init.d/inithooks). But after I'd worked through those other issues, I tested with your file and it worked great, so thanks for that.

I'm not 100% sure of the best way to proceed with the SystemD service file. I feel like we could do better than overlaying a file handled by package management. Ideally I think it would be good to leverage ConditionVirtualization=lxc somehow to make it so it works no matter where it's running. So I might see if I can get the attention of someone a little more versed in SystemD. OTOH I guess it's pretty unlikely that a user would update the inithooks pacakge (and thus restore the default file) before a container is initialised!

I haven't merged with TurnKey repo yet as I need to double check against the latest revision of your scripts, to make sure I'm not missing anything. But FWIW, here's a couple of things I've done:

• left resolveconf installed (it works fine without it, so not sure whether it's better to remove or leave?)
• don't bother installing quota (doesn't appear to be required anymore?!)

You can see my latest patches/container/confhere. I'll probably rework the commits a little before I merge too, but the file itself should be good (works fine for me on Proxmox v4.x).