77
WWW.LINUXJOURNAL.COM / MARCH 2015 /
73
WWW.LINUXJOURNAL.COM / MARCH 2015 /
73
Q
There were no facilities to
discover and scan for new storage
devices without locking the
system, especially when a disk
might not even power on until it
was scanned.
Q
There were no facilities to load
firmware for a device, which may
have needed to occur after it was
detected but before it was usable.
Inevitably, around the 2005/2006
time frame, several alternative
efforts tried to fix all the issues with
THE3YS6SCHEME"UTTHE EFFORT THAT
looked most promising during that
time was the Upstart init project
sponsored by Canonical.
Upstart
To be sure, Upstart init doesn’t share
ANYCODEWITHTHE3YS6INITSCHEME
but it’s rather a superset of it,
providing a good degree of backward-
compatibility. The main departure
FROMTHETRADITIONAL3YS6WAYOFDOING
things is that Upstart implements an
event-driven model that allows it to
respond to milestones asynchronously
as they are reached. Upstart also
implements the concept of jobs,
which are described by the files under
ETCINIT
CONF ANDWHOSEPURPOSE
is to execute a script section that
spawns a process. As such, system
initialization can be expressed as a
consecutive set of “spawn process X
when event Y occurs” rules.
*UST LIKEIN THE 3YS6 SCHEME
the Linux kernel gives control
to Upstart when it executes the
Upstart implementation of /sbin/init.
At this point, things may work a
little differently depending on your
distribution of Linux. For Red Hat
%NTERPRISE,INUX 2(%, USERS
you’ll still find a file at /etc/inittab,
but the sole function of this file is
to set the default runlevel for the
system. If your distribution is one of
the Ubuntu derivatives, /etc/inittab
doesn’t even exist anymore, and the
default runlevel is set in a file called
/etc/init/rc-sysinit.conf instead.
The Upstart version of /sbin/init will
emit a single event called startup,
which triggers the rest of the system
initialization. There are a few jobs
that specify the startup event as their
start condition, the most notable of
which is
mountall
, which mounts all
filesystems. The
mountall
job then
triggers various other events related
to disk and filesystem initialization.
These events, in turn, trigger the
udev kernel device manager to start,
and it emits the event that starts the
networking subsystem.
This is when one of the most critical
87
74
/ MARCH 2015 / WWW.LINUXJOURNAL.COM
FEATURE Initializing and Managing Services in Linux: Past, Present and Future
jobs is triggered by Upstart. This job is
called
rc-sysinit
, which has a start
dependency on the filesystem and
network-up events. The role of this
job is to bring the system to its default
runlevel. It executes the command
telinit <runlevel>
to achieve this.
The
telinit
command then emits
the runlevel event, which causes many
other jobs to start. This includes the
/etc/init/rc.conf job, which implements
ACOMPATIBILITYLAYERFORTHE3YS6INIT
scheme. It executes
/etc/init.d/rc
<runlevel>
and determines if a
/etc/rc#.d/ directory exists for the current
runlevel, executing all scripts in it.
In Upstart-based systems, such
AS5BUNTUAND2(%, YOUCAN
use the tools
sysv-rc-conf
or
chkconfig
, respectively, to manage
the runlevel of different services.
You also can manage jobs via the
initctl
utility. You can list all jobs
and their respective start and stop
events with the command
initctl
show-config
. You also can check
on job status, list available jobs and
start/stop jobs with the following
commands, respectively:
Q
$ initctl status <job>
Q
$ initctl list
Q
# initctl start|stop <job>
The Upstart scheme has been used
in popular distributions of Linux, such
AS&EDORAFROMVERSIONSUPTO
THE2(%,SERIESAND5BUNTUSINCE
version 6.10 to present. But for all
the flexibility that Upstart init brings
to Linux, it still falls short in a few
fundamental ways:
Q
It ignores the system state between
events. For instance, a system has
a power cord plugged in, then the
system runs on AC power for a
while, and then the user unplugs the
power cord. Upstart focuses on each
event above as a single discrete and
unrelated unit, instead of tracking
the chain of events as a whole.
Q
The event-driven nature of the
system turns the dependency chain
on its head. Instead of doing the
absolute minimum amount of
work needed to get the system to
a working state, when an event
is triggered, it executes all jobs
that could possibly follow it. For
example, just because networking
has started, it doesn’t mean that
NFS also should start. As a matter
of fact, the opposite is the correct
order of things: when a user
REQUESTSACCESSTOAN.&3SHARE
the system should validate that
networking is also up and running.
59
WWW.LINUXJOURNAL.COM / MARCH 2015 /
75
Q
The dependency chain is still
present. Although many more things
happen in parallel in Upstart, the
user has to port the original script
SEQUENCEFROM3YS6INITTOASET
of event trigger action rules in the
CONFFILESINETCINIT&URTHERMORE
because of the spanning tree
structure of the event system, it is
a real nightmare to figure out why
something happened and what
event triggered it.
There is another init scheme
whose purpose is to address the
issues listed above.
systemd
systemd is the latest milestone on the
road to init system nirvana. The main
design goals of this init scheme are,
according to Lennart Poettering, lead
developer of systemd, “to start less,
and to start more in parallel”. What
that means is that you execute only
that which is absolutely necessary to
get the system to a running state, and
you run as much as possible at the
same time.
To accomplish these goals, systemd
aims to act against two major trouble
spots of previous init schemes: the
shell and parallelism. The main
executable for systemd, /lib/systemd/
systemd, performs all calls that
originally were present in scripts, thus
eliminating the need to spawn a shell
environment. What about the call to
/sbin/init that’s hard-coded in the Linux
kernel? It’s still there in the form of a
symbolic link to /lib/systemd/systemd.
To address parallelism, you need to
remove the dependency chain between
the various services or at least make it
a secondary concern. If you look at the
problem at its most fundamental level,
the dependency between the various
services boils down to one thing: having
a socket available for the processes
to communicate among themselves.
systemd creates all sockets first and
then spawns all processes in parallel.
For example, services that need to write
to the system log need to wait for the
The main design goals of this init scheme
are, according to Lennart Poettering,
lead developer of systemd, “to start less,
and to start more in parallel”.
68
76
/ MARCH 2015 / WWW.LINUXJOURNAL.COM
FEATURE Initializing and Managing Services in Linux: Past, Present and Future
/dev/log socket to become available,
but as soon as it is available, these
services can start. Therefore, if systemd
creates the socket /dev/log first, then
that’s one less dependency that blocks
OTHERSERVICES%VENIFTHEREISNOTHING
to receive messages at the other end of
the socket, this strategy still works. The
kernel itself will manage a buffer for
the socket, and as soon as the receiving
service starts, it will flush the buffer
and handle all the messages. The ideas
above are not new or revolutionary.
They have been tried before in projects
like the xinetd superserver and the
launchd init scheme used in OS X.
systemd does introduce the new
concepts of units and targets. A target
is analogous to a runlevel in previous
schemes and is composed of several
units. systemd will execute units to
reach a target. The instructions for each
unit reside in the /lib/systemd/system/
directory. These files use a declarative
format that looks like a Windows INI
file. The most common type of these
units is the service unit, which is used
to start a service. The sshd.service file
from Arch Linux looks like this:
[Unit]
Description=OpenSSH Daemon
Wants=sshdgenkeys.service
After=sshdgenkeys.service
After=network.target
[Service]
ExecStart=/usr/bin/sshd -D
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always
[Install]
WantedBy=multi-user.target
This format is really simple and
really portable across several different
distributions. There are other types
of unit files that describe a system,
and they are socket, device, mount,
automount, swap, target, path, timer,
snapshot, slice and scope. Going into
all of them in detail is beyond the
scope of this article; however, I want to
mention one thing: target is a special
type of unit file that glues the other
types together into a cohesive whole.
For example, here are the contents of
basic.target from Arch Linux:
[Unit]
Description=Basic System
Documentation=man:systemd.special(7)
Requires=sysinit.target
Wants=sockets.target timers.target paths.target
´slices.target
After=sysinit.target sockets.target timers.target
´paths.target slices.target
JobTimeoutSec=15min
JobTimeoutAction=poweroff-force
94
WWW.LINUXJOURNAL.COM / MARCH 2015 /
77
You can follow the chain of
dependencies if you look at what
BASICTARGET REQUIRES AND WANTS
Those are actual unit files in the
same /lib/systemd/system/ directory.
The
Requires
and
Wants
directives
above are how systemd defines
the dependency chain among the
units. The
Requires
directive
DENOTES AHARD REQUIREMENT
and
Wants
denotes an optional
REQUIREMENT !LSO KEEPIN MIND THAT
Requires
and
Wants
don’t imply
order. If the
After
directive isn’t
specified, systemd will start the
units in parallel.
Timer units are also really
interesting. They are unit files
that contain a
[Timer]
section
and define how the
TimeDateD
subsystem of systemd will activate
a future event. In these timer units,
you can create two types of timers:
one that will activate after a time
period based on a variable starting
point, such as the systems boot,
and another that activates at fixed
intervals like a cron job. As a matter
of fact, timer units are an alternative
to cron jobs.
One last thing to mention about
systemd unit files is that they
provide the means to describe easily
what to do when a service crashes.
You can do that by using the
directives
Restart
or
RestartSec
in your unit files. This feature allows
systemd to take the role of process
supervisor as well.
systemd refers to the init
dæmon executable itself, namely
/lib/systemd/systemd, but it also
refers to the set of utilities and
programs used to manage the system
and services. Chief among these
utilities is the
systemctl
program
that’s used to manage services.
You can use it to enable, start and
disable services, find the status of a
given service and also list all loaded
units. For example:
Q
# systemctl enable sshd
Q
# systemctl start sshd
Q
# systemctl stop sshd
Q
# systemctl status sshd
Q
# systemctl list-units
Some Linux distributions, like
2(%, AND #ENT/3 PROVIDE A
compatibility layer that translates
3YS6 AND 5PSTARTCOMMANDSINTO
systemd commands. If you issue
the command
service sshd
status
in CentOS 7, you will get
the following output:
77
78
/ MARCH 2015 / WWW.LINUXJOURNAL.COM
Redirecting to /bin/systemctl status sshd.service
sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled)
Active: active (running) since Mon 2014-12-08 02:01:53 PST;
´12h ago
Process: 915 ExecStartPre=/usr/sbin/sshd-keygen (code=exited,
´status=0/SUCCESS)
Main PID: 937 (sshd)
CGroup: /system.slice/sshd.service
...937 /usr/sbin/sshd -D
Notice that first line of console
output above and how it indicates
THATTHE3YS6
STYLECOMMANDWAS
redirected to the systemd-style of
command. This allows the user to
ease into the systemd way of doing
things while still allowing the user to
leverage the previous skill set.
Another really important program
in the systemd toolbox is the
journalctl
utility. It allows you
to view and manage the systemd
logging subsystem called journald.
systemd’s logfile is a binary file and
using
journalctl
really simplifies
the user experience. Here are some
interesting examples:
Q
Display full log:
# journalctl --all
Q
Tail the log:
# journalctl -f
Q
Filter log by executable:
# journalctl /lib/systemd/systemd
Q
Display log since last boot:
# journalctl -b
Q
Display errors from last boot:
# journalctl -b -p err
I urge you to look at the
documentation of the different schemes
presented here to learn more.
Controversies
From my vantage point, the future is
not 100% certain when it comes to init
schemes for Linux. The clear leader, as
)WRITETHISINLATE
ISSYSTEMD!
lot of distributions are adopting it; the
LATESTONESARE2(%,AND$EBIAN
However, the adoption of systemd
has been controversial, and these
distributions have received a lot of
strong feedback from their respective
communities. Of note is the Debian
technical committee debate that
occurred in the Debian mailing list and
a complaint by Linus Torvalds himself
in the Linux kernel mailing list.
systemd is not just an init scheme.
It unifies everything that is related to
starting and managing system services
into a centralized and monolithic
whole: user login, cron jobs, network
services, virtual TTY management
and so on. The use of shell scripts to
control system startup has the benefit
of providing flexibility, and a lot of
FEATURE Initializing and Managing Services in Linux: Past, Present and Future
16
When you’re presented with new opportunities, you want to focus on turning
them into successes, not whether your IT solution can support them.
Peer 1 Hosting powers your business with our wholly owned FastFiber Network
TM
,
solutions that are secure, scalable, and customized for your business.
Unsurpassed performance and reliability help build your business foundation to
be rock-solid, ready for high growth, and deliver the fast user experience your
customers expect.
Want more on cloud?
Call: 844.855.6655 | go.peer1.com/linux | Vew Cloud Webinar:
Public and Private Cloud | Managed Hosting | Dedicated Hosting | Colocation
power your business to its full potential
break down
your innovation barriers
Where every interaction matters.
47
80
/ MARCH 2015 / WWW.LINUXJOURNAL.COM
members of the community want to
be able to choose their favorite init
scheme. This has spawned some forks
of systemd and even a faction of the
Linux community that is for completely
boycotting systemd. Check out the
site http://boycottsystemd.org.
Conclusion
The userspace initialization and
management of Linux systems has
had a rich and diverse history. I hope
that this article has given you a new
perspective for how we got to where
we are today with systemd becoming
the new standard. I have covered all
the pros and cons of the different
schemes and how those factors have
influenced user choice in this space
over time. I hope this article will foster
further discussion, and your feedback
is highly encouraged.
Q
Jonas Gorauskas is technically a software developer by trade
but also a generalist with background in operations. In the past
he has been one or more of the following: programmer, technical
support analyst, technical writer, systems designer, database
administrator, amateur cook and professional curmudgeon.
Jonas is currently working at Intuit in Reno, Nevada, as part of
the Application Operations Engineering team helping them with
operations, deployment, DevOps or anything else they can think of.
Resources
The source code of various Linux distributions, including:
Q Debian 7 and 8
Q CentOS 6.5 and 7
Q Slackware 14
Q Fedora 20
Q Ubuntu 12.4 and 14.4
Q Arch Linux
The Web site of Lennart Poettering: http://0pointer.net/blog
The systemd Documentation: http://freedesktop.org/wiki/Software/systemd
Upstart Documentation: http://upstart.ubuntu.com/cookbook
Send comments or feedback via
http://www.linuxjournal.com/contact
or to ljeditor@linuxjournal.com.
FEATURE Initializing and Managing Services in Linux: Past, Present and Future
Documents you may be interested
Documents you may be interested
