why.html (11540B)
1 <html> 2 <head> 3 <meta name="viewport" content="width=device-width, initial-scale=1.0" /> 4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 5 <meta http-equiv="Content-Language" content="en" /> 6 <title>s6: why another supervision suite</title> 7 <meta name="Description" content="s6: why another supervision suite" /> 8 <meta name="Keywords" content="s6 supervision daemontools runit perp service svscan supervise" /> 9 <!-- <link rel="stylesheet" type="text/css" href="//skarnet.org/default.css" /> --> 10 </head> 11 <body> 12 13 <p> 14 <a href="index.html">s6</a><br /> 15 <a href="//skarnet.org/software/">Software</a><br /> 16 <a href="//skarnet.org/">skarnet.org</a> 17 </p> 18 19 <h1> Why another supervision suite ? </h1> 20 21 <p> 22 Supervision suites are becoming quite common. Today, we already have: 23 </p> 24 25 <ul> 26 <li> Good (?) old System V init, which can be made to supervise services if you perform <tt>/etc/inittab</tt> voodoo. 27 BSD init can also be used the same way with the <tt>/etc/ttys</tt> file, but for some reason, nobody among BSD 28 developers is using <tt>/etc/ttys</tt> to this purpose, so I won't consider BSD init here. </li> 29 <li> <a href="https://cr.yp.to/daemontools.html">daemontools</a>, the pioneer </li> 30 <li> <a href="https://untroubled.org/daemontools-encore/">daemontools-encore</a>, Bruce Guenter's upgrade to daemontools </li> 31 <li> <a href="http://smarden.org/runit/">runit</a>, Gerrit Pape's suite, well-integrated with Debian </li> 32 <li> <a href="http://b0llix.net/perp/">perp</a>, Wayne Marshall's take on supervision </li> 33 <li> Integrated init systems providing a lot of features, process supervision being one of them. 34 For instance, <a href="https://upstart.ubuntu.com/">Upstart</a>, MacOS X's 35 <a href="https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man8/launchd.8.html">launchd</a>, 36 and Fedora's <a href="https://freedesktop.org/wiki/Software/systemd">systemd</a>. </li> 37 </ul> 38 39 <p> 40 Why is s6 needed ? What does it do differently ? Here are the criteria I used. 41 </p> 42 43 44 <h2> Supervision suites should not wake up unless notified. </h2> 45 46 <ul> 47 <li> System V init fails the test: it wakes up every 5 seconds, for the reason that 48 <tt>/dev/initctl</tt> might have changed. 49 <a href="https://i.imgur.com/iWKad22.jpg"><tt>m(</tt></a> </li> 50 <li> daemontools fails the test: it wakes up every 5 seconds to check for new services. </li> 51 <li> daemontools-encore does the same. </li> 52 <li> the current version of runit fails the test: it wakes up every 14 seconds. But this is a workaround for a bug in some Linux kernels; 53 there is no design flaw in runit that prevents it from passing the test. </li> 54 <li> perp works. </li> 55 <li> Upstart works. I have no idea what other integrated init systems do: it's much too difficult to strace them 56 to see exactly where they're spending their time, and when it is possible, the trace output is so big that it's 57 hard to extract any valuable information from it. </li> 58 <li> s6 works. The <tt>-t</tt> option to s6-svscan makes it check its services 59 with a configurable timeout; by default, this timeout is infinite, i.e. it 60 never wakes up unless it receives a command via s6-svscanctl. </li> 61 </ul> 62 63 64 <h2> Supervision suites should provide a program that can run as process 1. </h2> 65 66 <ul> 67 <li> System V init <em>is</em> process 1, so no problem here. </li> 68 <li> Integrated init systems, by definition, provide a process 1. </li> 69 <li> daemontools was not designed to take over init, although 70 <a href="https://code.dogmap.org./svscan-1/">it can be made to work</a> with 71 enough hacking skills. Same thing with daemontools-encore. </li> 72 <li> runit provides an <em>init</em> functionality, but the mechanism is 73 separate from the supervision itself; the <tt>runit</tt> process, not the 74 <tt>runsvdir</tt> process, runs as process 1. This lengthens the supervision 75 chain. </li> 76 <li> perp was not designed to run as process 1. It probably could be made to work too 77 without too much trouble. </li> 78 <li> s6-svscan was designed from the start to be run as process 1, although it 79 does not have to. </li> 80 </ul> 81 82 83 <h2> Supervision suites should be bug-free, lightweight and easy to understand. </h2> 84 85 <ul> 86 <li> daemontools, daemontools-encore, runit and perp all qualify. All of this is excellent quality 87 code, <a href="//skarnet.org/software/skalibs/djblegacy.html">unsurprisingly</a>. </li> 88 <li> System V init is understandable, and reasonably lightweight; but it is still 89 too big for what it does - poorly. The <tt>/etc/inittab</tt> file needs to be parsed; 90 that parser has to be in process 1. There is support in process 1 for the whole 91 "runlevel" concept, which is a primitive form of service management. The same 92 executable handles all 3 stages of the machine's lifetime and does not separate 93 them properly. All in all, System V init does its job, but is showing its age 94 and nowadays we know much better designs. </li> 95 <li> This is where integrated init systems fail, hard. By wanting to organize 96 the way a the machine is operated - so, machine state management - in the 97 <em>same package</em> as the init and process supervision system, they add 98 incredible complexity where it does not belong. 99 <ul> 100 <li> Upstart uses <tt>ptrace</tt> to watch its children fork(), and links 101 process 1 against libdbus. This is insane. 102 Process 1 should be <em>absolutely stable</em>, it should be guaranteed 103 to never crash, so the whole of its source code should be under control. At 104 Upstart's level of complexity, those goals are outright impossible to achieve, 105 so this approach is flawed by design. It is a shame, because the concepts 106 and ideas behind Upstart are <em>good</em> and <em>sound</em>; it's the 107 implementation choices that are its downfall. </li> 108 <li> launchd suffers from the same kind of problems. Example: Services 109 running under launchd must be configured 110 <a href="https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html">using 111 XML</a>; the launchctl process interprets the XML, converts it into a 112 key-value store (which is strictly 113 less powerful than XML, so why do they even use XML in the first place?) 114 and sends it to launchd via a Mach-specific IPC. Process 1 needs to be 115 linked against the library that handles the Mach IPC, it needs to decode 116 the key-value store, and use it to run and supervise a daemon. And it 117 needs to keep everything in memory. This is 118 a lot more complex and resource-consuming than it needs to be. </li> 119 <li> systemd is much, much worse than the other ones, and a real danger 120 for the future of GNU/Linux. I have a <a href="//skarnet.org/software/systemd.html">special page</a> 121 dedicated to it. </li> 122 </ul> 123 What those systems fail to recognize is that process supervision, rooted in 124 process 1, is a good thing, and machine management is also a good thing, but 125 <strong>those are two different functions</strong>, and a good init system 126 needs, and <strong>should</strong>, only provide process supervision, in 127 order to keep such a crucial piece of code as easy to maintain as possible. 128 Machine management can be added <em>on top of</em> a process supervision 129 suite, in a different package, and it has nothing to do with process 1. </li> 130 <li> s6, which has been designed with embedded environments in mind, tries 131 harder than anyone to pass this. It tries so hard that <tt>s6-svscan</tt> 132 and <tt>s6-supervise</tt>, the two long-running programs that make the 133 supervision chain, <em>do not even allocate heap memory</em>, and their main 134 program source files are less than 500 lines long. </li> 135 </ul> 136 137 138 <h2> Supervision suites should provide a basis for high-level service management. </h2> 139 140 <ul> 141 <li> Neither System V init, daemontools, runit or perp 142 provides any hooks to wait for a service to go up or down. runit provides a 143 waiting mechanism, but it's based on polling, and the <tt>./check</tt> script 144 has to be manually written for every service. </li> 145 <li> daemontools-encore qualifies: the <em>notify script</em> can be used for 146 inter-service communication. But it's just a hook: all the real notification 147 work has to be done by the notify script itself, no notification framework is 148 provided. </li> 149 <li> Integrated init systems provide high-level service management 150 themselves. Again, this is not good design: service management has nothing 151 to do with init or process supervision, and should be implemented on top 152 of it, not as a part of it. </li> 153 <li> s6 comes with <a href="ftrig.html">an event notification 154 library</a>, and command-line tools based on this library, thus providing a simple 155 API for future service management tools to build upon. </li> 156 </ul> 157 158 159 <h2> Artistic considerations </h2> 160 161 <ul> 162 <li> <tt>s6-svscan</tt> and <tt>s6-supervise</tt> are <em>entirely asynchronous</em>. 163 Even during trouble (full process table, for instance), they'll remain reactive 164 and instantly respond to commands they may receive. <tt>s6-supervise</tt> has 165 even been implemented as a full deterministic finite automaton, to ensure it 166 always does the right thing under any circumstance. Other supervision suites 167 do not achieve that for now. </li> 168 <li> daemontools' <a href="https://cr.yp.to/daemontools/svscan.html">svscan</a> 169 maintains an open pipe between a daemon and its logger, so even if the daemon, 170 the logger, <em>and</em> both 171 <a href="https://cr.yp.to/daemontools/supervise.html">supervise</a> processes 172 die, the pipe is still the same so <em>no logs are lost, ever</em>, unless 173 svscan itself dies. </li> 174 <li> runit has only one supervisor, <a href="http://smarden.org/runit/runsv.8.html">runsv</a>, 175 for both a daemon and its logger. The pipe is maintained by <tt>runsv</tt>. 176 If the <tt>runsv</tt> process dies, the pipe disappears and logs are lost. 177 So, runit does not offer as strong a guarantee as daemontools. </li> 178 <li> perp has only one process, <a href="http://b0llix.net/perp/site.cgi?page=perpd.8">perpd</a>, 179 acting both as a "daemon and logger supervisor" (like <tt>runsv</tt>) and as a 180 "service directory scanner" (like <tt>runsvdir</tt>). It maintains the pipes 181 between the daemons and their respective loggers. If perpd dies, everything 182 is lost. Since perpd cannot be run as process 1, this is a possible SPOF for 183 a perp installation; however, perpd is well-written and has virtually no risk of 184 dying, especially compared to process 1 behemoths provided by integrated 185 init systems. </li> 186 <li> Besides, the <tt>runsv</tt> model, which has to handle both a daemon 187 and its logger, is more complex than the <tt>supervise</tt> model (which 188 only has to handle a daemon). Consequently, the <tt>runsvdir</tt> model is 189 simpler than the <tt>svscan</tt> model, but there is only one <tt>svscan</tt> 190 instance when there are several <tt>runsv</tt>s and <tt>supervise</tt>s. 191 The <tt>perpd</tt> model is obviously the most complex; while very understandable, 192 <tt>perpd</tt> is unarguably harder to maintain than the other two. </li> 193 <li> So, to achieve maximum simplicity and code reuse, and minimal memory 194 footprint, s6's design is close to daemontools' one. 195 And when <a href="s6-svscan-1.html">s6-svscan is run as process 1</a>, 196 pipes between daemons and loggers are never lost. </li> 197 </ul> 198 199 200 <h2> Conclusion </h2> 201 202 <p> 203 All in all, I believe that s6 offers the best overall implementation of a 204 supervision suite <em>as it should be designed</em>. At worst, it's just another 205 take on daemontools with a <a href="//skarnet.org/software/skalibs/">reliable 206 base library</a> and a few nifty features. 207 </p> 208 209 </body> 210 </html>