1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
|
<html><head><title>toybox FAQ</title>
<!--#include file="header.html" -->
<h1>Frequently Asked Questions</h1>
<h2>General Questions</h2>
<ul>
<li><h2><a href="#capitalize">Do you capitalize toybox?</a></h2></li>
<li><h2><a href="#why_toybox">Why toybox? (What was wrong with busybox?)</a></h2></li>
<li><h2><a href="#support_horizon">Why a 7 year support horizon?</a></h2></li>
<li><h2><a href="#releases">Why time based releases?</a></h2></li>
<li><h2><a href="#code">Where do I start understanding the toybox source code?</a></h2></li>
</ul>
<h2>Using toybox</h2>
<ul>
<!-- get binaries -->
<li><h2><a href="#cross">How do I cross compile toybox?</h2></li>
<li><h2><a href="#mkroot">How do you build a working Linux system with toybox?</a></h2></li>
</ul>
<a name="capitalize" />
<h2>Q: Do you capitalize toybox?</h2>
<p>A: Only at the start of a sentence. The command name is all lower case so
it seems silly to capitalize the project name, but not capitalizing the
start of sentences is awkward, so... compromise. (It is _not_ "ToyBox".)</p>
<a name="why_toybox" />
<h2>Q: "Why is there toybox? What was wrong with busybox?"</h2>
<p>A: Toybox started back in 2006 when I
<a href=https://lwn.net/Articles/202106/>handed off BusyBox maintainership</a>
and <a href=http://landley.net/notes-2006.html#28-09-2006>started over from
scratch</a> on a new codebase after a
<a href=http://lists.busybox.net/pipermail/busybox/2006-September/058617.html>protracted licensing argument</a> took all the fun out of working on BusyBox.</p>
<p>Toybox was just a personal project until it got
<a href=http://landley.net/notes-2011.html#13-11-2011>relaunched
in November 2011</a> with a new goal to
<a href=http://landley.net/aboriginal/about.html#selfhost>make Android
self-hosting</a>. This involved me relicensing my own
code, which <a href=https://lwn.net/Articles/478308/>made people who had
never used or participated in the project loudly angry</a>. The switch came
after a lot of thinking <a href=http://landley.net/talks/ohio-2013.txt>about
licenses</a> and <a href=http://landley.net/notes-2011.html#21-03-2011>the
transition to smartphones</a>, which led to a
<a href=http://landley.net/talks/celf-2013.txt>2013</a>
<a href=https://www.youtube.com/watch?v=SGmtP5Lg_t0>talk</a> laying
out a strategy to make Android self-hosting using toybox. This helped
<a href=https://code.google.com/p/android/issues/detail?id=76861>bring
it to Android's attention</a>, and they
<a href=https://lwn.net/Articles/629362/>merged it</a> into Android M.</p>
<p>The answer to the second question is "licensing". BusyBox predates Android
by almost a decade but Android still doesn't ship with it because GPLv3 came
out around the same time Android did and caused many people to throw
out the GPLv2 baby with the GPLv3 bathwater.
Android <a href=https://source.android.com/source/licenses.html>explicitly
discourages</a> use of GPL and LGPL licenses in its products, and has gradually
reimplemented historical GPL components such as its bluetooth stack under the
Apache license. Similarly, Apple froze xcode at the last GPLv2 releases
(GCC 4.2.1 with binutils 2.17) for over 5 years while it sponsored the
development of new projects (clang/llvm/lld) to replace them,
implemented its SMB server from scratch to replace samba,
<a href=http://meta.ath0.com/2012/02/05/apples-great-gpl-purge/>and so
on</a>. Toybox itself exists because somebody with in a legacy position
just wouldn't shut up about GPLv3, otherwise I would probably
still happily be maintaining BusyBox. (For more on how I wound
up working on busybox in the first place,
<a href=http://landley.net/aboriginal/history.html>see here</a>.)</p>
<h2><a name="support_horizon">Q: Why a 7 year support horizon?</a></h2>
<p>A: Our <a href=http://lists.busybox.net/pipermail/busybox/2006-September/058440.html>longstanding rule of thumb</a> is to try to run and build on
hardware and distributions released up to 7 years ago, and feel ok dropping
support for stuff older than that. (This is a little longer than Ubuntu's
Long Term Support, but not by much.)</p>
<p>My original theory was "4 to 5 18-month cycles of moore's law should cover
the vast majority of the installed base of PC hardware", loosely based on some
research I did <a href=http://www.catb.org/esr/halloween/halloween9.html#id2867629>back in 2003</a>
and <a href=http://catb.org/esr/writings/world-domination/world-domination-201.html#id248066>updated in 2006</a>
which said that low end systems were 2 iterations of moore's
law below the high end systems, and that another 2-3 iterations should cover
the useful lifetime of most systems no longer being sold but still in use and
potentially being upgraded to new software releases.</p>
<p>It turns out <a href=http://landley.net/notes-2011.html#26-06-2011>I missed
industry changes</a> in the 1990's that stretched the gap
from low end to high end from 2 cycles to 4 cycles, and _that_ analysis
ignored the switch from PC to smartphone cutting off the R&D air supply of the
laptop market. Meanwhile the Moore's Law s-curve started bending
down in 2000 and these days is pretty flat because the drive for faster clock
speeds <a href=http://www.anandtech.com/show/613>stumbled</a>
then <a href=http://www.pcworld.com/article/118603/article.html>died</a>, and
the subsequent drive to go wide maxed out around 4x SMP with ~2 megabyte
caches for most applications. These days the switch from exponential to
linear growth in hardware capabilities is
<a href=https://www.cnet.com/news/end-of-moores-law-its-not-just-about-physics/>common</a>
<a href=http://www.acm.org/articles/people-of-acm/2016/david-patterson>knowledge</a>.</p>
<p>But the 7 year rule of thumb stuck around anyway: if a kernel or libc
feature is less than 7 years old, I try to have a build-time configure test
for it and let the functionality cleanly drop out. I also keep old Ubuntu
images around in VMs and perform the occasional defconfig build there to
see what breaks. (I'm not perfect about this, but I accept bug reports.)</p>
<h2><a name="releases" />Q: Why time based releases?</h2>
<p>A: Toybox targets quarterly releases (a similar schedule to the Linux
kernel) because Martin Michlmayr's
<a href=http://www.youtube.com/watch?v=IKsQsxubuAA>talk</a> on the
subject was convincing.</p>
<p>Releases provide synchronization points where the developers certify
"it worked for me". Each release is a known version with predictable behavior,
and right or wrong at least everyone should be seeing
similar results where you might be able to google an unexpected outcome.
Releases focus end-user testing on specific versions
where issues can be reproduced, diagnosed, and fixed.
Releases also force the developers to do periodic tidying, packaging,
documentation review, finish up partially implemented features languishing
in their private trees, and give regular checkpoints to measure progress.</p>
<p>Over time feature sets change, data formats change, control knobs change...
For example toybox's switch from "ls -q" to "ls -b" as the default output
format wasn't exactly a bug, it was a design improvement... but the
difference is academic if the change breaks somebody's script.
Releases give you the option to schedule upgrades later, and not to rock
the boat just now: just use a known working release version.</p>
<p>The counter-argument is that "continuous integration"
can be made robust with sufficient automated testing. But like the
<a href=http://www.shirky.com/weblog/2013/11/healthcare-gov-and-the-gulf-between-planning-and-reality/>waterfall method</a>, this places insufficent
emphasis on end-user feedback and learning from real world experience.
Developer testing is either testing that the code does what the developers
expect given expected inputs running in an expected environment, or it's
regression testing against bugs previously found in the field. No plan
survives contact with the enemy, and technology always breaks once it
leaves the lab and encounters real world data and use cases, not just
at runtime but in different build environments.</p>
<p>The best way to give new users a reasonable first experience is to point
them at specific stable versions where development quiesced and
extra testing occurred. There will still be teething troubles, but multiple
people experiencing the _same_ teething troubles can potentially
help each other out.</p>
<p>As for why releases on a schedule are better than releases "when it's
ready", watch the video.</p>
<h2><a name="code" />Q: Where do I start understanding the source code?</h2>
<p>A: Toybox is written in C. There are longer writeups of the
<a href=design.html>design ideas</a> and a <a href=code.html>code walkthrough</a>,
and the <a href=about.html>about page</a> summarizes what we're trying to
accomplish, but here's a quick start:</p>
<p>Toybox uses the standard three stage configure/make/install
<a href=code.html#building>build</a>, in this case "<b>make defconfig;
make; make install</b>". Type "<b>make help</b>" to
see available make targets.</p>
<p><b>The configure stage is copied from the Linux kernel</b> (in the "kconfig"
directory), and saves your selections in the file ".config" at the top
level. The "defconfig" target selects the
maximum sane configuration (enabling all the commands and features that
aren't unfinished, only intended as examples, debug code, etc) and is
probably what you want. You can use "make menuconfig" to manually select
specific commands to include, through an interactive menu (cursor up and
down, enter to descend into a sub-menu, space to select an entry, ? to see
an entry's help text, esc to exit). The menuconfig help text is the
same as the command's --help output.</p>
<p><b>The "make" stage creates a toybox binary</b> (which is stripped, look in
generated/unstripped for the debug versions), and "install" adds a bunch of
symlinks to toybox under the various command names. Toybox determines which
command to run based on the filename, or you can use the "toybox" name in which case the first
argument is the command to run (ala "toybox ls -l"). <b>You can also build
individual commands as standalone executables</b>, ala "make sed cat ls".
(The "make change" target builds all of them, as in "change for a $20".)</p>
<p><b>The main() function is in main.c at the top level</b>,
along with setup plumbing and selecting which command to run this time.
The function toybox_main() in the same file implements the "toybox"
multiplexer command that lists and selects the other commands.</p>
<p><b>The individual command implementations are under "toys"</b>, and are grouped
into categories (mostly based on which standard they come from, posix, lsb,
android...) The "pending" directory contains unfinished commands, and the
"examples" directory contains example code that aren't really useful commands.
Commands in those two directories
are _not_ selected by defconfig. (Most of the files in the pending directory
are third party submissions that have not yet undergone
<a href=cleanup.html>proper code review</a>.)</p>
<p><b>Common infrastructure shared between commands is under "lib"</b>. Most
commands call lib/args.c to parse their command line arguments before calling
the command's own main() function, which uses the option string in
the command's NEWTOY() macro. This is similar to the libc function getopt(),
but more powerful, and is documented at the top of lib/args.c. A NULL option
string prevents this code from being called for that command.</p>
<p>Most of the actual <b>build/install infrastructure is shell scripts under
"scripts"</b> (starting with scripts/make.sh and scripts/install.sh).
<b>These populate the "generated" directory</b> with headers
created from other files, which are <a href=code.html#generated>described</a>
in the code walkthrough. All the
build's temporary files live under generated, including the .o files built
from the .c files (in generated/obj). The "make clean" target deletes that
directory. ("make distclean" also deletes your .config and deletes the
kconfig binaries that process .config.)</p>
<p>Each command's .c file contains all the information for that command, so
<b>adding a command to toybox means adding a single file under "toys"</b>.
Usually you <a href=code.html#adding>start a new command</a> by copying an
existing command file to a new filename
(toys/examples/hello.c, toys/examples/skeleton.c, toys/posix/cat.c,
and toys/posix/true.c have all been used for this purpose) and then replacing
all instances of its old name with the new name (which should match the
new filename), and modifying the help text, argument string, and what the
code does. You might have to "make distclean" before your new command
shows up in defconfig or menuconfig.</p>
<p><b>The toybox test suite lives in the "tests" directory</b>, and is
driven by scripts/test.sh and scripts/runtest.sh. From the top
level you can "make tests" to test everything, or "make test_sed" to test a
single command's standalone version (which should behave identically,
but that's why we test). You can set TEST_HOST=1 to test the host versionn
instead of the toybox version (in theory they should work the same),
and VERBOSE=1 to see diffs of the expected and actual output when a
test fails (VERBOSE=fail to stop at the first such failure)</p>
<a name="cross" />
<h2>Q: How do I cross compile toybox?</h2>
<p>A: toybox is tested against three C libraries (bionic, musl, glibc)
with 2 compilers (llvm, gcc). The easy way to get coverage (if not every
combination) is:</p>
<p><b>1) gcc+glibc = host toolchain</b></p>
<p>Most Linux distros come with that as a host compiler, just build normally
(<b>make distclean defconfig toybox</b>, or <b>make menuconfig</b> followed
by <b>make</b>).</p>
<p>You can use LDFLAGS=--static if you want static binaries, but static
glibc is hugely inefficient ("hello world" is 810k on x86-64) and throws a
zillion linker warnings because one of its previous maintainers
<a href=https://www.akkadia.org/drepper/no_static_linking.html>was insane</a>
(which meant at the time he refused to fix
<a href=https://elinux.org/images/2/2d/ELC2010-gc-sections_Denys_Vlasenko.pdf>obvious bugs</a>), plus it uses dlopen() at runtime to implement basic things like
<a href=https://stackoverflow.com/questions/15165306/compile-a-static-binary-which-code-there-a-function-gethostbyname>DNS lookup</a> (which is impossible
to support properly from a static binary because you wind up with two
instances of malloc() managing two heaps which corrupt as soon as a malloc()
from one is free()d into the other), although glibc added
<a href=https://stackoverflow.com/questions/14289488/use-dlsym-on-a-static-binary>improper support</a> which still requires the shared libraries to be
installed on the system alongside the static binary:
<a href=https://www.youtube.com/watch?v=Ih-3vK2qLls>in brief, avoid</a>.)
These days glibc is maintained by a committee instead of a single
maintainer, if you consider that an improvement.</p>
<p><b>2) gcc+musl = musl-cross-make</b>
<p>The cross compilers I test this with are built from the
<a href=http://musl.libc.org/>musl-libc</a> maintainer's
<a href=https://github.com/richfelker/musl-cross-make>musl-cross-make</a>
project, built by running toybox's scripts/mcm-buildall.sh in that directory,
and then symlink the resulting "ccc" subdirectory into toybox where
"make root CROSS=" can find them, ala:</p>
<blockquote><pre>
cd ~
git clone https://github.com/landley/toybox
git clone https://github.com/richfelker/musl-cross-make
cd musl-cross-make
../toybox/scripts/mcm-buildall.sh # this takes a while
ln -s $(realpath ccc) ../toybox/ccc
</pre></blockquote>
<p>Instead of symlinking ccc, you can specify a CROSS_COMPILE= prefix
in the same format the Linux kernel build uses. You can either provide a
full path in the CROSS_COMPILE string, or add the appropriate bin directory
to your $PATH. I.E:</p>
<blockquote>
<p>make LDFLAGS=--static CROSS_COMPILE=~/musl-cross-make/ccc/m68k-linux-musl-cross/bin/m68k-linux-musl- distclean defconfig toybox</p>
</blockquote>
<p>Is equivalent to:</p>
<blockquote><p>
export "PATH=~/musl-cross-make/ccc/m68k-linux-musl-cross/bin:$PATH"<br />
LDFLAGS=--static make distclean defconfig toybox CROSS=m68k-linux-musl-
</p></blockquote>
<p>Note: a non-static build won't run unless you install musl on your host.
In theory you could "make root" a dynamic root filesystem with musl by copying
the shared libraries out of the toolchain, but I haven't bothered implementing
that yet because a static linked musl hello world is 10k on x86
(5k if stripped).</p>
<p><b>3) llvm+bionic = Android NDK</b></p>
<p>The <a href=https://developer.android.com/ndk/downloads>Android
Native Development Kit</a> provides an llvm toolchain with the bionic
libc used by Android. To turn it into something toybox can use, you
just have to add an appropriately prefixed "cc" symlink to the other
prefixed tools, ala:</p>
<blockquote><pre>
unzip android-ndk-r21b-linux-x86_64.zip
cd android-ndk-21b/toolchains/llvm/prebuilt/linux-x86_64/bin
ln -s x86_64-linux-android29-clang x86_64-linux-android-cc
PATH="$PWD:$PATH"
cd ~/toybox
make distclean
make LDFLAGS=--static CROSS_COMPILE=x86_64-linux-android- defconfig toybox
</pre></blockquote>
<p>Again, you need a static link unless you want to install bionic on your
host. Binaries statically linked against bionic are almost as big as with
glibc, but at least it doesn't have the dlopen() issues.</p>
<p>Unfortunately, although the resulting toybox will run a bionic-based
chroot will not, because even "hello world" statically linked
against bionic will segfault before calling main() if /dev/null isn't
present, and the init script written by mkroot.sh has to run a shell linked
against bionic in order to mount devtmpfs on /dev to provide /dev/null.</p>
<a name="mkroot" />
<h2>Q: How do you build a working Linux system with toybox?</h2>
<p>A: To build a native chroot, "<b>make root</b>" then "<b>sudo chroot
root/host/fs /init</b>". Type "exit" to get back out.</p>
<p>You can also cross compile simple two package (toybox+linux) systems that
boot to a shell prompt under <a href=https://qemu.org>qemu</a>, by specifying
a target type with CROSS=
(or setting CROSS_COMPILE= to a cross compiler prefix with optional absolute
path), and pointing the build at a Linux kernel source directory, ala:</p>
<blockquote><p><b>make root CROSS=sh4 LINUX=~/linux</b></p></blockquote>
<p>Then you can <b>cd root/sh4; ./qemu-sh4.sh</b> to launch the emulator
(you need the appropriate qemu-system-* binary installed). Type "exit"
when done and it should shut down the emulator on the way out.</p>
<p>The cross compilers I test this with are built from the musl-libc
maintainer's
<a href=https://github.com/richfelker/musl-cross-make>musl-cross-make</a>
project, built by running toybox's scripts/mcm-buildall.sh in that directory,
and then symlink the resulting "ccc" subdirectory into toybox where CROSS=
can find them, ala:</p>
<blockquote><pre>
cd ~
git clone https://github.com/landley/toybox
git clone https://github.com/richfelker/musl-cross-make
cd musl-cross-make
../toybox/scripts/mcm-buildall.sh # this takes a while
ln -s $(realpath ccc) ../toybox/ccc
</pre></blockquote>
<p>An unknown CROSS= should list all the available cross compilers under
ccc. After the above musl-cross-make build and install,
"<b>make root CROSS=help</b>" should say something like:
<b>aarch64 armv4l armv5l armv7l armv7m armv7r i486 i686 m68k microblaze mips mips64 mipsel powerpc powerpc64 powerpc64le s390x sh2eb sh4 x32 x86_64</b>.
(A long time ago I
<a href=http://landley.net/aboriginal/architectures.html>tried to explain</a>
what some of these architectures were.)</p>
<p>You can build all the targets at once, and can add additonal packages
to the build, by calling the script directly and listing packages on
the command line:</p>
<blockquote>
<p><b>scripts/mkroot.sh CROSS=all LINUX=~/linux dropbear</b></p>
</blockquote>
<p>An example package build script (building the dropbear ssh server, adding a
port forward from 127.0.0.1:2222 to the qemu command line, and providing a
ssh2dropbear.sh convenience script to the output directory) is provided
in the scripts/root directory. If you add your own scripts elsewhere, just
give a path to them on the command line. (No, I'm not merging more package build
scripts, I <a href=https://speakerdeck.com/landley/developing-for-non-x86-targets-using-qemu?slide=78>learned that lesson</a> long ago. But if you
want to write your own, feel free.)</p>
<p>(Note: if you don't have a <b>.config</b> it'll <b>make defconfig</b> and
add CONFIG_SH and CONFIG_ROUTE to it. If you already have a <b>.config</b> that
_doesn't_ have CONFIG_SH in it, you won't get a shell prompt or be able to run
the init script without a shell. This is currently a problem because sh
and route are still in pending and thus not in defconfig, and "make root"
cheats when it adds them. I'm working on it. tl;dr if make root doesn't work
"rm .config" and run it again, and all this should be fixed up in future when
those two commands are promoted out of pending so "make defconfig" would have
what you need anyway. It's designed to let yout tweak your config, which is
why it uses the .config that's there when there is one, but the default is
currently wrong because it's not quite finished yet.)</p>
<!--#include file="footer.html" -->
|
