We have several potential use cases for a new set of command line utilities, and are using those to determine which commands to implement for Toybox's 1.0 release.
Our current candidate list combines the commands toybox already implements, the development environment command list, the toolbox standard commands, various vendor configurations of busybox, a selected subset of the POSIX/SUSv4 standard, a couple of the less-insane bits of LSB, a few outright requests, plus additional to-be-determined shell functionality.
See the status page for the combined list and progress towards implementing it.
The best standards are the kind that describe reality, rather than attempting to impose a new one. (I.E. a good standard should document, not legislate.)
The kind of standards which describe existing reality tend to be approved by more than one standards body, such ANSI and ISO both approving C. That's why the IEEE POSIX committee's 2008 standard, the Single Unix Specification version 4, and the Open Group Base Specification edition 7 are all the same standard from three sources.
The "utilities" section of these standards is devoted to the unix command line, and are the best such standard for our purposes. (My earlier work on BusyBox was implemented with regard to SUSv3, an earlier version of this standard.)
Unfortunately, these standards describe a subset of reality, lacking any mention of commands such as init, login, or mount required to actually boot a system. It provides ipcrm and ipcs, but not ipcmk, so you can use System V IPC resources but not create them.
These standards also contain a large number of commands that are inappropriate for toybox to implement in its 1.0 release. (Perhaps some of these could be reintroduced in later releases, but not now.)
Starting with the full "utilities" list, we first remove generally obsolete commands (compess ed ex pr uncompress uccp uustat uux), commands for the pre-CVS "SCCS" source control system (admin delta get prs rmdel sact sccs unget val what), fortran support (asa fort77), and batch processing support (batch qalter qdel qhold qmove qmsg qrerun qrls qselect qsig qstat qsub).
Some commands are for a compiler toolchain (ar c99 cflow ctags cxref gencat iconv lex m4 make nm strings strip tsort yacc), which is outside of toybox's mandate and should be supplied externally. (Again, some of these may be revisited later, but not for toybox 1.0.)
Some commands are part of a command shell, and cannot be implemented as separate executables (alias bg cd command fc fg getopts hash jobs kill read type ulimit umask unalias wait). These may be revisited as part of a built-in toybox shell, but are not exported into $PATH via symlinks. (If you fork a child process and have it "cd" then exit, you've accomplished nothing.)
A few other commands are judgement calls, providing command-line internationalization support (iconv locale localedef), System V inter-process communication (ipcrm ipcs), and cross-tty communication from the minicomputer days (talk mesg write). The "pax" utility was supplanted by tar, "mailx" is a command line email client, and "lp" submits files for printing to... what exactly? (cups?) The standard defines crontab but not crond.
Removing all of that leaves the following commands, which toybox should implement:
at awk basename bc cal cat chgrp chmod chown cksum cmp comm cp csplit cut date dd df diff dirname du echo env expand expr false file find fold fuser getconf grep head id join kill link ln logger logname ls man mkdir mkfifo more mv newgrp nice nl nohup od paste patch pathchk printf ps pwd renice rm rmdir sed sh sleep sort split stty tabs tail tee test time touch tput tr true tty uname unexpand uniq unlink uudecode uuencode vi wc who xargs zcat
One attempt to supplement POSIX towards an actual usable system was the Linux Standard Base. Unfortunately, the quality of this "standard" is fairly low.
POSIX allowed its standards process to be compromised by leaving things out, thus allowing IBM mainframes and Windows NT to drive a truck through the holes and declare themselves compilant. But it means what they DID standardize tends to be respected.
The Linux Standard Base's failure mode is different, they respond to pressure by including special-case crap, such as allowing Red Hat to shoehorn RPM on the standard even though all sorts of distros (Debian, Slackware, Arch, Gentoo) don't use it and probably never will. This means anything in the LSB is at best a suggestion: arbitrary portions of this standard are widely ignored.
The LSB does specify a list of command line utilities:
ar at awk batch bc chfn chsh col cpio crontab df dmesg du echo egrep fgrep file fuser gettext grep groupadd groupdel groupmod groups gunzip gzip hostname install install_initd ipcrm ipcs killall lpr ls lsb_release m4 md5sum mknod mktemp more mount msgfmt newgrp od passwd patch pidof remove_initd renice sed sendmail seq sh shutdown su sync tar umount useradd userdel usermod xargs zcat
Where posix specifies one of those commands, LSB's deltas tend to be accomodations for broken tool versions which aren't up to date with the standard yet. (See more and xargs for examples.)
Since we've already committed to using our own judgement to skip bits of POSIX, and LSB's "judgement" in this regard is purely bug workarounds to declare various legacy tool implementations "compliant", this means we're mostly interested in the set of tools that aren't specified in posix at all.
Of these, gettext and msgfmt are internationalization, install_initd and remove_initd aren't present on ubuntu 10.04, lpr is out of scope, and lsb_release is a distro issue (it's a nice command, but the output of lsb_release -a is the name and version number of the linux distro you're running, which toybox doesn't know).
This leaves:
chfn chsh dmesg egrep fgrep groupadd groupdel groupmod groups gunzip gzip hostname install killall md5sum mknod mktemp mount passwd pidof sendmail seq shutdown su sync tar umount useradd userdel usermod zcat
The following commands are enough to build the Aboriginal Linux development environment, boot it to a shell prompt, and build Linux From Scratch 6.8 under it. (Aboriginal Linux currently uses BusyBox for this, thus provides a drop-in test environment for toybox. We install both implementations side by side, redirecting the symlinks a command at a time until the older package is no longer used, and can be removed.)
This use case includes running init scripts and other shell scripts, running configure, make, and install in each package, and providing basic command line facilities such as a text editor. (It does not include a compiler toolchain or C library, those are outside the scope of this project.)
bzcat cat cp dirname echo env patch rmdir sha1sum sleep sort sync true uname wc which yes zcat awk basename bzip2 chmod chown cmp cut date dd diff egrep expr find grep gzip head hostname id install ln ls mkdir mktemp mv od readlink rm sed sh tail tar touch tr uniq wget whoami xargs chgrp comm gunzip less logname man split tee test time bunzip2 chgrp chroot comm cpio dmesg dnsdomainname ftpd ftpget ftpput gunzip ifconfig init less logname losetup man mdev mount mountpoint nc pgrep pkill pwd route split stat switch_root tac umount vi
Note: Aboriginal Linux installs bash 2.05b as #!/bin/sh and its scripts require bash extensions not present in shells such as busybox ash. This means that toysh needs to supply several bash extensions _and_ work when called under the name "bash".
Android has a policy against GPL in userspace, so even though BusyBox predates Android by many years, they couldn't use it. Instead they grabbed an old version of ash and implemented their own command line utility set called "toolbox".
Toolbox doesn't have its own repository, instead it's part of Android's system/core git repository (this analysis looked at commit 51ccef27cab58).
According to core/toolbox/Android.mk the toolbox directory builds the following commands:
ls mount cat ps kill ln insmod rmmod lsmod ifconfig setconsole rm mkdir rmdir reboot getevent sendevent date wipe sync umount start stop notify cmp dmesg route hd dd df getprop setprop watchprops log sleep renice printenv smd chmod chown newfs_msdos netstat ioctl mv schedtop top iftop id uptime vmstat nandread ionice touch lsof md5 r
If selinux is enabled, you also get:
getenforce setenforce chcon restorecon runcon getsebool setsebool load_policy
The Android.mk file also refers to dynarray.c and toolbox.c as library code. This leaves the following apparently unused C files in toolbox/*.c, each of which has a command_main() function and seems to implement a standalone command:
alarm exists lsusb readtty rotatefb setkey syren
The core/sh subdirectory contains a fork of ash 1.17, and sucks in liblinenoise to provide command line history/editing.
Other than the toolbox and sh directories, the currently interesting subdirectories in the core repository are fs_mgr, gpttool, init, logcat, logwrapper, mkbootimg, netcfg, run-as, and sdcard.
Almost all of these reinvent an existing wheel with less functionality and a different user interface. We may want to provide that interface, but implementing the full commands (mount, fdisk, init, ifconfig with dhcp, and sudo) come first.
Although logcat/logwrapper also reinvent a wheel, Android did so in the kernel and these provide an interface to that.
Also, gpttool and mkbootimg are install tools, and sdcard looks like a testing tool. These aren't a priority if android wants to use its own bespoke code to install itself.
For reference, combining everything listed above, we get:
alarm ash cat chcon chmod chown cmp date dd df dmesg exists fs_mgr getenforce getevent getprop getsebool gpttool hd id ifconfig iftop init insmod ioctl ionice kill ln load_policy log logcat logwrapper ls lsmod lsof lsusb md5 mkbootimg mkdir mount mv nandread netcfg netstat newfs_msdos notify printenv ps r readtty reboot renice restorecon rm rmdir rmmod rotatefb route run-as runcon schedtop sdcard sendevent setconsole setenforce setkey setprop setsebool sleep smd start stop sync syren top touch umount uptime vmstat watchprops wipe
We may eventually implement all of that, but for toybox 1.0 we need to focus a bit. For our first pass, let's ignore selinux, strip out the "unlisted" commands except lsusb, and grab just logcat and logwrapper from the "core" commands (since the rest have some full/standard version providing that functionality, which we can implement a shim interface for later).
This means toybox should implement:
cat chmod chown cmp date dd df dmesg getevent getprop hd id ifconfig iftop insmod ioctl ionice kill ln log logcat logwrapper ls lsmod lsof lsusb md5 mkdir mount mv nandread netstat newfs_msdos notify printenv ps r reboot renice rm rmdir rmmod route schedtop sendevent setconsole setprop sleep smd start stop sync top touch umount uptime vmstat watchprops wipe
The following Toolbox commands are already covered in previous sections of this analysis:
cat chmod chown cmp date dd df dmesg id ifconfig insmod kill ln ls lsmod mkdir mount mv ps renice rm rmdir rmmod route sleep sync top touch umount
Which leaves the following commands as new from Toolbox:
getevent getprop hd iftop ioctl ionice log lsof nandread netstat newfs_msdos notify printenv r reboot schedtop sendevent setconsole setprop smd start stop top uptime vmstat watchprops wipe
The following additional commands have been requested by various users:
freeramdisk getty halt hexdump hwclock klogd modprobe ping ping6 pivot_root poweroff rev sfdisk sudo syslogd taskset telnet telnetd tracepath traceroute unzip usleep vconfig zip free login modinfo unshare netcat help w