From 9a69a929eced859415a46175d891d547ddfbb3bd Mon Sep 17 00:00:00 2001 From: Rob Landley Date: Sat, 23 Feb 2013 18:32:08 -0600 Subject: Accumulated web page changes. (Release announcement for previous release, roadmap updates, etc.) --- scripts/mkstatus.py | 12 +- www/about.html | 10 +- www/code.html | 0 www/design.html | 0 www/news.html | 41 ++++++ www/roadmap.html | 356 +++++++++++++++++++++++++++++++++++++++++++++++++--- www/status.html | 11 +- 7 files changed, 403 insertions(+), 27 deletions(-) mode change 100644 => 100755 www/code.html mode change 100644 => 100755 www/design.html diff --git a/scripts/mkstatus.py b/scripts/mkstatus.py index 51adc2cd..ec225ed4 100755 --- a/scripts/mkstatus.py +++ b/scripts/mkstatus.py @@ -1,5 +1,7 @@ #!/usr/bin/python +# Create status.html + import subprocess,sys def readit(args): @@ -32,6 +34,8 @@ for i in toystuff: pending=[] done=[] +print "all commands=%s" % len(reverse) + outfile=open("www/status.gen", "w") outfile.write("

All commands

\n") @@ -42,11 +46,14 @@ for i in blah: if "posix" in reverse[i]: out='[%s]' % (i,out) elif "lsb" in reverse[i]: out='<%s>' % (i,out) elif "development" in reverse[i]: out='(%s)' % (i,out) - elif "request" in reverse[i]: out='%s' % (i,out) elif "toolbox" in reverse[i]: out='{%s}' % out + elif "klibc_cmd" in reverse[i]: out='=%s=' % out + elif "sash_cmd" in reverse[i]: out='#%s#' % out + elif "sbase_cmd" in reverse[i]: out='@%s@' % out + elif "beastiebox_cmd" in reverse[i]: out='*%s*' % out + elif "request" in reverse[i]: out='+%s+' % (i,out) elif "ready" in reverse[i]: pass else: sys.stderr.write("unknown %s %s\n" % (i, reverse[i])) - if "ready" in reverse[i] or "pending" in reverse[i]: done.append(out) out='%s' % out @@ -54,6 +61,7 @@ for i in blah: outfile.write(out+"\n") +print "done=%s" % len(done) outfile.write("

\n") outfile.write("

TODO

%s

\n" % "\n".join(pending)) diff --git a/www/about.html b/www/about.html index 74eca0f2..7d98de4b 100755 --- a/www/about.html +++ b/www/about.html @@ -65,19 +65,19 @@ tinycc) can rebuild itself from source code.

Most commands are implemented according to The -Single Unix Specification version 4 where applicable. This does not mean -that Toybox is implementing every SUSv3 utility: some such as SCCS and ed are +Single Unix Specification version 4 where applicable. This does not mean +that Toybox is implementing every SUSv4 utility: some such as SCCS and ed are obsolete, while others such as c99 are outside the scope of the project. Toybox also isn't implementing full internationalization support: it should be 8-bit clean and handle UTF-8, but otherwise we leave this to X11 and higher -layers. And some things (like $CDPATH support in "cd") await a good -explanation of why to bother with them. (The standard provides an important +layers. And some things (like $CDPATH support in "cd") await a good +explanation of why to bother with them. (The standard provides an important frame of reference, but is not infallable set of commandments to be blindly obeyed.)

The other major sources of commands are the Linux man pages, and testing the behavior of existing commands (although not generally looking at their -source code), including the commands in Android's toolbox. SUSv4 does not +source code), including the commands in Android's toolbox. SUSv4 does not include many basic commands such as "mount", "init", and "mke2fs", which are kind of nice to have.

diff --git a/www/code.html b/www/code.html old mode 100644 new mode 100755 diff --git a/www/design.html b/www/design.html old mode 100644 new mode 100755 diff --git a/www/news.html b/www/news.html index 1ed7ae9b..4f413217 100755 --- a/www/news.html +++ b/www/news.html @@ -1,6 +1,47 @@

News

+
January 18, 2013 +

This must be Thursday. I never could get the hang of Thursdays. - The Hitchhiker's Guide to the Galaxy.

+ +

Toybox 0.4.3 is based on +commit 793. There +are now exactly 100 commands in defconfig (of a little over 220 on the +todo list).

+ +

Elie De Brauwer added the rev command, cleaned up tac, implemented the -s +and -f flags for seq, added -v and -i to killall (and fixed killall not to +kill itself before finishing its pid list), and added to the test suite. +Felix Janda added -m to mkdir, pwd -L and -P, and more test suite entries.

+ +

Rob Landley added the losetup command, and fixed the existing ls, cp, and +readlink commands. The segfault in ls +happened when it couldn't determine the screen size (last release changed the +default to -C and a screen size of 0 made column view unhappy), and cp got an +extensive rewrite bringing it up to date with the dirtree changes and fixing +a number of things it never did right in the first place. The xabspath() +code in the library now handles a symlink after ".." properly (and the test +suite checks for it).

+ +

Infrastructure-wise the code is better about automatically setting the +error return code properly. Now error_msg() sets the exit code to 1 if it's +still defaulting to 0, and the global exit path does a fflush(NULL) with error +bit check rather than trying to be quite so granular about flushing. (That +means if we use printf() instead of xprintf() it still exits with the right +error code, it just doesn't end the program early on an output error.) +Minor bugfix so TOYBOX_DEBUG +doesn't always warn about the lack of suid bit when toybox is built with +at least one STAYROOT command. Bugfix for the option [grouping] logic +(and then further fixes to the error reporting pointed out by Ashwini Sharma). +dirtree_handle_callback() now has a prefix like the rest of the dirtree +functions. A lot of stuff doing manual path handling was switched to using +libc basename() (including, embarassingly, the basename command), which means +it now correctly detects "/trailing/slash/" which the previous code didn't.

+ +

Also, last release included some accidentally checked in debug code that +disabled compiler optimization, so the binary size bloated a bit. It's back +to -Os by default now.

+
December 15, 2012

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly diff --git a/www/roadmap.html b/www/roadmap.html index 30d4dd4f..549c9486 100755 --- a/www/roadmap.html +++ b/www/roadmap.html @@ -7,15 +7,36 @@ utilities, and are using those to determine which commands to implement for Toybox's 1.0 release.

-

Our current candidate list combines the commands toybox already implements, -the development environment command list, the toolbox standard commands, -various vendor configurations of busybox, a selected subset of the POSIX/SUSv4 -standard, a couple of the less-insane bits of LSB, a few outright requests, -plus additional to-be-determined shell functionality.

+

The most interesting standards are POSIX-2008 (also known as the Single +Unix Specification version 4) and the Linux Standard Base (version 4.1). +The main test harness including toybox in Aboriginal Linux and if that can +build itself using the result to build Linux From Scratch (version 6.8). +We also aim to replace Android's Toolbox.

+ +

At a secondary level we'd like to meet other use cases. We've analyzed +the commands provided by similar projects (klibc, sash, sbase, s6, embutils, +nash, and beastiebox), along with various vendor configurations of busybox, +and some end user requests.

+ +

Finally, we'd like to provide a good replacement for the Bash shell, +which was the first program Linux ever ran and remains the standard shell +of Linux no matter what Ubuntu says. This doesn't mean including the full +set of Bash 4.x functionality, but does involve {various,features} beyond +posix.

See the status page for the combined list and progress towards implementing it.

+ +

Use case: standards compliance.

@@ -203,6 +224,7 @@ rm mkdir rmdir reboot getevent sendevent date wipe sync umount start stop notify cmp dmesg route hd dd df getprop setprop watchprops log sleep renice printenv smd chmod chown newfs_msdos netstat ioctl mv schedtop top iftop id uptime vmstat nandread ionice touch lsof md5 r +cp du grep watchdogd

If selinux is enabled, you also get:

@@ -259,13 +281,15 @@ bespoke code to install itself.

For reference, combining everything listed above, we get:

-alarm ash cat chcon chmod chown cmp date dd df dmesg exists fs_mgr getenforce -getevent getprop getsebool gpttool hd id ifconfig iftop init insmod ioctl +alarm ash cat chcon chmod chown cmp cp date dd df dmesg du exists fs_mgr +getenforce +getevent getprop getsebool gpttool grep hd id ifconfig iftop init insmod ioctl ionice kill ln load_policy log logcat logwrapper ls lsmod lsof lsusb md5 mkbootimg mkdir mount mv nandread netcfg netstat newfs_msdos notify printenv ps r readtty reboot renice restorecon rm rmdir rmmod rotatefb route run-as runcon schedtop sdcard sendevent setconsole setenforce setkey setprop setsebool -sleep smd start stop sync syren top touch umount uptime vmstat watchprops wipe +sleep smd start stop sync syren top touch umount uptime vmstat watchdogd +watchprops wipe

We may eventually implement all of that, but for toybox 1.0 we need to @@ -277,12 +301,12 @@ functionality, which we can implement a shim interface for later).

This means toybox should implement:

-cat chmod chown cmp date dd df dmesg getevent getprop hd id ifconfig iftop -insmod ioctl ionice kill ln log logcat logwrapper ls lsmod lsof lsusb md5 mkdir -mount mv nandread +cat chmod chown cmp cp date dd df dmesg du getevent getprop grep hd id ifconfig +iftop insmod ioctl ionice kill ln log logcat logwrapper ls lsmod lsof lsusb md5 +mkdir mount mv nandread netstat newfs_msdos notify printenv ps r reboot renice rm rmdir rmmod route schedtop sendevent setconsole setprop sleep smd start stop sync top touch -umount uptime vmstat watchprops wipe +umount uptime vmstat watchprops watchdogd wipe
@@ -290,8 +314,8 @@ umount uptime vmstat watchprops wipe sections of this analysis:

-cat chmod chown cmp date dd df dmesg id ifconfig insmod kill ln ls lsmod -mkdir mount mv ps renice rm rmdir rmmod route sleep sync top touch umount +cat chmod chown cmp cp date dd df dmesg du grep id ifconfig insmod kill ln ls +lsmod mkdir mount mv ps renice rm rmdir rmmod route sleep sync top touch umount

Which leaves the following commands as new from Toolbox:

@@ -299,9 +323,311 @@ mkdir mount mv ps renice rm rmdir rmmod route sleep sync top touch umount
getevent getprop hd iftop ioctl ionice log lsof nandread netstat newfs_msdos notify printenv r reboot schedtop sendevent setconsole -setprop smd start stop top uptime vmstat watchprops wipe +setprop smd start stop top uptime vmstat watchprops watchdogd wipe +
+ +
 +

kblic:

+ +

Long ago some kernel developers came up with a project called +klibc. +After a decade of development it still has no web page or HOWTO, +and nobody's quite sure if the license is BSD or GPL. It inexplicably +requires perl to build, and seems like an ideal candidate for +replacement.

+ +

In addition to a C library even less capable than bionic (obsoleted by +musl), klibc builds a random assortment of executables to run init scripts +with. There's no multiplexer command, these are individual executables:

+ +

+cat chroot cpio dd dmesg false fixdep fstype gunzip gzip halt ipconfig kill +kinit ln losetup ls minips mkdir mkfifo mknodes +mksyntax mount mv nfsmount nuke pivot_root poweroff readlink reboot resume +run-init sh sha1hash sleep sync true umount uname zcat +

+ +

To get that list, build klibc according to the instructions (I +looked at version +2.0.2 and did cd klibc-*; ln -s /output/of/kernel/make/headers_install +linux; make) then echo $(for i in $(find . -type f); do file $i | grep -q +executable && basename $i; done | grep -v '[.]g$' | sort -u) to find +executables, then eliminated the *.so files and *.shared duplicates.

+ +

Some of those binaries are build-time tools that don't get installed, +which removes mknodes, mksyntax, sha1hash, and fixdep from the list. +(And sha1hash is just an unpolished sha1sum anyway.)

+ +

The run-init command is more commonly called switch_root, nuke is just +"rm -rf -- $@", and minips is more commonly called "ps". I'm not doing aliases +for the oddball names.

+ +

Yet more stale forks of dash and gzip sucked in here (see "dubious +license terms" above), adding nothing to the other projects we've looked at. +But we still need sh, gunzip, gzip, and zcat to replace this package.

+ +

By the time I did the analysis toybox already had cat, chroot, dmesg, false, +kill, ln, losetup, ls, mkdir, mkfifo, readlink, rm, switch_root, sleep, sync, +true, and uname.

+ +

The low hanging fruit is cpio, dd, ps, mv, and pivot_root.

+ +

The "kinit" command is another gratuitous rename, it's init running as PID 1. +The halt, poweroff, and reboot commands work with it.

+ +

I've got mount and umount queued up already, fstype and nfsmount go with +those. (And probably smbmount and p9mount, but this hasn't got one. Those +are all about querying for login credentials, probably workable into the +base mount command.)

+ +

The ipconfig command here has a built in dhcp client, so it's ifconfig +and dhcpcd and maybe some other stuff.

+ +

The resume command is... weird. It finds a swap partition and reads data +from it into a /proc file, something the kernel is capable of doing itself. +(Even though the klibc author +attempted +to remove that capability from the kernel, current kernel/power/hibernate.c +still parses "resume=" on the command line). And yet various distros seem to +make use of klibc for this> +Given the history of swsusp/hibernate (and +TuxOnIce +and kexec jump) I've lost track +of the current state of the art here. Ah, Documentation/power/userland-swsusp.txt +has the API docs, and here's a better +tool...

+ +

So the list of things actually in klibc are:

+ +
+ +cat chroot dmesg false kill ln losetup ls mkdir mkfifo readlink rm switch_root +sleep sync true uname + +cpio dd ps mv pivot_root +mount nfsmount fstype umount +sh gunzip gzip zcat +kinit halt poweroff reboot +ipconfig +resume +
+
+ +

Stand-Alone Shell

+ +

Wikipedia has a good +summary of sash, with links. The original Stand-Alone Shell project reached +a stopping point, and then "sash plus +patches" extended it a bit further. The result is a megabyte executable +that provides 40 commands.

+ +

Sash is a shell with built-in commands. It doesn't have a multiplexer +command, meaning "sash ls -l" doesn't work (you have to go "sash -c 'ls -l'"). +

+ +

The list of commands can be obtained via building it and doing +"echo help | ./sash | awk '{print $1}' | sed 's/^-//' | xargs echo", which +gives us:

+ +
+alias aliasall ar cd chattr chgrp chmod chown cmp cp chroot dd echo ed exec +exit file find grep gunzip gzip help kill losetup losetup ln ls lsattr mkdir +mknod more mount mv pivot_root printenv prompt pwd quit rm rmdir setenv source +sum sync tar touch umask umount unalias where +
+ +

Plus sh because it's a shell. A dozen or so commands can only sanely be +implemented as shell builtins (alias aliasall cd exec exit prompt quit setenv +source umask unalias), where is an alias for which, and at triage time toybox +already has chgrp, chmod, chown, cmp, cp, chroot, echo, help, kill, losetup, +ln, ls, mkdir, mknod, printenv, pwd, rm, rmdir, sync, and touch.

+ +

This leaves:

+ +
+ +ar chattr dd ed file find grep gunzip gzip lsattr more mount mv pivot_root +sh sum tar umount + +
+ +

(For once, this project doesn't include a fork of gzip, instead +it sucks in -lz from the host.)

+ +
+ +

sbase:

+ +

It's on suckless. So far it's +implemented:

+ +

+ +basename cat chmod chown cksum cmp cp date dirname echo false fold grep head +kill ln ls mc mkdir mkfifo mv nl nohup pwd rm seq sleep sort tail tee test +touch true tty uname uniq wc yes + +

+ +

And has a TODO list:

+ +

+ +cal chgrp chvt comm cut df diff du env expand expr id md5sum nice paste +printenv printf readlink rmdir seq sha1sum split sync test tr unexpand unlink +who + +

+ +

At triage time, of the first list I still need to do: fold grep mc mv nl. Of +the second list: diff expr paste printf split test tr unexpand who.

+ +
+ +

s6

+ +

The website skarnet has a bunch +of small utilities as part of something called "s6". This includes the +s6-portabile-utils +and the s6-linux-utils. +

+ +

Both packages rely on multiple bespoke external libraries without which +they can't compile. The source is completely uncommented and doesn't wrap at +80 characters. Doing a find for *.c files brings up the following commands:

+ +
+ +basename cat chmod chown chroot clock cut devd dirname echo env expr false +format-filter freeramdisk grep halt head hiercopy hostname linkname ln +logwatch ls maximumtime memoryhog mkdir mkfifo mount nice nuke pause +pivotchroot poweroff printenv quote quote-filter reboot rename rmrf sleep +sort swapoff swapon sync tail test touch true umount uniquename unquote +unquote-filter update-symlinks + +
+ +

Triage: memoryhog isn't even listed on the website nor does it have +a documentation file, clock seems like a subset +of date, devd is some sort of netlink wrapper that spawns its command line +every time it gets a message (maybe this is meant to implement part of +udev/mdev?), format-filter is sort of awk's '{print $2}' function split out +into its own command, hiercopy a subset of "cp -r", maximumtime is something +I implemented as a shell script (more/timeout.sh in Aboriginal Linux), +nuke isn't the same as klibc (this one's "kill SIG -1" only with hardwared +SIG options), pause is a program that literally waits to be killed (I +generally sleep 999999999 which is a little over 30 years), +pivotchroot is a subset of switch_root, rmrf is rm -rf...

+ +

I see "nuke" resurface, and if "rmrf" wasn't also here I might think +klibc had a point. + +

+basename cat chmod chown chroot cut dirname echo env expr false +freeramdisk grep halt head hostname linkname ln +logwatch ls mkdir mkfifo mount nice +pivotchroot poweroff printenv quote quote-filter reboot rename sleep +sort swapoff swapon sync tail test touch true umount uniquename unquote +unquote-filter update-symlinks +
+ + +
+ +

nash:

+ +

Red Hat's nash was part of its "mkinitrd" package, replacement for a shell +and utilities on the boot floppy back in the 1990's (the same general idea +as BusyBox, developed independently). Red Hat discontinued nash development +in 2010, replacing it with dracut (which collects together existing packages, +including busybox).

+ +

I couldn't figure out how to beat source code out of +Fedora's current git +repository. The last release version that used it was Fedora Core 12 +which has a source rpm +that can be unwound with "rpm2cpio mkinitrd.src.rpm | cpio -i -d -H newc +--no-absolute-filenames" and in there is a mkinitrd-6.0.93.tar.bz2 which +has the source.

+ +

In addition to being a bit like a command shell, the nash man page lists the +following commands:

+ +

+access echo find losetup mkdevices mkdir mknod mkdmnod mkrootdev mount +pivot_root readlink raidautorun setquiet showlabels sleep switchroot umount +

+ +

Oddly, the only occurrence of the string pivot_root in the nash source code +is in the man page, the command isn't there. (It seems to have been removed +when the underscoreless switchroot went in.)

+ +

A more complete list seems to be the handlers[] array in nash.c:

+ +

+access buildEnv cat cond cp daemonize dm echo exec exit find kernelopt +loadDrivers loadpolicy mkchardevs mkblktab mkblkdevs mkdir mkdmnod mknod +mkrootdev mount netname network null plymouth hotplug killplug losetup +ln ls raidautorun readlink resume resolveDevice rmparts setDeviceEnv +setquiet setuproot showelfinterp showlabels sleep stabilized status switchroot +umount waitdev +

+ +

This list is nuts: "plymouth" is an alias for "null" which is basically +"true" (which thie above list doesn't have). Things like buildEnv and +loadDrivers are bespoke Red Hat behavior that might as well be hardwired in +to nash's main() without being called.

+ +

Instead of eliminating items +from the list with an explanation for each, I'm just going to cherry pick +a few: the device mapper (dm, raidautorun) is probably interesting, +hotplug (may be obsolete due to kernel changes that now load firmware +directly), and another "resume" ala klibc.

+ +

But mostly: I don't care about this one. And neither does Red Hat anymore.

+ +
+ +

Beastiebox

+ +

Back in 2008, the BSD guys vented some busybox-envy +on sourceforge. Then stopped. +Their repository is still in CVS, hasn't been touched in years, it's a giant +hairball of existing code sucked together. (The web page says the author +is aware of crunchgen, but decided to do this by hand anyway. This is not +a collection of new code, it's a katamari of existing code rolled up in a +ball.)

+ +

Combining the set of commands listed on the web page with the set of +man pages in the source gives us:

+ +

+[ cat chmod cp csh date df disklabel dmesg echo ex fdisk fsck fsck_ffs getty +halt hostname ifconfig init kill less lesskey ln login ls lv mksh more mount +mount_ffs mv pfctl ping poweroff ps reboot rm route sed sh stty sysctl tar test +traceroute umount vi wiconfig +

+ +

Apparently lv is the missing link ed and vi, copyright 1982-1997 (do not +want), ex is another obsolete vi mode, lesskey is "used to +specify a set of key bindings to be used with less", and csh is a shell they +sucked in, [ is an alias for test. Several more bsd-isms that don't have Linux +equivalents (even in the ubuntu "install this package" search) are +disklabel, fsck_ffs, mount_ffs, and pfctl. And wiconfig is a wavelan interface +network card driver utility. Subtracting all that and the commands toybox +already implements at triage time, we get:

+ +

+ +fdisk fsck getty halt ifconfig init kill less mksh more mount mv ping poweroff +ps reboot route sed sh stty sysctl tar test traceroute umount vi + +

+ +

Not a hugely interesting list, but eh.

+

Requests:

diff --git a/www/status.html b/www/status.html index 2e9e5732..709efc0c 100755 --- a/www/status.html +++ b/www/status.html @@ -3,18 +3,19 @@

How are we doing on implementing stuff so far?

-

Legend: {android} [posix] (development) <lsb> other implemented

+

Legend: [posix] <lsb> (development) {android} =klibc= #sash# @sbase@ +*beastiebox* +request+ other implemented

-

The current status of toybox (as of 0.4.0 release):

+

The current status of toybox (as of 0.4.1 release):

These commands are reasonably finished:

basename cal cat catv chgrp chmod chown chvt cksum clear cmp comm count -df dirname dmesg dos2unix echo env false gethostname killall link logname ls -lsmod md5sum mkdir +df dirname dmesg dos2unix echo env false gethostname killall link logname +losetup ls lsmod md5sum mkdir mkfifo mkswap mktemp nice nohup od oneit pwd realpath rev rm seq setsid sha1sum sleep sort swapoff swapon sync tac taskset tee true truncate tty uniq unix2dos unlink usleep wc which whoami yes @@ -32,7 +33,7 @@ rmmod switch_root tail touch uname unshare uptime vconfig vmstat w who xargs

Work on these is underway, but not usable yet:

-bzip2 mke2fs more mount umount losetup sed tar sh grep/egrep/fgrep +bzip2 mke2fs more mount umount sed tar sh grep/egrep/fgrep

See the todo list for details.

-- cgit v1.2.3