From a4314f2da0cd501593083c29b02777bd9ac3f3b5 Mon Sep 17 00:00:00 2001 From: Rob Landley Date: Mon, 14 Oct 2019 15:43:47 -0500 Subject: Freshen up first half of roadmap. --- www/roadmap.html | 195 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 105 insertions(+), 90 deletions(-) diff --git a/www/roadmap.html b/www/roadmap.html index f26f24c3..ba155dbb 100755 --- a/www/roadmap.html +++ b/www/roadmap.html @@ -6,12 +6,14 @@

We have several potential use cases for a new set of command line utilities, and are using those to determine which commands to implement -for Toybox's 1.0 release.

+for Toybox's 1.0 release. (Most of these have their own section in the +status page.)

-

The most interesting standards are POSIX-2008 (also known as the Single +

The most interesting publicly available standards are POSIX-2008 +(also known as the Single Unix Specification version 4) and the Linux Standard Base (version 4.1). -The main test harness including toybox in Aboriginal Linux and if that can -build itself using the result to build Linux From Scratch (version 6.8). +The main test harness is including toybox in Aboriginal Linux and if that can +build itself using the result to build Linux From Scratch (version 6.8). We also aim to replace Android's Toolbox.

At a secondary level we'd like to meet other use cases. We've analyzed @@ -22,7 +24,7 @@ and some end user requests.

Finally, we'd like to provide a good replacement for the Bash shell, which was the first program Linux ever ran and remains the standard shell of Linux no matter what Ubuntu says. This doesn't mean including the full -set of Bash 4.x functionality, but does involve {various,features} beyond +set of Bash 4.x functionality, but does involve {various,features} <(beyond) posix.

See the status page for the combined list @@ -46,34 +48,33 @@ and progress towards implementing it.

Use case: standards compliance.

POSIX-2008/SUSv4

-

The best standards are the kind that describe reality, rather than -attempting to impose a new one. (I.E. a good standard should document, not -legislate.)

- -

The kind of standards which describe existing reality tend to be approved by -more than one standards body, such ANSI and ISO both approving C. That's why +

The best standards describe reality rather than attempting to impose a +new one. A good standard should document, not legislate. +Standards which document existing reality tend to be approved by +more than one standards body, such ANSI and ISO both approving C. That's why the IEEE POSIX committee's 2008 standard, the Single Unix Specification version 4, and the Open Group Base Specification edition 7 are all the same standard -from three sources.

- -

The "utilities" -section -of these standards is devoted to the unix command line, and are the best such -standard for our purposes. (My earlier work on BusyBox was implemented with -regard to SUSv3, an earlier version of this standard.)

+from three sources, but most people just call it "posix" (portable operating +system derived from unix). It's available online in full, and may be downloaded as a tarball.

Problems with the standard

-

Unfortunately, these standards describe a subset of reality, lacking any -mention of commands such as init, login, or mount required to actually boot a -system. It provides ipcrm and ipcs, but not ipcmk, so you can use System V IPC -resources but not create them.

- -

These standards also contain a large number of commands that are -inappropriate for toybox to implement in its 1.0 release. (Perhaps some of -these could be reintroduced in later releases, but not now.)

- -

Starting with the full "utilities" list, we first remove generally obsolete +

Unfortunately posix describes an incomplete subset of reality, +lacking any mention of commands such as init or mount required to +actually boot a system. It describes logname but not login. It provides ipcrm +and ipcs, but not ipcmk, so you can use System V IPC resources but not create +them. And widely used real-world commands such as tar and cpio (the basis +of initramfs and RPM) which were present in earlier +versions of the standard have been removed, while obsolete commands like +cksum, sccs and uucp remain with no mention of modern counterparts like +crc32/sha1sum, svn/git or scp/rsync. Meanwhile the commands themselves +are missing dozens of features and specify silly things like ebcdic +support in dd or that wc should use %d (not %ld) for byte counts. So +we have to extensively filter posix to get a useful set of recommendations.

+ +

Starting with the +full "utilities" list, +we first remove generally obsolete commands (compess ed ex pr uncompress uccp uustat uux), commands for the pre-CVS "SCCS" source control system (admin delta get prs rmdel sact sccs unget val what), fortran support (asa fort77), and batch processing support (batch @@ -88,22 +89,16 @@ revisited later, but not for toybox 1.0.)

separate executables (alias bg cd command fc fg getopts hash jobs kill read type ulimit umask unalias wait). These may be revisited as part of a built-in toybox shell, but are not exported into $PATH via symlinks. (If you fork a -child process and have it "cd" then exit, you've accomplished nothing. -This is not a complete list, a shell also needs exit, if, while, for, case, -export, set, unset, trap, exec... And for bash compatability, function and -source.)

- -
- -alias bg cd command fc fg getopts hash jobs kill read type ulimit umask -unalias wait exit if while for case export set unset trap exec function source - -
+child process and have it "cd" then exit, you've accomplished nothing.) +Again, what posix provides is incomplete: a shell also needs exit, if, while, +for, case, export, set, unset, trap, exec... (And for bash compatibility +function, source...)

A few other commands are judgement calls, providing command-line internationalization support (iconv locale localedef), System V inter-process communication (ipcrm ipcs), and cross-tty communication from the minicomputer -days (talk mesg write). The "pax" utility was supplanted by tar, "mailx" is +days (talk mesg write). The "pax" utility failed to replace tar, +"mailx" is a command line email client, and "lp" submits files for printing to... what exactly? (cups?) The standard defines crontab but not crond.

@@ -134,9 +129,10 @@ a truck through the holes and declare themselves compilant. But it means what they DID standardize tends to be respected (if sometimes obsolete).

The Linux Standard Base's failure mode is different, they respond to -pressure by including special-case crap, such as allowing Red Hat to shoehorn +pressure by including anything their members pay them enough to promote, +such as allowing Red Hat to push RPM into the standard even though all sorts of distros (Debian, Slackware, Arch, -Gentoo) don't use it and probably never will. This means anything in the LSB is +Gentoo) don't use it and never will. This means anything in the LSB is at best a suggestion: arbitrary portions of this standard are widely ignored.

@@ -176,7 +172,7 @@ for examples.)

Since we've already committed to using our own judgement to skip bits of POSIX, and LSB's "judgement" in this regard is purely bug workarounds to declare various legacy tool implementations "compliant", this means we're mostly -interested in the set of tools that aren't specified in posix at all.

+interested in the set of LSB tools that aren't mentioned in posix.

Of these, gettext and msgfmt are internationalization, install_initd and remove_initd weren't present in Ubuntu 10.04, lpr is out of scope, @@ -197,14 +193,9 @@ su sync tar umount useradd userdel usermod zcat

Use case: provide a self-hosting development environment

-

The following commands are enough to build the Aboriginal Linux development +

The following commands were enough to build the Aboriginal Linux development environment, boot it to a shell prompt, and build Linux From Scratch 6.8 under it.

-

This use case includes running init scripts and other shell scripts, running -configure, make, and install in each package, and providing basic command line -facilities such as a text editor. (It does not include a compiler toolchain or -C library, those are outside the scope of this project.)

-
bzcat cat cp dirname echo env patch rmdir sha1sum sleep sort sync @@ -221,57 +212,77 @@ resize2fs tune2fs fsck.ext2 genext2fs mke2fs xzcat
-

Aboriginal Linux also installs bash 2.05b as #!/bin/sh and its scripts -require bash extensions not present in shells such as busybox ash. -This means that toysh needs to supply several bash extensions _and_ work +

This use case includes running init scripts and other shell scripts, running +configure, make, and install in each package, and providing basic command line +facilities such as a text editor. (It does not include a compiler toolchain or +C library, those are outside the scope of the toybox project, although mkroot +has a potentialy follow-up project. +For now we use distro toolchains, +musl-cross-make, +and the Android NDK for build testing.) +That build system also instaled bash 2.05b as #!/bin/sh and its scripts +required bash extensions not present in shells such as busybox ash. +To replace that toysh needs to supply several bash extensions _and_ work when called under the name "bash".

-

The Aboriginal Linux -self-bootstrapping build still uses the following busybox commands, -not yet supplied by toybox:

+

The development methodology used a command logging wrapper +that intercepted each command called out of the $PATH and append the +command line to a log file, then analyze the result to create a +list of commands, +then create a directory of symlinks pointing to those commands out of the +host $PATH. Then the new implementation can replace these commands one +at a time, checking the results and the log output to spot any behavior +changes.

+ +

Stages and moving targets

-

-awk bunzip2 bzcat dd diff expr fdisk ftpd ftpget -ftpput gunzip gzip less ping route sh -sha512sum tar test tr unxz vi wget xzcat zcat -

+

This use case has two stages: 1) building a bootable system that can +rebuild itself from source, and 2) a build environment capable +of bootstrapping up to arbitrary complexity (as exemplified by building +Linux From Scratch and Beyond Linux From Scratch under the resulting +system). To accomplish just the first goal, the old build +still needs the following busybox commands for which toybox does not yet +supply adequate replacements:

+ +
+awk dd diff expr fdisk ftpd gzip less route sh sha512sum tr unxz vi wget xzcat +
+ +

All of those except awk, ftpd, and less have partial implementations +in "pending".

-

Many of those are in "pending". The remaining "difficult" -commands are vi, awk, and sh.

+

In 2017 Aboriginal Linux development ended, replaced by the +mkroot project +designed to use an existing cross+native toolchain (such as +musl-cross-make +or the Android NDK) instead of building its own. In 2019 the still-incomplete +mkroot was merged into toybox as the "make root" target. This is intended +as a simpler way of providing essentially the same build environment, and doesn't +significantly affect the rest of this analysis (although the "rebuild itself +from source" test now includes building musl-cross-make under either mkroot +or toybox's "make airlock" host environment).

Building Linux From Scratch is not the same as building the Android Open Source Project, -but after toybox 1.0 focus may shift to modifying the AOSP build +but after toybox 1.0 we plan to try +modifying the AOSP build to reduce dependencies. (It's fairly likely we'll have to add at least a read-only git utility so repo can download the build's source code, but that's actually not that hard. We'll probably also need our own "make" at some point after -1.0.)

- -

Note: Aboriginal Linux was -restarted (I.E. replaced) due to toolchain licensing issues, but the new project is a simpler implementation of the same goals, -and when finished should provide the same test environment for toybox. -This section needs to be rewritten whem mkroot is complete enough to build -Linux From Scratch.

- -

The development methodology used a command logging wrapper -that intercepted each command called out of the $PATH and append the -command line to a log file, then analyze the result to create a -list of commands, -then create a directory of symlinks pointing to those commands out of the -host $PATH. Then the new implementation can replace these commands one -at a time, checking the results and the log output to spot any behavior -changes.

+1.0, which is its own moving target thanks to cmake and ninja and so on.) +The ongoing Android hermetic build work is already advancing +this goal.

Use case: Replacing Android Toolbox

Android has a policy against GPL in userspace, so even though BusyBox predates Android by many years, they couldn't use it. Instead they grabbed -an old version of ash and implemented their own command line utility set -called "toolbox". ash was later replaced by -mksh; toolbox is being -replaced by toybox.

+an old version of ash (later replaced by +mksh) +and implemented their own command line utility set +called "toolbox" (which toybox has already mostly replaced).

Toolbox doesn't have its own repository, instead it's part of Android's system/core @@ -325,26 +336,30 @@ binaries in /system/bin are:

  • traceroute/traceroute6 - trace network route (iputils)
    • -

    The names in parentheses are the source.

    +

    The names in parentheses are the upstream source of the command.

    Analysis

    -

    For reference, combining everything listed above, we get:

    +

    For reference, combining everything listed above that's still "fair game" +for toybox, we get:

    arping blkid e2fsck dd fsck.f2fs fsck_msdos getevent gzip ip iptables -ip6tables iw logwrapper make_ext4fs make_f2fs newfs_msdos ping ping6 +ip6tables iw logwrapper make_ext4fs make_f2fs modpobe newfs_msdos ping ping6 reboot resize2fs sh ss tc tracepath tracepath6 traceroute traceroute6

    We may eventually implement all of that, but for toybox 1.0 we need to -focus a bit. For our first pass, let's just replace all the "toolbox" -commands.

    +focus a bit. If Android has an acceptable external package, and the command +isn't needed for system bootstrapping, replacing the external package is +not a priority.

    -

    This means toybox should implement (or finish implementing):

    +

    However, several commands toybox plans to implement anyway could potentially +replace existing Android versions, so we should take into account Android's use +cases when doing so. This includes:

    -dd getevent gzip newfs_msdos +dd getevent gzip modprobe newfs_msdos sh
    -- cgit v1.2.3