aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2019-07-31Move the empty regex workaround into xregcomp.Elliott Hughes
No current caller except grep needs this, but consistency seems like a good idea. Also change the xregcomp error message to be a bit more human-readable, rather than mention an implementation detail.
2019-07-31Fix error path and tweak comments.Rob Landley
2019-07-31man: switch to xregcomp.Elliott Hughes
All regcomp callers are now going through xregcomp.
2019-07-30Fix signed typecast bug.Rob Landley
We use (char *)1 and (char *)2 to indicate errors (they can never be valid pointers because both malloc() and mmap() return aligned memory and those align down to NULL, plus Linux maps 4k at the bottom to catch null dereferences anyway), and then typecast it to long (trusting in LP64) to do an integer <=2 comparison... except that needs to be UNSIGNED long or else pointers in the top half of the virtual memory space become negative and the <=2 false positives them as errors. (Oops.)
2019-07-29grep: fake GNU behavior for non-POSIX empty regex.Elliott Hughes
POSIX says there's no such thing as an empty regular expression. The grammar excludes the possibility: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html BSD agrees with POSIX, and Android and macOS' BSD-based implementations reject the empty regular expression. GNU apparently disagrees. Luckily, BSD does accept the empty *sub* expression `()`, despite their error message for REG_EMPTY being "empty (sub)expression". This is presumably a bug, except there's explicit code to support it that is at least 26 years old: https://github.com/freebsd/freebsd/blame/master/lib/libc/regex/regcomp.c#L383 This workaround also works fine with glibc. If we want GNU behavior, I'm struggling to come up with another way to fake it. If we want POSIX behavior, we could easily just add a check to reject "" on glibc. Also switch to xregcomp().
2019-07-29Add ASAN=1 to the build system.Elliott Hughes
Just use `ASAN=1 make test_grep` or whatever. You'll probably want to set $ASAN_SYMBOLIZER_PATH to point to llvm-symbolizer, but Debian makes that annoying by calling the symbolizer /usr/bin/llvm-symbolizer-4.0 or whatever, and ASan refuses to use it: ==43370==ERROR: External symbolizer path is set to '/usr/bin/llvm-symbolizer-4.0' which isn't a known symbolizer. Please set the path to the llvm-symbolizer binary or other known tool. My usual workaround for this is to drop an llvm-symbolizer symlink in the current directory, and I'm happy to automate that in the script to make it require no knowledge of any of this nonsense, but haven't done so in this initial patch. I tested that this is actually working by reverting the grep fix and running `ASAN=1 make test_grep`.
2019-07-27losetup.test: bail out if loopback devices are already in use.Elliott Hughes
There's probably a way to filter out the existing ones, but it's likely to be pretty painful and isn't immediately necessary. (I hit this case on Android cloud x86 devices.)
2019-07-27modinfo.test: bail out early if there are no modules.Elliott Hughes
We need /proc/modules *and* a directory of modules.
2019-07-27ifconfig.test: actually skip tests if dummy0 up fails.Elliott Hughes
Rather than just claim we're doing so. Android cloud emulators don't have dummy0 like regular devices do.
2019-07-26truncate.test: work around ext extended attributes space usage.Elliott Hughes
Weaken the test slightly so that the extra 4KiB used for SELinux is ignored. Unfortunately all the du tests suffer the same problem, and I'm less sure what to do there. Mounting a read-only file system without the extended attributes might be the best idea? But since this test modifies the file system...
2019-07-26hostname: fix tests for non-root on Android.Elliott Hughes
On Android you can't even get the hostname unless you're root, to avoid device fingerprinting/tracking. Also modernize by using skipnot, which fixes the incorrect test name in the SKIP: output.
2019-07-25killall: implement -w.Elliott Hughes
This isn't currently used in AOSP, but it is used in some of the other codebases I can easily search, and it's a better fix for test flakiness than inserting a sleep.
2019-07-25killall.test: fix flakiness.Elliott Hughes
And when I say "fix", I mean "hack around in the worst possible way". Android+mksh is often slow enough that we can do the pgrep before the killed process has actually exited. (I'm seeing roughly 100% failure rate on cloud x86 emulators, and about 75% on a real Pixel 2.)
2019-07-24tests: fix for empty /etc/passwd or /etc/group.Elliott Hughes
It turns out some Android devices have an empty /etc/passwd and/or /etc/group, which was defeating the previous workaround. Switch to testing the intention more directly: we'll try the file in /etc, and if that didn't work, we'll assume we need a workaround.
2019-07-24Avoid double-close of fd in loopfiles_lines.Elliott Hughes
Test: no EBADF in `strace -e close ./toybox tac /proc/version`
2019-07-24crontab: switch to getline().Elliott Hughes
2019-07-24Rewrite tac to make better use of lib functions.Rob Landley
2019-07-24tac: switch to getline().Elliott Hughes
2019-07-23nl: switch from getline() to loopfiles_lines().Elliott Hughes
This was basically just to help me think out loud while discussing loopfiles_lines().
2019-07-22Start replacing get_line() with getline().Elliott Hughes
I started this last night, but thought I'd aim to send multiple small patches rather than work through all the callers and send one big patch. I've deliberately chosen the ugly name `allocated_length` because we've had historical bugs where folks think this a line length in the sense of the return value. I do wonder whether we should actually have some kind of getline() wrapper that hides the `char *`/`size_t` pair in lib/, which makes the function easier to use in most cases but does add the less common gotcha that you wouldn't be able to getline() through multiple files at once (which does happen in at least one toy). But maybe the real fix is to look harder for places where we can just use loopfiles_lines? Speaking of which, should we actually add two more arguments to that? Specifically: switch it to getdelim() rather than getline() behind the scenes, and also add a way to have the trailing '\n' automatically removed, since that seems to be what most callers want? Anyway, that seemed like enough questions that it was time to send this initial patch out before doing too much more...
2019-07-22test.tar: fix tar tests on Android.Elliott Hughes
There's no /etc/group on Android. Loop devices are in different places, but that test doesn't seem to offer anything beyond the block/char special device tests later, so remove that rather than fix it. Fix the block/char special tests (the expectations were missing, but they're never normally run on the host because the mknod fails). Likewise add the missing expectation to the "ownership" test, which isn't normally run on the host because the chown fails. The "sparse without overflow" test was broken because of the mksh/bash disagreement about whether in `VAR=val shell_function` that's a temporary assignment to VAR or one that outlives the command. Switch SUM to being a function like LST before it. Also rewrite the TARHD stuff because Android (a) doesn't have hd and (b) doesn't have process substitution in its shell. (I wish I'd noticed TARHD earlier --- I'd been manually sending the tar files to xxd for debugging instead!) With this patch, all the tar tests now pass for me on both the device and host.
2019-07-22Next round of plumbing evolution. Left in the todo and debug code this time.Rob Landley
2019-07-20Update design page to answer somebody's question.Rob Landley
2019-07-19xargs: don't pretend to support -I.Elliott Hughes
This was embarrassing... A build script that was using xargs -I silently started to ignore -I when we switched over to toybox xargs. The owner of the script has rewritten it to use the shell read builtin instead, but it's pretty unhelpful to silently ignore an option that radically changes how xargs behaves. (The -I behavior sounds sufficiently different from normal behavior that I'm not sure I've understood, and am just sending this cleanup rather than actually implementing -I correctly.) Bug: http://b/137832162
2019-07-17ifconfig: fix for non-root on Android.Elliott Hughes
The change from ioctl to xioctl in 844f0d1aae28c54bbb672f53ba2cda182c572129 broke ifconfig for non-root on Android. SIOCGIFHWADDR has been protected to prevent device tracking since Nougat. Bug: https://issuetracker.google.com/137671590
2019-07-16Fix unaligned access, tweak test suite.Rob Landley
2019-07-16grep: fix two bugs found by hwasan.Elliott Hughes
The first bug appeared as a memory overwrite, but was actually visible without hwasan: basically any `grep -F` that let to multiple matches on the same line was broken. The second bug was another memory overwrite, visible when I ran the existing grep tests. Bug: http://b/137573082
2019-07-16ip.c route change support for extended attributes.Mike Bennett
This fixes the extended attribute settings for the ip route change command; extended attributes are collected in a separate buffer and added to the RTM message in a specific RTA_METRICS attribute. Only added setter support for the RTAX_MTU (per-route MTU), as that was the only option currently parsed. Also added support for displaying extended attributes, even those not currently settable.
2019-07-13net.c: add missing static.Elliott Hughes
Pointed out during code review of the recent refactor.
2019-07-12diff: implement --strip-trailing-cr.Elliott Hughes
Used by some ART tests and also some LLVM tests. (The motivating example is the latter, but I noticed the former when looking for other users.) Bug: http://b/137298656
2019-07-12pidof: fix default behavior, add -x.Elliott Hughes
Before this patch, we're effectively doing `pidof -x` all the time. This patch changes names_to_pid() to allow us to say whether or not we want to include scripts, and adjusts the callers appropriately. Also add tests for `pidof` versus `pidof -x` which pass after this patch, without regressing the existing killall tests.
2019-07-12grep: add -R as well as -r.Elliott Hughes
On BSD these are actually the same, and there's a -S that you need in addition. So strictly this is a behavior change for Android (which is going from BSD grep to toybox grep), but it's a behavior preserving change for the AOSP build (which is going from GNU grep to toybox grep), and the latter actually has a checked-in use of -R where the former doesn't.
2019-07-11Next sh checkpoint. Not remotely load bearing yet.Rob Landley
2019-07-11bind/connect cleanup.Elliott Hughes
Rename the existing xbind/xconnect to xbindany/xconnectany, to make room for new xbind/xconnect that are more like 'x' versions of the regular bind and connect. Move explicit bind/connect callers over to xbind/xconnect. Of the affected commands, only netcat is actually used by Android. It was the most recent patch to netcat that made the lack of a more traditional xbind/xconnect apparent.
2019-07-11xargs: bug compatibility with BSD/busybox/findutils.Elliott Hughes
The other xargs implementations in the wild don't seem to count the space taken by each `char *`. Go for bug compatibility unless future experience proves that to be a bad idea.
2019-07-11env.test: fix new test for toybox echo.Elliott Hughes
The test relied on the non-builtins not being toybox (or another multicall binary). A multicall binary will actually look at the symlinked name (in this case "true") and do what it says on the tin. Use a tiny shell script instead.
2019-07-10dd: iflags, oflags, fix ^C, fix the fundamental loop.Elliott Hughes
Investigating why the toybox tar tests fail on Android with toybox dd, I realized I was rewriting a part of dd I'd rewritten before! This is a re-send of my 2019-02-22 patch, rebased against the current ToT... This patch was originally motivated because after suggesting to the author of https://stackoverflow.com/questions/17157820/access-vdsolinux/54797221#54797221 that he could tell dd to work in bytes rather than blocks, I realized that our dd doesn't actually support that. But the rewrite of the main loop is necessary to fix the incorrect output from the dd calls in the tar test. Without this patch, `yes | dd bs=65536 count=1 > fweep` basically gives random output, based on how many bytes the pipe feels like giving you in your first read. (As far as I know, dd *without* bs= was fine, but I can't guarantee that that's true, just that I haven't seen it fail.) Also switch to TAGGED_ARRAY and comma_* for conv rather than add two more copies of an undesired idiom. It turned out -- contrary to the belief of cp(1) -- that comma_scan isn't suitable for this because of its magic handling of "no" prefixes. (It's actually harmless in cp because none of the --preserve options begin with "no", but some dd options do.) To this end, comma_remove is a less-magic comma_scan. I've also changed an `if` to a `while` because other implementations allow things like `--preserve=mode,mode` or `conv=sync,sync`. (If we decide this is a bug rather than a feature, we should at least fix the error message to be clear that we're rejecting the *duplication*, not the option itself.) I've also fixed the ^C behavior by simply adding a direct SIGINT handler rather than trying to be clever inside the read loop (which is why we weren't handling the SIGINT until the read returned). I've also removed `strstarteq` and just added the '=' to each literal when calling regular `strstart`. Plus basic tests.
2019-07-10Add dlist_lpop() to remove last entry (use dlist as stack).Rob Landley
2019-07-09dd: fix seek= on stdout.Elliott Hughes
2019-07-06file: fix xz, add a few other Android types.Elliott Hughes
"xz compressed data" was missing a newline. Also switch a couple of other unnecessary (but correct) xprintf calls over to xputs. Also add .otf fonts, perf/simpleperf data files, and Android boot images, sparse images, and device tree blobs/overlays.
2019-07-06killall: better handling of long names.Elliott Hughes
Change names_to_pid() so that we can actually match shell scripts with long names (the code to get the shell script's name was correct, but there was an extra test preventing us from actually comparing it to the sought name). In kill.c itself, remove a dead test for -l and switch to the FLAG() macro. Also extend the tests to explicitly cover long and short names.
2019-07-06ifconfig.test: disable the pointopoint tests.Elliott Hughes
These tests don't work for me as root on either my Debian desktop or my Android devices. The original mail thread implies that they were as close as the original comitter could get to _something_ that seemed to work, even if the kernel doesn't seem to bother with this: http://lists.landley.net/pipermail/toybox-landley.net/2014-November/003795.html toybox ifconfig *is* missing support for `-pointtopoint` and `pointopoint` (without an addess), similar for `broadcast` according to the man page. But since we don't appear to have a way to test this (other than looking at strace output!) I'm leaning towards YAGNI anyway...
2019-07-06Add comment and test.Rob Landley
2019-07-06Have env exec, not recurse to builtin.Denys Nykula
My app runs npm i -S better-sqlite3, and node-gyp-bin from the npm upstream has #!/usr/bin/env sh, expecting the system shell.
2019-07-03nl: nl -vEric Molitor
Implement nl -v supporting negative and zero starting values.
2019-07-01netcat: Add UNIX domain socket support (-U)Tom Cherry
Other implementations of netcat support UNIX domain sockets via -U, so this change adds it to toybox too. Example usage: Create ./uds as a UNIX domain socket and listen on it, forwarding messages to bash: > nc -U -s ./uds -L /bin/bash Connect to ./uds as a UNIX domain socket > nc -U ./uds
2019-06-29More toysh flow control plumbing.Rob Landley
2019-06-29Bash Compatibility PatchEric Molitor
I know that you are working on toysh which I'm looking forward to. In the meantime below is a patch to improve compatibility with older/odd versions of bash. This fixed a minor build issue I was having on MacOS (which was using zsh in emulated bash mode) as well as an oddball embedded SDK using a non-gnu version of bash. I believe these changes are minimal and should be safe to apply, if not I wanted to at least get them on the mailing list in case others ran into these issues.
2019-06-28modinfo: various small fixes.Elliott Hughes
Improve Android support (see code comments). Remove unnecessary fixed-length limits. Show error if module not found (plus test). Expand help text. Switch to FLAG macro. Stop hard-coding module assumptions in the tests.
2019-06-28split.test: don't rely on bash process substitution.Elliott Hughes
mksh doesn't support process substitution, so use an explicit temporary file.