At Randori, our automated attack platform emulates a realistic adversary. Real adversaries will often take advantage of targets of opportunity discovered on a network, even if those targets are not directly related to an objective. Embedded devices or appliances make attractive targets because they are seldom supported by endpoint vendors, often run outdated or unpatched software, and, in many cases, run a well-known and documented (often Linux-based) operating system, even if the firmware is proprietary.
Why spend time compromising HVAC controllers, desktop telephones, or water pumps? These devices don’t usually have very sensitive information, and their compromise would, at first, seem like more of an annoyance than a threat or business risk. While these are usually low-power, low-information devices, if compromised, they do provide an attacker with control over a valuable point of presence in the network.
Once we gain access to a point of presence, our attack toolkit is able to turn most devices into generic data link layer remote access devices, which enable our platform to tunnel its reconnaissance, exploitation, and exfiltration traffic as if we were plugged directly into the network, side by side with the compromised device. There have been several occasions where the platform has discovered additional network access though seldom examined channels. How would your router handle the traffic if the “isolated” HVAC port on the Facilities VLAN suddenly started receiving traffic pre-tagged with the Accounting VLAN ID? Have you tested for that? We have, on more than one occasion, pivoted access to more valuable devices through this type of (mis)configuration.
One of the challenges with utilizing the embedded devices we find attached to our target networks is building our tools to run correctly in such diverse environments. You may have heard of the “Android fragmentation” problem that mobile developers face when supporting a diversity of hardware across device manufacturers and installed OS versions. Our challenge is similar, except that we need to support not just ARM and x86 devices, but also MIPS, PowerPC, and others. We encounter not just the six latest versions of Android, but also dozens of variants and often customized kernels – sometimes forked from kernels from decades ago (e.g., Linux 2.4 kernel, which we support, was released in 2001).
Luckily for us, Linux is highly backward compatible. By building our tools with the oldest toolchains we need, we can maintain support for the broadest set of platforms. Obtaining very old hardware with build environments or setting up full system emulators and doing “remote” builds is possible, but it can be a lot of work to troubleshoot and is complicated to reliably scale and automate. So, how do we set up these old toolchains for such a broad range of architectures?
The simple and effective technique we use is to rely on QEMU for userland emulation, and Debian (debootstrap) to install some very old userlands. Once we have all the tools we need in a userland with the correct architecture, we package it up into a Docker container and deploy it to our build environment.
The first component in our solution is QEMU, an emulator, which, in one mode, works in conjunction with a Linux kernel feature called binfmt. The binfmt feature allows the Linux kernel to invoke a custom interpreter for a binary based on a file signature. Similar, in some respects, to the behavior when an executable script is run and the #! line is invoked to interpret it, the kernel will match a binary’s signature (including the CPU type specified in the ELF header), and invoke QEMU to emulate it. In this way an ARM binary can “run” on an x86 CPU, for example, with QEMU translating the instructions and syscalls on the fly. Debian includes packages for both components. After running these commands you will see a list of binary signatures your machine can now emulate/interpret.
apt install qemu-user-static binfmt-support debootstrap pv
update-binfmts --display Next, we need to bootstrap an old Debian userland to give us the toolchain. We’ve found that Debian Sarge works well for this purpose, and its GCC toolchain supports building static binaries for kernels as old as 2.2.0 (depending on the architecture), which is old enough for our purposes.
#!/bin/bash -x
ARCH=arm
REL=sarge
RDIR=3.1_r8
RTAG=31r8
ISO=debian-$RTAG-$ARCH-netinst.iso
mkdir -p $REL/$ARCH/rootfs/iso # make a dir for the iso download
mkdir -p $REL/$ARCH/rootfs/usr/bin # make a dir for the qemu static binary
mkdir -p $REL/$ARCH/iso # make a dir for the iso mount point
cd $REL/$ARCH/rootfs/iso
wget https://cdimage.debian.org/mirror/cdimage/archive/$RDIR/$ARCH/iso-cd/$ISO # download the iso
cd ../..
mount -o loop,ro rootfs/iso/$ISO iso # mount the iso
cp /usr/bin/qemu-$ARCH-static rootfs/usr/bin/
/usr/sbin/debootstrap --arch $ARCH --no-check-gpg $REL rootfs/ file://`pwd`/iso/debian # bootstrap sarge into the rootfs dir
echo "deb http://archive.debian.org/debian $REL main" > rootfs/etc/apt/sources.list # overwrite the default sources list
rm -rf rootfs/var/lib/apt # clean up for apt
mkdir -p rootfs/var/lib/apt/lists/partial
chroot rootfs apt-get update # update from the archive
chroot rootfs apt-get install --force-yes -y gcc make libc6-dev gdb strace screen tcpdump vim file
umount iso
This script results in a Sarge (Debian 3.1) userland capable of running on modern Linux kernels thanks to QEMU, but much of our build system uses Docker containers and it would be nicer to provide a container to make post-build cleanup as easy as running docker kill. Docker can be installed by following the instructions here:
https://docs.docker.com/engine/install/debian/ Taking it one step further we can create a Docker container with our root filesystem:
(tar cf - -C $REL/$ARCH/rootfs . | pv -s $(du -sb $REL/$ARCH/rootfs | awk '{print $1}')) | docker import - randori-build-$ARCH-$RELWe now have a container named randori-build-arm-sarge. Since the QEMU binary is loaded outside the container, we have qemu-user-static and binfmt-support installed on the host where we run it. Let’s try it out!
$ uname -mo
x86_64 GNU/Linux
$ docker run --rm -it randori-build-arm-sarge bash
# uname -mo
armv7l GNU/Linux
# echo -e '#include<stdio.h>\nint main(){printf("hello world.\\n");return 0;}' > main.c && gcc -static main.c -o a.out
# file a.out
a.out: ELF 32-bit LSB executable, ARM, version 1 (ARM), for GNU/Linux 2.2.0, statically linked, not stripped
# ./a.out
hello world.
Forget digging up old hardware or full system emulation (though you might want to test ;)), Debian, QEMU, and Docker make building for turn-of-the-century Linuxes on modern hardware easy!
Bonus: When installing the qemu-user-static package, binfmt support is registered for each of the QEMU static binaries (one for each supported architecture). Besides using the update-binfmts tool, it is possible to examine the registered binfmts via the proc filesystem:
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: OCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
The “F” (fix-binary) flag directs the kernel to preload the interpreter prior to entering the mount namespace for the container (and is default on Debian). If we re-register the binfmt without the “F” flag, or if running on a host where this feature isn’t supported (older kernels or other distributions), we can run the QEMU binary in the container instead of the one on the host. The script above copies the static binary into the container so the container works in either case.
Let us know what other build tips you have or tricks you use or would want to see more of on Twitter @RandoriAttack. We look forward to sharing more adversarial tips, techniques, and PoCs here as we continue to enhance the capabilities of our automated attack platform.
To read more posts like this from the Randori Attack Team, check out their ongoing TTP series (Tools, Techniques & POCS)
Eric McIntyre is the Director of R&D at Randori and a member of the Randori Attack Team. Previously, he served as a security researcher at Kyrus Tech, Inc., where he worked for 6 years leading teams of software engineers and other researchers to develop novel solutions to unique software challenges in the national and commercial security spaces. Before joining Kyrus, Eric operated a software development firm, EM Technology, and was an adjunct faculty member at the University of Colorado-Boulder. At CU-Boulder, he was a lecturer for undergraduate electrical engineers and, as a postgraduate researcher, conducted NASA-sponsored research primarily in the field of sub-orbital passive microwave radiometry.
You can follow him on twitter at @pwnpnw
