xhyve: a quick how to

So here we are, xhyve finally usable, so how to use it?

Fetching the source and compiling it

To compile xhyve you need either Xcode or at least the Xcode command line tools from Apple. If you not already have those installed run the following in a Terminal window:

xcode-select --install

Now you're ready to checkout the source code and compile it:

git clone https://github.com/dunkelstern/xhyve.git xhyve
cd xhyve
git checkout sparse-disk-image

If everything worked the last lines of the make output should look something like this:

ld xhyve.sym
dsym xhyve.dSYM
strip xhyve

If you want you could install the xhyve binary to /usr/local/bin now or just copy it somewhere where you'll find it:

sudo mkdir -p /usr/local/{bin,share/man/man1}
sudo cp build/xhyve /usr/local/bin/
sudo cp xhyve.1 /usr/local/share/man/man1/

(It will ask for your password when using sudo, if you have Homebrew installed you might skip the sudo)

The first config file

A xhyve config file is a simple shell script, as xhyve takes all configuration in it's command line parameters.

What's possible

The best option to see what xhyve actually can do is the man page, but I will describe the basics to you nonetheless.

Open the man page with man ./xhyve.1 while you're in the source directory or just with man xhyve if you copied the binary and man-page into /usr/local.

So what can xhyve emulate:


Command line parameter: -A

This is an option that should be set all the time except if you get weird ACPI related crashes.

Number of CPUs

Command line parameter: -c <number>

How many CPU cores you want to share with the guest operating system, defaults to one, maximum is 16.

Allocated RAM

Command line parameter: -m <number>

Amount of RAM to give to the guest operating system. You may use suffixes like k, m or g for Kilobytes, Megabytes or Gigabytes.

Virtual COM ports

Command line parameter: -l <source-device>,<destination-device>

This maps a virtual COM port (serial port) to a host device.
<source-device> may be either com1 or com2 and <destination-device> may be either stdio (to connect the serial port to the terminal window's console) or any character device in /dev/.

Virtual PCI bus

Command line parameter: -s <slot>,<emulation>,[config]

This is a bit more complex as it defines which hardware is connected to your VM.

<slot> is a number from 0 to 31 that defines the PCI slot.

<emulation> is the device to emulate. xhyve knows the following devices:

  • hostbridge, this is usually needed at slot 0 for most guests
  • virtio-net, a virtual network card
  • virtio-blk, a virtual block device (a disk)
  • ahci-cd, a virtual CD-ROM/DVD-ROM drive
  • ahci-hd, a virtual disk for guests that have no virtio driver
  • uart, a virtual serial port (COM3+)
  • lpc, a PCI to ISA bridge, used for COM1 and COM2

Each of these has a specific configuration that may be set, see the man page for further instructions, an example of the usage follows.

Linux kernel to boot

Command line parameter: -f kexec,<kernel>,<initrd>,<kernel command line>

This essentially specifies which linux kernel to load. The problem with xhyve is that it has no BIOS or EFI emulation, so it loads the Linux kernel directly, this leads to a small inconvenience: We need the Kernel outside of the disk image.

What's needed

So this is all nice and good but which of these command line flags do we really need to get a standard Ubuntu running?

  • -A, for ACPI mode
  • -m 1G, 1GB RAM
  • -s 0,hostbridge -s 31,lpc, basically the minimum PCI config that works
  • -l com1,stdio, map the first serial port to the Terminal window
  • -s 1,ahci-cd,ubuntu-15.10-server-amd64.iso, the CD-ROM with the Ubuntu ISO inserted
  • -s 2,virtio-blk,hdd.img,sectorsize=4096,size=20G,split=1G,sparse, the main disk, 20GB max size, split into 1GB parts, use a sector size of 4096 (best choice for SSDs) and do not eat my harddisk space (sparse)
  • -f kexec,vmlinuz,initrd.gz,"earlyprintk console=ttyS0", load a linux kernel from the files vmlinuz and initrd.gz the kernel parameters tell Ubuntu to map the console to the first serial port so we can see the boot process in the Terminal.

All in one (save as install.sh):

# Linux
CMDLINE="earlyprintk=serial console=ttyS0"

# Guest Config
MEM="-m 1G"
IMG_CD="-s 1,ahci-cd,ubuntu-15.10-server-amd64.iso"
IMG_HDD="-s 2,virtio-blk,hdd.img,size=20G,split=1G,sparse"
NET="-s 3,virito-net,vmnet0"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"

# and now run

As you might have noticed, we run the xhyve binary as root this is because the xhyve binary has to be code signed with an Apple Developer Key or run as root to use the networking infrastructure. Let's just install Ubuntu and think about that later.

Installing ubuntu

As there is no graphical output we have to install Ubuntu Server (as only the Server version has the text-mode installer).

Go get it here: http://www.ubuntu.com/download/server

As you have read in the What's possible section, we need a Linux Kernel outside of our disk images to boot. So let's get one.

Extracting the setup kernel

As the Ubuntu install CD contains all files we need, why not just extract it from the disk image? Well there's one catch: Ubuntu uses a so called Mixed-Mode CD image and Mac OS X doesn't really like to mount such a disk so we have to resort to a small trick, execute the following on a Terminal to get the Kernel:

# create a 2k temporary file filled with zeroes
dd if=/dev/zero bs=2k count=1 of=tmp.iso

# append the ubuntu server image to that
dd if=ubuntu-15.10-server-amd64.iso bs=2k skip=1 >> tmp.iso

# mount it
hdiutil attach tmp.iso

# copy needed files
cp /Volumes/Ubuntu-Server\ 15/install/vmlinuz .
cp /Volumes/Ubuntu-Server\ 15/install/initrd.gz .

# unmount it
hdiutil detach /Volumes/Ubuntu-Server\ 15

You may want to keep tmp.iso or as well delete it as we have what we want. (Thanks to the people at pagetable for this neat trick)

Running the setup

If you saved the All in one script from above, make it executable with chmod a+x install.sh and run it.

If everything worked Ubuntu should boot in your Terminal.
Do not finish the setup yet, as you'll need to do something before:

Extracting the system kernel

When you finished setup you'll need to exchange which Kernel to load. The easiest thing to do that is by opening a console in the ubuntu installer. If it asks you if you want to finish the installation and reboot, don't do that but return to the main installer menu, there's an option Open a console that we need now.

After being dumped into the console make the installed target your change root:

chroot /target
cd /boot

Now open a second Terminal window on your Mac and run the following:

nc -l -p 1234 | tar x

This starts a netcat server that listens on the port 1234 for a connection from the Ubuntu guest and expects to receive a tar file.

Now on the Ubuntu guest execute the following:

tar cv vmlinuz* initrd.img* | nc <ip_of_mac> 1234

If everything worked you should have two files on your Mac:

  • vmlinuz-4.2.0-16-generic
  • initrd.img-4.2.0-16-generic

(The version number may differ)

Now finish the setup and let Ubuntu reboot. Rebooting should exit xhyve and dump you to your Mac console. This is correct!

Boot config

Now copy install.sh to boot.sh and make the following changes:

  • replace the files for INITRD and KERNEL with those you extracted from the VM
  • add root=/dev/vda1 to the Kernel command line after console=ttyS0
  • You may comment out the IMG_CD line

In future, to boot Ubuntu, you'll only need to run boot.sh and it will just work.

xhyve: lightweight vm for Mac OS X

xhyve is a port of bhyve a qemu equivalent for FreeBSD to the Mac OS X Hypervisor framework. This means it has zero dependencies that are not already installed on every Mac that runs at least OS X Yosemite (10.10). The cool thing though is that Mac OS X has full control of the system all the time as no third party kernel driver hijacks the CPU when a VM is running, so the full power management that OS X provides is always in charge of everything. No more battery draining VMs \o/.

xhyve logo

xhyve is Open Source

This is really cool as everyone is able to hack it, so did I. The code is, like every bit of lowlevel C code I saw in my life, a bit complex to read and has not many comments but is very structured so you can easily find what you want to modify and do that.

The project is quite young so don't expect miracles. It has for example no graphics emulation. Running Ubuntu or FreeBSD is reduced to a serial tty and networking. If you want to run a virtual server on your Mac for development purposes it's quite perfect though.

There was one downer that got me: A virtual disk of say 30 GB has a backing file that is exactly 30 GB big even if you only store 400 MB on it. That's bad for everyone running on a SSD where space is limited as of now.

Introducing: Sparse HDD-Images for xhyve

Because the VM code is pretty small (the compiled executable of xhyve is about 230 KB) I though it might be possible for me to change this one last thing that prevented me to use xhyve on my Macbook. It turns out it is really easy to hack the virtual block device subsystem. All disk access code is neatly contained in one file: blockif.c. It is neatly separated from the virtio-block and ahci drivers.

So what I went out to do was three things:

  • Split the big disk image file into multiple segments (as for why read on)
  • Make the disk image segments only store blocks that have actual content in them (vs. storing only zeroes)
  • Make xhyve create the backing image files if they do not exist.

Splitting the disk image into segments

You may ask why, this is rather an optimization for maintaining speed and aid debugging but turned out to have the following advantages:

  • Some file systems may only allow files of a maximum size (prime example: FAT32 only allows 2GB per file)
  • Sparse image lookup tables can be filled with 32 bit values instead of defaulting to 64 bit (which saves 50% space in the lookup tables)
  • Debugging is easier as you may hexdump those smaller files on the terminal instead of loading a multi gigabyte file into the hex editor of your choice
  • Fragmentation of sparse images is reduced somewhat (probably not an issue for SSD backed files)
  • Growing disks is easy: just append a segment
  • Shrinking disks should be possible with help of the guest operating system, if it is able to clear the end of the disk of any data you could just delete a segment.

So splitting was implemented and rather easy to think of, just divide the disk read offset by the segment size and use a modulo operation to get to the in-segment-address. There's one catch: I had to revert from using preadv and pwritev to regular reads and writes. Usually you really want those v functions as they allow executing multiple read and write calls in one system call, thus beeing atomic. But these functions only work with one file descriptor and our reads probably span multiple segments and thus multiple file descriptors.

To make the thing easier and configurable I introduced two additional parameters for the block device configuration:

  • size the size of the backing file for the virtual disk
  • split the segment size. size should be a multiple of split to avoid wasting space.

You may use suffixes like k, m, g like on the RAM settings to avoid calculating the real byte sizes in your head ;)

Be aware: You may convert a disk from plain to split image either by using dd and splitting the image file (exact commands are left as an exercise to the reader) or by setting split to the old size of the image and size to a multiple of split effectively increasing the size of the disk by a multiple of the old size. New segments will be created automatically on next VM start.

Example config

Implementing sparse images

So the last step for making xhyve usable to me: Don't waste my disk space.

I think there are multiple methods for implementing efficient sparse images, I went for the following:

  • Only save sectors that contain actual data and not only zeroes
  • Minimum allocation size is one sector
  • Maintain a lookup table that references where each sector is saved in the image
  • Deallocation of a sector (e.g. overwriting with zeroes) is only handled by a shrink disk tool offline

So how does such a lookup table look?

A sparse disk lookup table is just an array of 32 bit unsigned integers, one for each sector. If you want to read sector 45 you just take the value of array position 45, multiply it by sector size and seek into the image segment to read from that address. Simple, isn't it?

In the current implementation the lookup table is written to a separate file with the extension .lut, all writes to this file are synchronous. The other backing files will be initially created as zero byte length files and when the guest os starts writing data the new sector is appended to the respective segment file and a new offset is written to the lookup table.

The lookup table starts as an array full of UINT32_MAX values (0xffffffff) as this is the marker used to describe that this sector is not yet in the image and thus should be returned as a series of zero values. If a read finds an entry other than that marker the corresponding data is read from the segment file.

All lookup tables for all segment files are appended to the .lut file, so it contains multiple tables, not just one. Positive side of this is that 32 bits of offset data map to a maximum segment size of about just under 4 TB divided by the sector size. If you use an SSD as backing storage you probably should configure your sector size to 4KB as that is the sector size of most SSDs and will result in additional performance. So this will result in a maximum segment size of about 16 PB and I never heard of a Mac that has this much storage. (If yours has please send me a photo)

Writes of new sectors (those appended to the segment file) are synchronous to avoid two sectors with the same address. Other writes are as the user configured them on the command line.

To enable sparse disk images just add sparse as a parameter to your configuration.

Sparse config example

Be aware: You'll have to recreate your disk image to profit of this setting, sparse disks are not compatible with normal disks.


I used this configuration:

-s 4,virtio-blk,test/hdd/hdd.img,sectorsize=4096,size=20G,split=1G,sparse

So this is how it looks on disk:

$ ls -lah
total 2785264
drwxr-xr-x  23 dark  staff   782B Jan 16 01:27 .
drwxr-xr-x   9 dark  staff   306B Jan 16 01:27 ..
-rw-rw----   1 dark  staff   531M Jan 16 02:50 hdd.img.0000
-rw-rw----   1 dark  staff    12K Jan 16 01:18 hdd.img.0001
-rw-rw----   1 dark  staff   5.9M Jan 16 02:50 hdd.img.0002
-rw-rw----   1 dark  staff    24K Jan 16 01:18 hdd.img.0003
-rw-rw----   1 dark  staff   110M Jan 16 02:48 hdd.img.0004
-rw-rw----   1 dark  staff     0B Jan 16 01:11 hdd.img.0005
-rw-rw----   1 dark  staff   172M Jan 16 02:50 hdd.img.0006
-rw-rw----   1 dark  staff     0B Jan 16 01:11 hdd.img.0007
-rw-rw----   1 dark  staff   207M Jan 16 02:50 hdd.img.0008
-rw-rw----   1 dark  staff     0B Jan 16 01:11 hdd.img.0009
-rw-rw----   1 dark  staff    15M Jan 16 02:48 hdd.img.0010
-rw-rw----   1 dark  staff     0B Jan 16 01:11 hdd.img.0011
-rw-rw----   1 dark  staff    46M Jan 16 01:24 hdd.img.0012
-rw-rw----   1 dark  staff     0B Jan 16 01:11 hdd.img.0013
-rw-rw----   1 dark  staff   151M Jan 16 02:50 hdd.img.0014
-rw-rw----   1 dark  staff    12K Jan 16 01:18 hdd.img.0015
-rw-rw----   1 dark  staff    97M Jan 16 02:50 hdd.img.0016
-rw-rw----   1 dark  staff   5.3M Jan 16 02:48 hdd.img.0017
-rw-rw----   1 dark  staff     0B Jan 16 01:11 hdd.img.0018
-rw-rw----   1 dark  staff   4.0K Jan 16 01:15 hdd.img.0019
-rw-rw----   1 dark  staff    20M Jan 16 02:48 hdd.img.lut

The dump you see here is of an image where I just installed Ubuntu server 15.10.
Instead of wasting 20GB of space this one only needs about 1.3GB. Speed is about the same as before (with a SSD as backing storage) but may suffer severely on a spinning rust disk as there are way more seeks if the sectors become fragmented.

Where to get it

Currently you will have to compile yourself, just fetch the sparse-disk-image branch from https://github.com/dunkelstern/xhyve and execute make.

Post mortem analysis of Swift server code

Just a very quick idea of how you could handle server side crashes of a Swift binary. Swift itself has no Stack unwinding functions that you could use for debugging purposes but lldb has.

So what if the currently crashing program would attach lldb to itself and create stack traces before vanishing into nirvana?

import Darwin

private func signalHandler(signal: Int32) {
    // need my pid for telling lldb to attach to parent
    let pid = getpid()
    // create command file
    let filename = "/tmp/backtrace.\(pid)"
    var fp = fopen(filename, "w")
    if fp == nil {
        print("Could not open command file")
    // attach to pid
    var cmd = "process attach --pid \(pid)\n"
    fputs(cmd, fp)
    // backtrace for all threads
    cmd = "bt all\n"
    fputs(cmd, fp)

//    // save core dump
//    cmd = "process save-core coredump\n"
//    fputs(cmd, fp)

    // kill the process
    cmd = "process kill\n"
    fputs(cmd, fp)
    // delete the command file
    cmd = "script import os\nscript os.unlink(\"\(filename)\")\n"
    fputs(cmd, fp)
    // quit lldb
    cmd = "quit\n"
    fputs(cmd, fp)

    // add signal type to backtrace.log header
    fp = fopen("backtrace.log", "w")
    if fp == nil {
        print("Could not open log file")
    fputs("Signal \(signal) caught, executing lldb for backtrace\n", fp)
    // run lldb
    let command = "/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/bin/lldb --file \"\(Process.arguments[0])\" --source \"\(filename)\" >>backtrace.log"

// Install signal handler
signal(SIGILL, signalHandler)
signal(SIGTRAP, signalHandler)
signal(SIGABRT, signalHandler)
signal(SIGSEGV, signalHandler)
signal(SIGBUS, signalHandler)
signal(SIGFPE, signalHandler)

// Now crash
print("Hello, World!")
var forcedUnwrap: Int! = nil


This code traps all fatal error signals and calls lldb with a small command file that looks like this and is generated by the crashing program:

process attach --pid <my_pid>
bt all
process kill
script import os
script os.unlink("<command_file>")

So it attaches lldb to a PID, fetches a backtrace for all threads, kills the parent process and deletes the command file. Log output is diverted to backtrace.log and contains all lldb output.

Additionally you could include process save-core coredump to write a core dump to the current directory which can be loaded for further inspection, but beware a core dump for our simple program up there will be around 500MB and will take about 30 seconds to write to disk (SSD).

You can load the core dump like this:

lldb -f <binary> -c coredump

Now you can inspect memory and variables like the program crashed while the debugger was attached.

The cool thing is, this code will not disable the Xcode internal debugger, so you get the usual EXC_BAD_INSTRUCTION when running the code in Xcode.

Useful Xcode breakpoints

Here I will document useful breakpoints when you're developing for OSX or iOS with Xcode. This is primarily for me to remember what is useful as I am googling some of these all the time.

It's sad that there are no "breakpoint-templates" that will automatically apply to all Xcode projects that you'll ever create. But enough of the introductory words, here comes the list:

Objective C exception breakpoint

Obviously the most important breakpoint there is, this one has a "template" of some sort as it has its very own menu entry:

Objective C Exception Breakpoint

So far so good, but the annoying thing is that the exception message will not be printed when the breakpoint is reached but only if you continue (and crash your program for real, and of course void your stack backtrace)

So add this to the default:

i386/iOS simulator (32Bit)

Objective C Exception Breakpoint modifications

Just add the following actions:

  • po *(id*)($esp+4)

which will print for example:

-[__NSCFConstantString characterAtIndex:]: Range or index out of bounds

x86_64/iOS simulator (64Bit)

Use the following actions:

  • po $rdx
  • po $rcx
  • po $r8

iOS device (ARM/ARM64)

Use these:

  • po $r2
  • po $r3
  • po *(id*)($sp)

If all this is to verbose to you look at the prebuild lldb script from here: http://qwan.org/2013/06/18/how-to-snatch-the-error-code-from-the-trap-frame-in-xcode/

Memory errors

Attention: This may be superseeded by the new Address sanitizer in Xcode 7

Sometimes it happens that you find a nasty memory error that is not at all obvious where it came from. Mostly it appears somewhere far away from the original error because the stack was smashed or a buffer overflow bleed into neighboring variables and overwrote something.

To find those errors you'll usually first enable all the memory protection error loggers that are available:

Memory debugging

And now you'll add a symbolic breakpoint at malloc_error_break to catch the offender that is smashing your stack right away (or at least very near to the cause)

Core Graphics Errors

All of Core Graphics error logging will go through the function CGPostError so it is sensible to add a breakpoint at exactly that location. It will break at anything Core Graphics will spit out. You know those 'XXX tried to draw to a nil context` errors may be hard to find if a lot is going on in your application at the same time.

Slow loading views

Perhaps because you used a wrong font-name? Who knows! Set a symbolic breakpoint to CTFontLogSuboptimalRequest to be notified when Core Text does not find the font directly but has to resort to a detailed search (that is very slow, on an iPad 4 it takes about 3 seconds to complete)

Auto layout

This one is obvious when it happens. But it's better to catch it directly instead of waiting for one to happen so for completeness, set a symbolic breakpoint to UIViewAlertForUnsatisfiableConstraints

Improve the debugger

Sometimes lldb can be quite stubborn:

(lldb) p self.window.bounds
error: property 'bounds' not found on object of type 'UIWindow *'
error: 1 errors parsing expression

But this can be fixed:

(lldb) expr @import UIKit
(lldb) p self.window.bounds
(CGRect) $4 = (origin = (x = 0, y = 0), size = (width = 375, height = 667))

Wow, much better, but what has this to do with breakpoints you may ask? Just try the following: Set a new breakpoint in your app delegate that is hit immediately after starting your application, set it to auto-continue and add an action expr @import UIKit.

The next run of your app will stop briefly at that breakpoint, execute that command and continue running. If you hit any other breakpoint (or press pause) the command already has been executed and lldb knows everything about UIKit already! Great!

(You may want to @import Foundation and @import CoreGraphics too)

Xcode build phases

This is just a short brain-dump what to put in each and every project's build phases:

Show all TODO: and FIXME: comments as warnings

Based on another blog article of Jeffrey Sambells, put this as a run script build phase into your build process:

echo "searching ${SRCROOT} for ${TAGS}"
find "${SRCROOT}/Source" \( -name "*.h" -or -name "*.m" \) -print0 | xargs -0 egrep --with-filename --line-number --only-matching "($TAGS).*\$" | sed -e 's/\([^ ]*[0-9][0-9]*:\)\(.*\)/\1 warning: \2/'

My changes include the doxygen @ tags and replace the invocation of perl with a better working version in sed script.

Automatically build documentation

Install the awesome appledoc to build better documentation than doxygen might do (all with the same comment markup ;) )

Best install not via homebrew but via the install-script in the git repo (as you will get no templates in a homebrew install)

Add a new custom target, name it documentation and add this script to the build phases:

cd "${SRCROOT}"
if [ -x /usr/local/bin/appledoc ] ; then
    /usr/local/bin/appledoc . 2>&1 |sed -e '/warning: Ignoring/d' >&2
    echo "error: appledoc not installed" >&2
    exit 1

You'll need a config-file named AppledocSettings.plist in your $(SRCROOT):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">