A 2-post collection

Bash on Windows

So with the Windows 10 Anniversary update came the "Bash on Windows" feature. It is still beta and this really shows. So let me tell you of my experience getting it to work.

How to setup?

There are many blog posts for this but it all boils down to one single command on the Windows command prompt:

 LxRun.exe /install /y

It will then download a very minimal version of Ubuntu from the Windows Store (yes it really displays this). After this you can run bash on the command prompt or search the start menu for "Bash on Ubuntu on Windows" (wow what a name!)

What will work and what will not

So the complete subsystem starts and stops with the bash.exe running which literally means if you close the bash window everything is killed with it. So forget running any daemons in the background.

Microsoft itself said that running server software is not supported and is currently not planned to be supported. If you run your server daemon in the foreground in a bash window it will be accessible on the localhost network though. Part of this is probably because neither Upstart nor systemd could be run on the kernel interface that is available currently. There is no process 1, there is no init. Imagine that bash is running a emergency system repair console on a failed linux boot and you'll understand what is available right now.

You can see the complete Linux kernel interface is thought for developers. You will have access to the local disk, you will have access to the local network, but that's it primarily.

It's a bit sad that Microsoft decided to use Ubuntu 14.04 instead of the 2016 version, but i suppose this is just a symptom of "We need to get it running first" and will be updated at some later date.

How to get a decent terminal emulator

The first thing after getting it to work for me was to get rid of the Windows terminal emulator (if you want to call it that). Don't get me wrong they really put effort into upgrading it to support at least part of the VT100 spec, but copy and paste and mouse support aren't there yet, which is a no-go for me. So how can we get a better emulator on top of it?

Of course you could use what everyone on Windows seems to use: ConEmu. You'll have to put some time into configuring it correctly (get my config here: Github gist) but it even has a bash profile (which is thought for cygwin bash but works for the native bash too ;) )

ConEmu Screenshot

On the other hand you could do it the Unix way: Install an X-Server and run let's say lxterminal which is rather easy to do too. I used lxterminal as an example because I confirmed it works, urxvt for example does not as it requests a PTY from the kernel and this is currently not supported on Windows.

  1. Get an X-Server (like VCXsrv) installed

  2. Run it ;)

  3. Install lxterminal on the Linux subsystem:

     sudo apt-get install lxterminal
  4. Find the Bash on Ubuntu on Windows executable in C:\Windows\System32\bash.exe and create a shortcut to that

  5. Open the properties window of that shortcut and add -c "DISPLAY=:0 lxterminal" to the Target

Properties Window

  1. Start the Shortcut and do not close the cmd window (the one with the Ubuntu logo) but instead minimize it.

  2. Use the lxterminal window

LX Terminal

If you install an X-Server this way you can even use most GUI programs from the Ubuntu repositories, just remember to export DISPLAY=:0 before running any GUI tool.

Beware: There is no hardware acceleration as all drawing commands go through the loopback network interface (It should be faster than RDP though), so better do not run something like Firefox or Chromium on that.

How to get rid of it again

This is as easy as installation. To remove everything, including your home dir:

lxrun /uninstall /full

To just remove the Ubuntu installation:

lxrun /uninstall

xhyve: a quick how to

So here we are, xhyve finally usable, so how to use it?

Fetching the source and compiling it

To compile xhyve you need either Xcode or at least the Xcode command line tools from Apple. If you not already have those installed run the following in a Terminal window:

xcode-select --install

Now you're ready to checkout the source code and compile it:

git clone https://github.com/dunkelstern/xhyve.git xhyve
cd xhyve
git checkout sparse-disk-image

If everything worked the last lines of the make output should look something like this:

ld xhyve.sym
dsym xhyve.dSYM
strip xhyve

If you want you could install the xhyve binary to /usr/local/bin now or just copy it somewhere where you'll find it:

sudo mkdir -p /usr/local/{bin,share/man/man1}
sudo cp build/xhyve /usr/local/bin/
sudo cp xhyve.1 /usr/local/share/man/man1/

(It will ask for your password when using sudo, if you have Homebrew installed you might skip the sudo)

The first config file

A xhyve config file is a simple shell script, as xhyve takes all configuration in it's command line parameters.

What's possible

The best option to see what xhyve actually can do is the man page, but I will describe the basics to you nonetheless.

Open the man page with man ./xhyve.1 while you're in the source directory or just with man xhyve if you copied the binary and man-page into /usr/local.

So what can xhyve emulate:


Command line parameter: -A

This is an option that should be set all the time except if you get weird ACPI related crashes.

Number of CPUs

Command line parameter: -c <number>

How many CPU cores you want to share with the guest operating system, defaults to one, maximum is 16.

Allocated RAM

Command line parameter: -m <number>

Amount of RAM to give to the guest operating system. You may use suffixes like k, m or g for Kilobytes, Megabytes or Gigabytes.

Virtual COM ports

Command line parameter: -l <source-device>,<destination-device>

This maps a virtual COM port (serial port) to a host device.
<source-device> may be either com1 or com2 and <destination-device> may be either stdio (to connect the serial port to the terminal window's console) or any character device in /dev/.

Virtual PCI bus

Command line parameter: -s <slot>,<emulation>,[config]

This is a bit more complex as it defines which hardware is connected to your VM.

<slot> is a number from 0 to 31 that defines the PCI slot.

<emulation> is the device to emulate. xhyve knows the following devices:

  • hostbridge, this is usually needed at slot 0 for most guests
  • virtio-net, a virtual network card
  • virtio-blk, a virtual block device (a disk)
  • ahci-cd, a virtual CD-ROM/DVD-ROM drive
  • ahci-hd, a virtual disk for guests that have no virtio driver
  • uart, a virtual serial port (COM3+)
  • lpc, a PCI to ISA bridge, used for COM1 and COM2

Each of these has a specific configuration that may be set, see the man page for further instructions, an example of the usage follows.

Linux kernel to boot

Command line parameter: -f kexec,<kernel>,<initrd>,<kernel command line>

This essentially specifies which linux kernel to load. The problem with xhyve is that it has no BIOS or EFI emulation, so it loads the Linux kernel directly, this leads to a small inconvenience: We need the Kernel outside of the disk image.

What's needed

So this is all nice and good but which of these command line flags do we really need to get a standard Ubuntu running?

  • -A, for ACPI mode
  • -m 1G, 1GB RAM
  • -s 0,hostbridge -s 31,lpc, basically the minimum PCI config that works
  • -l com1,stdio, map the first serial port to the Terminal window
  • -s 1,ahci-cd,ubuntu-15.10-server-amd64.iso, the CD-ROM with the Ubuntu ISO inserted
  • -s 2,virtio-blk,hdd.img,sectorsize=4096,size=20G,split=1G,sparse, the main disk, 20GB max size, split into 1GB parts, use a sector size of 4096 (best choice for SSDs) and do not eat my harddisk space (sparse)
  • -f kexec,vmlinuz,initrd.gz,"earlyprintk console=ttyS0", load a linux kernel from the files vmlinuz and initrd.gz the kernel parameters tell Ubuntu to map the console to the first serial port so we can see the boot process in the Terminal.

All in one (save as install.sh):

# Linux
CMDLINE="earlyprintk=serial console=ttyS0"

# Guest Config
MEM="-m 1G"
IMG_CD="-s 1,ahci-cd,ubuntu-15.10-server-amd64.iso"
IMG_HDD="-s 2,virtio-blk,hdd.img,size=20G,split=1G,sparse"
NET="-s 3,virito-net,vmnet0"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"

# and now run

As you might have noticed, we run the xhyve binary as root this is because the xhyve binary has to be code signed with an Apple Developer Key or run as root to use the networking infrastructure. Let's just install Ubuntu and think about that later.

Installing ubuntu

As there is no graphical output we have to install Ubuntu Server (as only the Server version has the text-mode installer).

Go get it here: http://www.ubuntu.com/download/server

As you have read in the What's possible section, we need a Linux Kernel outside of our disk images to boot. So let's get one.

Extracting the setup kernel

As the Ubuntu install CD contains all files we need, why not just extract it from the disk image? Well there's one catch: Ubuntu uses a so called Mixed-Mode CD image and Mac OS X doesn't really like to mount such a disk so we have to resort to a small trick, execute the following on a Terminal to get the Kernel:

# create a 2k temporary file filled with zeroes
dd if=/dev/zero bs=2k count=1 of=tmp.iso

# append the ubuntu server image to that
dd if=ubuntu-15.10-server-amd64.iso bs=2k skip=1 >> tmp.iso

# mount it
hdiutil attach tmp.iso

# copy needed files
cp /Volumes/Ubuntu-Server\ 15/install/vmlinuz .
cp /Volumes/Ubuntu-Server\ 15/install/initrd.gz .

# unmount it
hdiutil detach /Volumes/Ubuntu-Server\ 15

You may want to keep tmp.iso or as well delete it as we have what we want. (Thanks to the people at pagetable for this neat trick)

Running the setup

If you saved the All in one script from above, make it executable with chmod a+x install.sh and run it.

If everything worked Ubuntu should boot in your Terminal.
Do not finish the setup yet, as you'll need to do something before:

Extracting the system kernel

When you finished setup you'll need to exchange which Kernel to load. The easiest thing to do that is by opening a console in the ubuntu installer. If it asks you if you want to finish the installation and reboot, don't do that but return to the main installer menu, there's an option Open a console that we need now.

After being dumped into the console make the installed target your change root:

chroot /target
cd /boot

Now open a second Terminal window on your Mac and run the following:

nc -l -p 1234 | tar x

This starts a netcat server that listens on the port 1234 for a connection from the Ubuntu guest and expects to receive a tar file.

Now on the Ubuntu guest execute the following:

tar cv vmlinuz* initrd.img* | nc <ip_of_mac> 1234

If everything worked you should have two files on your Mac:

  • vmlinuz-4.2.0-16-generic
  • initrd.img-4.2.0-16-generic

(The version number may differ)

Now finish the setup and let Ubuntu reboot. Rebooting should exit xhyve and dump you to your Mac console. This is correct!

Boot config

Now copy install.sh to boot.sh and make the following changes:

  • replace the files for INITRD and KERNEL with those you extracted from the VM
  • add root=/dev/vda1 to the Kernel command line after console=ttyS0
  • You may comment out the IMG_CD line

In future, to boot Ubuntu, you'll only need to run boot.sh and it will just work.