Ubuntu as a bhyve guest

My home router/server runs FreeBSD and recently got a CPU upgrade that allows for some more advanced services. I don’t have any machine running Linux at this point but I currently need one for some tasks. That’s why I thought setting up an Ubuntu virtual machine on the server would be neat. Luckily, FreeBSD 10 comes with the neat bhyve virtualization solution.

Prerequisites

Load the kernel extension and create a tap interface for the 192.168.2.1/24 subnet to use with the virtual machines:

# kldload vmm
# sysctl net.link.tap.up_on_open=1
# ifconfig tap0 create
# ifconfig tap0 inet 192.168.2.1/24

Add the following rules in the /etc/pf.conf to enable access to the tap network and NAT to the outside world:

tap_if = "tap0"
nat on $ext_if from $tap_if:network to any -> ($ext_if)
pass in on tap0 keep state

Reload PF rules:

# pfctl -f /etc/pf.conf

Create a new ZFS file system in device mode that will be used as the hard disk for the VM:

# zfs create -V256G -o volmode=dev pool/ubuntu_vm

The next step is to download an Ubuntu net installer image. This is required as Ubuntu will be installed without a graphical interface.

Install the linux boot loader grub and create a device.map for the grub bootloader:

# pkg install grub2-bhyve
# cat device.map 
(hd0) /dev/zvol/pool/ubuntu_vm
(cd0) ./mini.iso

Installation

Run grub and when prompted select the text based installation option:

grub-bhyve -r cd0 -m device.map -M 1024 ubuntu

Start the VM and follow the instructions. When prompted, manually configure the IP address to e.g. 192.168.2.2.

# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/dev/zvol/pool/ubuntu_vm -s 4,ahci-cd,./mini.iso -l com1,stdio -c 2 -m 1024M ubuntu

When the installation is done and the VM terminates, it has to be manually destroyed:

# bhyvectl --destroy --vm=ubuntu

Starting the VM

Create a grub input file, run grub and run bhyve:

# cat grub.in 
set root=(hd0,msdos1)
linux /vmlinuz root=/dev/vda1
initrd /initrd.img
boot

# grub-bhyve -r hd0 -m device.map -M 1024 ubuntu < grub.in > /dev/null
# bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/dev/zvol/pool/ubuntu_vm -l com1,stdio -c 2 -m 1024M ubuntu

Now would be a good time to install the sshd in ubuntu so it will also be accessible without having a console attached to the VM.

Making things persistent

To ensure that the changes to the system are persistent, the following steps are necessary:

# echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf

Add the following to /boot/loader.conf:

vmm_load="YES"                                                                                                                                                 
if_tap_load="YES"

Add the following to /etc/rc.conf:

cloned_interfaces="tap0"
ifconfig_tap0="inet 192.168.2.1/24"

Sources

[1] [2] [3]