Prerequisite¶
Debian Jessie install with LVM and space left in a LVM volume group ( vg ) for the domUs.
Note: diffs in this howto are against files in the /etc directory.
Setup time keeping with ntp¶
$ sudo apt-get install ntp
Make sure it does not listen on all interfaces (i.e. so we get no traffic from domUs to this dom0 service)
--- a/default/ntp +++ b/default/ntp @@ -1 +1 @@ -NTPD_OPTS='-g' +NTPD_OPTS='-g -I eth0'
Remove NFS¶
If you're not going to use NFS, it is better to purge it
$ sudo apt-get purge rpcbind nfs-common
Disable IPv6¶
We must disable IPv6 or else the dom0 will have an IPv6 link-local address in each bridge for the domUs
--- a/default/grub +++ b/default/grub @@ -7,7 +7,7 @@ GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet" -GRUB_CMDLINE_LINUX="" +GRUB_CMDLINE_LINUX="ipv6.disable=1"
If you are running exim4 you need to disable IPv6 for exim as well, or you will get this in the paniclog
2015-03-13 21:01:33 IPv6 socket creation failed: Address family not supported by protocol
Use dpkg-reconfigure exim4-config do disable IPv6 in exim by removing ; ::1 from "IP-addresses to listen on for incoming SMTP connection"
Install the XEN system¶
$ sudo apt-get install xen-system-amd64 xen-tools bridge-utils
Dom0 mem config¶
Configure memory for dom0 and disable auto-ballooning for certain ;-)
Note: 1024M is space enough for the dom0 to be able to cache a bit, if you have less RAM in your system 512MB will be fine as well.
--- a/default/grub +++ b/default/grub @@ -9,6 +9,9 @@ GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet" GRUB_CMDLINE_LINUX="ipv6.disable=1" +# Xen boot parameters for all Xen boots +GRUB_CMDLINE_XEN="dom0_mem=1024M,max:1024M" + # Uncomment to enable BadRAM filtering, modify to suit your needs # This works with Linux (no patch required) and with any kernel that obtains # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...) diff --git a/xen/xl.conf b/xen/xl.conf index 374b6bb..3cd2902 100644 --- a/xen/xl.conf +++ b/xen/xl.conf @@ -3,7 +3,7 @@ # Control whether dom0 is ballooned down when xen doesn't have enough # free memory to create a domain. "auto" means only balloon if dom0 # starts with all the host's memory. -#autoballoon="auto" +autoballoon=0 # full path of the lockfile used by xl during domain creation #lockfile="/var/lock/xl"
Configure Grub to boot XEN first¶
Make sure grub loads the XEN hypervisor first, also make sure that grub does not list the domUs in the grub menu
Add this to /etc/default/grub
GRUB_DISABLE_OS_PROBER=true
$ sudo dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen $ sudo update-grub
Reboot domUs instead of saving¶
Disable save and restore of domUs (shutdown and restart instead)
--- a/default/xendomains +++ b/default/xendomains @@ -21,12 +21,12 @@ # (e.g. because you rather shut domains down). # If domain saving does succeed, SHUTDOWN will not be executed. # -XENDOMAINS_SAVE=/var/lib/xen/save +XENDOMAINS_SAVE="" # This variable determines whether saved domains from XENDOMAINS_SAVE # will be restored on system startup. # -XENDOMAINS_RESTORE=true +XENDOMAINS_RESTORE=false # This variable sets the directory where domains configurations # are stored that should be started on system startup automatically.
Configure Xen-tools¶
Initial minimal xen-tools configuration (my vg for domUs is named vg_raid1)
--- a/xen-tools/xen-tools.conf +++ b/xen-tools/xen-tools.conf @@ -52,7 +52,7 @@ # LVM volume group here instead # ## -# lvm = vg0 +lvm = vg_raid1 # @@ -130,10 +130,10 @@ memory = 128M # Suffix (G, M, k) required #maxmem = 256M # Suffix (G, M, k) optional swap = 128M # Suffix (G, M, k) required # noswap = 1 # Don't use swap at all for new systems. -fs = ext3 # Default file system for any disk -dist = `xt-guess-suite-and-mirror --suite` +fs = ext4 # Default file system for any disk +dist = wheezy # Default distribution is determined by Dom0's distribution -image = sparse # Specify sparse vs. full disk images (file based images only) +image = full # Specify sparse vs. full disk images (file based images only) # # See the README for currently supported and tested distributions. You can @@ -152,7 +152,7 @@ image = sparse # Specify sparse vs. full disk images (file based images only) # new instances static IP addresses. # # gateway = 192.168.1.1 -# netmask = 255.255.255.0 +netmask = 255.255.255.0 # broadcast = 192.168.1.255 # # Uncomment this if you wish the images to use DHCP @@ -241,7 +241,7 @@ initrd = /boot/initrd.img-`uname -r` # Uncomment the following line if you wish to use pygrub by default # for all distributions. # -# pygrub = 1 +pygrub = 1 #
Setup Tab completion¶
Debian currently lacks tab completion for xl #768005
Basic (not as good as the xm completion) tab completion can be fetched here xen.git xl.
Store it as /etc/bash_completion.d/xl
Enable mcelog¶
To get machine check exceptions like ECC errors logged with mcelog, you need to blacklist any used edac modules.
Check for them with lsmod | fgrep edac and blacklist if you find them to be loaded:
Example:
echo "blacklist edac_core" >> /etc/modprobe.d/edac-blacklist.conf echo "blacklist i7core_edac" >> /etc/modprobe.d/edac-blacklist.conf
After this mcelog can be installed to handle events from the kernel
$ sudo apt-get install mcelog
Setup pci passthrough (optional)¶
Find the pci id for the device you would like to pass to a domU
$ lspci | fgrep Ethernet 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe
Create the following init.d script as /etc/init.d/pci-release to pass 03:00.1
#!/bin/sh ### BEGIN INIT INFO # Provides: pci-release # Required-Start: $syslog $remote_fs xend # Required-Stop: # X-Start-Before: xendomains # Default-Start: 2 3 4 5 # Default-Stop: # Short-Description: Provides pci-release # Description: Releases some pci devices to be used by xen domUs ### END INIT INFO if [ "$1" != "start" ]; then exit 0 fi echo "Adding devices to assignable list for xen domUs" echo "Before:" xl pci-assignable-list for d in 03:00.1; do echo "Adding $d" lspci | grep -i "^$d" xl pci-assignable-add $d echo "Done adding $d" done echo "After:" xl pci-assignable-list echo "Done adding devices" exit 0
Edit 03:00.1 for your needs (hint lspci) and add as many pci devices as you need to passthrough.
Enable the init.d script
$ sudo chmod a+x /etc/init.d/pci-release $ sudo update-rc.d pci-release defaults
Make sure xen-pciback is loaded at boot by adding
# xen pci to domU xen-pciback
in /etc/modules, thats it! After this the pci devices can be handed to domUs by adding a line like this is the domU config file
pci = [ '03:00.1' ]
or if you see problems with the exported device in the domU (check dmesg):
pci = [ '03:00.1,permissive=1' ]