Guide to configure an XEN Dom0 server based on Alpine 3.15¶
I have the following configuration for my file server.
- Server HP ProLiant MicroServer Gen8 G1610T
- Memory Kingston KCP316ED8/8 8GB DDR3 1600MHz ECC Module, 16GB total.
- USB Boot USB 16GB Stick Cruzer
During the installation we also need an USB stick (1G+), as well as a workstation/laptop.
- Temporary USB Stick, to be used as installation media.
Various references¶
Here are some various references I have been looking at
Preparation¶
This chapter should be executed on your normal desktop/laptop.
First we need to download the Alpine XEN Image from Alpines download page or use this direct link to alpine-xen-3.15.0-x86_64.iso
Store the image under /tmp.
Insert the installation Media USB stick into your desktop/laptop, and check which device it is located at
# dmesg | tail ... [162562.819054] sd 5:0:0:0: [sdb] 2046240 512-byte logical blocks: (1.04 GB/999 MiB) [162562.823977] sdb: sdb1
You can also check dmesg for removable disks, as per this example.
# dmesg | grep -i removable [14607526.725995] sd 16:0:0:0: [sdb] Attached SCSI removable disk [15742096.383712] sd 17:0:0:0: [sdb] Attached SCSI removable disk #
On my system, we can see that the Installation Media USB stick has been attached as /dev/sdb, but please note that this varies from system to system.
Please, ensure that the Installation Media USB stick has not been automatically mounted.
$ mount | grep <Removable Installation Media USB Stick> $ df | grep <Removable Installation Media USB Stick>
If it had been auto-mounted, please unmount it. Since the dd command expects it to be unmounted.
# umount <Removable Installation Media USB Stick>
Time to move the just downloaded Alpine Xen 3.15 Image to the installation Media USB stick using dd.
The command we should use is dd, and syntax
# dd if=/tmp/alpine-xen-3.15.0-x86_64.iso of=<Removable Installation USB Stick>
In my case the command would be
$ sudo -i # dd if=/tmp/alpine-xen-3.15.0-x86_64.iso of=/dev/sdb 430080+0 records in 430080+0 records out 220200960 bytes (220 MB, 210 MiB) copied, 76.6593 s, 2.9 MB/s # sync
At this time, we are done working on our workstation, time to move over to the new server.
Prepare Boot USB Stick¶
Time to start the installation on the new Alpine Server. Insert the newly created Installation Media USB Stick, and reboot the server. Ensure that you boot from the installation media.
When server has booted, insert the Boot USB Stick (16GB USB Cruzer in my case)
At login prompt, login as root, no password at this stage.
Login: root
Time to figure out which device the Boot USB Stick device has.
Look in the dmesg output and search for your USB stick. In my case Cruzer.
It should be located towards the end of the dmesg output.
# dmesg [ 12.296717] usb 1-1.1: new high-speed USB device number 3 using ehci-pci [ 12.313381] usb 2-1.3: new high-speed USB device number 3 using ehci-pci [ 12.398454] usb 1-1.1: New USB device found, idVendor=0781, idProduct=5571 [ 12.398457] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 12.398459] usb 1-1.1: Product: Cruzer Fit [ 12.398461] usb 1-1.1: Manufacturer: SanDisk [ 12.398463] usb 1-1.1: SerialNumber: 4C530001231102113292 [ 12.398886] usb-storage 1-1.1:1.0: USB Mass Storage device detected [ 12.399172] scsi host8: usb-storage 1-1.1:1.0 [ 13.410934] scsi 8:0:0:0: Direct-Access SanDisk Cruzer Fit 1.00 PQ: 0 ANSI: 6 [ 13.412237] sd 8:0:0:0: [sde] 30595072 512-byte logical blocks: (15.7 GB/14.6 GiB) [ 13.414122] sd 8:0:0:0: [sde] Write Protect is off [ 13.414126] sd 8:0:0:0: [sde] Mode Sense: 43 00 00 00 [ 13.415246] sd 8:0:0:0: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [ 13.420719] sde: sde1 sde2 [ 13.423963] sd 8:0:0:0: [sde] Attached SCSI removable disk
In my case the device we will be using is sde (Cruzer)
We need to format the Boot USB stick with two partitions
- We need a small bootable boot partition for dom0, about 1GB is enough.
- The rest will be used as a LVM partition, for holding supporting domU's
We will format using fdisk. Make sure you target the Boot USB stick.
# fdisk /dev/sde
End result should look like this
# fdisk -l /dev/sde Disk /dev/sde: 15 GB, 15664676864 bytes, 30595072 sectors 1904 cylinders, 255 heads, 63 sectors/track Units: cylinders of 16065 * 512 = 8225280 bytes Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type /dev/sde1 * 0,1,1 122,254,63 63 1975994 1975932 964M c Win95 FAT32 (LBA) /dev/sde2 123,0,1 1023,254,63 1975995 30587759 28611765 13.6G 8e Linux LVM
NOTE observe that sde1 is a bootable partition.
NOTE you might need to unplug/replug the stick after partitioning it.
Or run
# partprobe; sleep 2; mdev -s
Time to add the syslinux package
# apk add syslinux
Load the VFAT kernel module
# modprobe vfat
Create bootable file system
# mkfs.vfat /dev/sde1
Check the UUID of the newly created filesystem, we need the UUID later so make a note of it
# blkid | fgrep sde1 /dev/sde1: UUID="61DF-01FB" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="2a6b8479-01"
We need to know which device is the Installation Media, to do this do a df, and in my case the Installation media was mounted at /media/sdc (it might be mounted on different place depending on your situation)
# df | grep '/media/' /dev/sdc 986036 585164 400872 59% /media/sdc
Run the setup-bootable script to add Alpine Linux to the Boot USB stick and make it bootable (replacing sde with your Boot USB stick name):
# setup-bootable <Installation media> <Boot USB Stick> # setup-bootable /dev/sdc /dev/sde1
This steps might take a few minutes.
When the setup-bootable script is finished, the installation is done, and we can remove the Installation media, and reboot.
<remove installation media> # reboot
Basic alpine host setup¶
After reboot, login on console again, (still no password for root), and it is time to configure this alpine installation.
# setup-alpine
- Configure
eth0, and addbr0, end result should be like below. - Confirm that the IP, DNS, GW are correct and valid
- Based on some script dependencies, it is best to name the bridge
br0 - For DNS IP, please use your own, or if unsure, use Google
8.8.8.8
below is printout of my interfaces file after the setup-alpine script was run.
auto lo iface lo inet loopback auto br0 iface br0 inet static bridge-ports eth0 address 192.168.1.5 netmask 255.255.255.0 gateway 192.168.1.1
- Skip
eth1, answerdone - Which disk(s) would you like to use:
none - Enter where to store configs:
sdb1 - Enter apk cache directory:
/media/sdb1/cache
After setup has finish, you need to commit the changes to the Boot USB stick.
# lbu commit
Switch to mount with UUID¶
After the installation a line with /dev/sdb1 mounted as /media/sdb1 was added in /etc/fstab, the problem is that sdb is a dynamic name so
we should switch to mounting via UUID.
fstab Before
/dev/sdb1 /media/sdb1 vfat ro,relatime,fmask=0022,dmask=0022,errors=remount-ro 0 0
fstab After
UUID=61DF-01FB /media/flash vfat ro,relatime,fmask=0022,dmask=0022,errors=remount-ro 0 0
Mount it
# mkdir /media/flash; mount /media/flash
Fix lbu.conf and the apk repositories file
# sed -i's/sdb1/flash/' /etc/lbu/lbu.conf /etc/apk/repositories
Fix apk cache symlink
# rm /etc/apk/cache; ln -sf /media/flash/cache /etc/apk/cache
Save
# lbu commit
Add main user¶
I took a small short cut here, and added a normal user.
This so I could SSH to the new Alpine server, and copy paste commands from this document.
Note Skip the use of etckeeper as its not installed if taking a short-cut (use lbu commit)
Reboot
# reboot
At login prompt, login as root, use the password selected in setup-alpine
Login: root
... or login via ssh as the "main user" optionally created above and use su - to switch to root
Add etckeeper¶
Etckeeper will store whole /etc in a git repository, and keep track of every change
that is made there.
Just remember to do a etckeeper commit "Describe the change"
# apk add etckeeper # cat << EOF > /etc/.git/hooks/post-commit #!/bin/sh set -e lbu commit EOF # chmod 755 /etc/.git/hooks/post-commit # lbu commit
Adding LVM¶
First, we need to confirm what name the Boot USB stick has (in my case sde).
# df
Time to add and configure LVM.
# apk add lvm2 # pvcreate /dev/sde2 Physical volume "/dev/sde2" successfully created. # vgcreate vg_domU /dev/sde2 Volume group "vg_domU" successfully created # lvcreate -n lv_domU_installer -L 1G vg_domU Logical volume "lv_domU_installer" created. # rc-update add lvm * service lvm added to runlevel default # etckeeper commit "Added LVM"
And setup the filesystem.
# apk add e2fsprogs # mkfs.ext4 /dev/vg_domU/lv_domU_installer
And prepare for domU installation medias.
# mkdir /domU_installer # echo "/dev/vg_domU/lv_domU_installer /domU_installer ext4 noauto,noatime 0 0" >> /etc/fstab # mount /domU_installer # etckeeper commit "Added /domU_installer to fstab"
NOTE That the /domU_installer is not mounted automatically on boot, but has to be specifically mounted by user when needed, as well as unmounted when not needed anymore.
When you try to create a domU, and xl complains about a missing media is usually a sure sign that you forgot to
mount /domU_installer
xen-hypervisor¶
We need to add the xen-hypervisor manually, since we run diskless.
# apk add xen-hypervisor
Need to ensure xendomains starts at reboot
# rc-update add xendomains # etckeeper commit "Start xendomains at reboot"
TMUX Console¶
We will use tmux to capture the various domU's console.
# apk add tmux
Then we need to uncomment a line in the /etc/conf.d/xendomains configuration file
Before:
#XENDOMAINS_CONSOLE="tmux"
After:
XENDOMAINS_CONSOLE="tmux"
# etckeeper commit "Configure xendomains to use tmux"
After you have created some domU's the output might look like below, and based on the below output the tmux session is called xen.
# tmux ls 0: 1 windows (created Wed Sep 5 05:39:38 2018) [127x53] 4: 1 windows (created Sun Sep 9 01:24:43 2018) [127x53] xen: 4 windows (created Mon Sep 10 14:44:51 2018) [127x53]
And you would then connect to the xen tmux session with tmux attach-session using the following command:
# tmux attach-session -t xen
For further information on how to navigate a tmux session, please check the following excellent Tmux Cheat Sheet
Prepare for domU¶
Alpine Image¶
Prepare domU's installer image (needed to create you first domU later) Download Alpine Extended from link to Alpine 3.15.0 Extended 64 bits, or if You need an other version go the Alpine Download Page and download Your preferred "Extended" version.
The reason we use the -extended flavor, is so when we create our domU;s we do not have to have network configured and protected when we do the installations.
Remember to download the checksum file as well.
# mount /domU_installer # cd /domU_installer # wget https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/x86_64/alpine-extended-3.15.0-x86_64.iso.sha256 # wget https://dl-cdn.alpinelinux.org/alpine/v3.15/releases/x86_64/alpine-extended-3.15.0-x86_64.iso
After you have downloaded both ISO file as well as checksum file, you need to verify that the ISO file has the correct checksum. If not, please re-download both and try again.
# ls -lh total 598M -rw-r--r-- 1 root root 598.0M Oct 10 23:42 alpine-extended-3.15.0-x86_64.iso -rw-r--r-- 1 root root 100 Oct 10 23:39 alpine-extended-3.15.0-x86_64.iso.sha256
And now we verify that the checksum is OK.
# sha256sum -c alpine-extended-3.15.0-x86_64.iso.sha256 alpine-extended-3.15.0-x86_64.iso: OK
After we have verified that the iso image has the correct checksum, place it in the /domU_installer directory (if not already there).
# mv <Downloaded Alpine Extended 64 ISO> /domU_installer
domU preparation¶
We prepare for the domU's installation by fetching the domU boot loader here.
# apk add grub-xenhost
The installation kernel and ramdisk we fetch from the ISO image.
# mount -t iso9660 -o ro alpine-extended-3.15.0-x86_64.iso /mnt # cp /mnt/boot/vmlinuz-lts /domU_installer # cp /mnt/boot/initramfs-lts /domU_installer # umount /mnt
Add dom0 Memory¶
Limit the memory used by the dom0 so domUs does not have to "steal" available memory from dom0, when starting.
Since dom0 is booting from the Boot USB stick, we need to update syslinux at /media/flash/boot/syslinux/syslinux.cfg
Previous version it was enough with 1GB ram, but I found that you could not do a system update/upgrade after fresh install with only 1G. Apparently XEN requires a bit more during upgrades, so I'm using 2G here.
# mount /media/flash -o remount,rw # vi /media/flash/boot/syslinux/syslinux.cfg
in particular, we need to add a parameter (dom0_mem=2048M) to the row starting with APPEND. (A lower value will not work as the dom0 is running on a RAM disk.)
APPEND /boot/xen.gz dom0_mem=2048M --- /boot/vmlinuz-lts modules=loop,squashfs,sd-mod,usb-storage quiet nomodeset --- /boot/initr
And remount Boot USB stick read only again.
# mount /media/flash -o remount,ro
And lastly we reboot and confirm the newly created Alpine Dom0 server is booting up.
# sync # reboot
Post steps¶
When system comes up it is time to do the final touches.
- Add normal user
- Check network etc
Ping working?¶
# ping www.google.com
Lets update the system¶
# apk update # apk upgrade
Add sudo¶
For security reasons, and good practice, lets install sudo
# apk add sudo # sed -e 's/# %wheel ALL=(ALL:ALL) ALL/%wheel ALL=(ALL:ALL) ALL/g' -i /etc/sudoers # etckeeper commit "Configured sudoers"
Add a user¶
# adduser <normal-user-ID> # adduser <normal-user-ID> wheel # lbu include /home # lbu exclude /home/*/.ash_history # etckeeper commit "Added a user"
Next steps¶
After this step, comes perhaps, and it can be argued if the ZFS host should come before the Network Domain or not...
- Installing a Basic Server as a domU.
- Installing a ZFS Storage Domain as a domU.
- Installing a Network Domain (firewall) as a domU.
- Installing a DNS Caching server as a domU.
- etc...
These HowTo's are being prepared and will be linked in here later. In the mean time, have a look at the old link below.
Appendix¶
Hypervisor upgrade¶
When the package xen-hypervisor package has been updated, the following needs to be done on the dom0 server.
- The XEN related files on the boot device (
/media/flash/boot/*xen*) needs to be updated.
After the upgrade the XEN boot related files are located in the ramfs, so we need to copy those files to the restart safe place on the boot device
First, remount the Boot USB stick writable.
# mount /media/flash -o remount,rw
Second, copy the files needed.
# cp /boot/*xen* /media/flash/boot
Third, remount Boot USB stick read only again.
# mount /media/flash -o remount,ro
Fourth, time to verify; we need to confirm that the XEN reference in syslinux.cfg points to the correct XEN version.
# grep xen /media/flash/boot/syslinux/syslinux.cfg APPEND /boot/xen.gz dom0_mem=2048M --- /boot/vmlinuz-lts modules=loop,squashfs,sd-mod,usb-storage quiet nomodeset --- /boot/initr
From above we see that we are using /boot/xen.gz.
Lets confirm that xen.gz is the one we want. Confirming that checksum is same for xen.gz and xen-NewVersion.gz.
# cksum /media/flash/boot/xen*.gz 2396837473 1175423 /media/flash/boot/xen-4.15.1.gz 2396837473 1175423 /media/flash/boot/xen-4.15.gz 2396837473 1175423 /media/flash/boot/xen-4.gz 2396837473 1175423 /media/flash/boot/xen.gz
And last, confirm the above xen.gz checksum are the same as under /boot.
# cksum /boot/xen*.gz 2396837473 1175423 /boot/xen-4.15.1.gz 2396837473 1175423 /boot/xen-4.15.gz 2396837473 1175423 /boot/xen-4.gz 2396837473 1175423 /boot/xen.gz
dom0 Kernel Upgrade¶
You can update the kernel by putting <hostname>.apkovl.tar.gz on a freshly installed USB stick or with the update-kernel command.
However using update-kernel requires more dom0_mem (update-kernel uses both RAM and ramfs on /tmp), hence we are temporarily
using a 4GB dedicated LVM disk for /tmp during the kernel upgrade to be able to complete the update without running out of memory.
The tempfs volume is to small for the kernel upgrade.
# df -h /tmp Filesystem Size Used Available Use% Mounted on tmpfs 155.7M 137.2M 18.5M 88% /
We add a kernel upgrade volume.
# lvcreate -n lv_kernel_upgrade -L 4G vg_domU Logical volume "lv_kernel_upgrade" created. # mkfs.ext4 /dev/vg_domU/lv_kernel_upgrade
And prepare for domU installation medias.
# mount -t /dev/vg_domU/lv_kernel_upgrade /tmp # df -h /tmp Filesystem Size Used Available Use% Mounted on /dev/vg_domU/lv_kernel_upgrade 3.9G 16.0M 3.6G 0% /tmp
Check free RAM.
# free -m total used free shared buffers cached Mem: 311 270 40 137 2 203 -/+ buffers/cache: 64 247 Swap: 0 0 0
Now update-kernel can be used.
# mount -o remount,rw /media/flash # update-kernel # mount -o remount,ro /media/flash
If there is a complaint on missing mksquash-fs command, just install it, and re-run the update-kernel command.
# apk add squashfs-tools
And we need to umount the temporary /tmp LVM disk
# umount /tmp
Now you can reboot the dom0 to the updated kernel.
Localized keyboard¶
If You have troubles with nationalized keyboard (it is a bug in setup-alpine?), then do the following work-a-round:
Into the file /etc/profile.d/loadkeymap.sh enter the following content:
#!/bin/ash if [ "/dev/tty1" = "`tty`" ]; then echo "Loading selected keymap." /etc/init.d/loadkmap start fi
Then commit the changes.
# etckeeper commit "Added loadkeymap"
domU installation config¶
To use this installation image, you could use the below template as a base. You have to make sure that your MAC address is UNIQUE. A tool that can help you with this is, for example, random_mac.py.
If you do it manually, please start with 00:16:3E followed by a unique combination for you.
For instance 00:16:3e:AA:AA:01 or 00:16:3e:BE:EF:01.
##### ##### <Hostname> domU ##### vcpus = '1' memory = '256' maxmem = '256' kernel = "/domU_installer/vmlinuz-lts" ramdisk = "/domU_installer/initramfs-lts" extra = "alpine_dev=hdc:iso9660 modules=loop,squashfs,sd-mod,usb-storage console=hvc0" disk = [ 'file://domU_installer/alpine-extended-3.15.0-x86_64.iso,hdc:cdrom,r', 'phy:<Physical Path to your disk>,xvda1,w', ] name = '<Hostname>' ## ENSURE THAT THE MAC ADDRESS IS UNIQ!!! vif = [ 'mac=<Unique MAC Address>,bridge=br0' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart'
domU running config¶
And slight modification is needed with using the proper kernel, and removing the installation image.
##### ##### <Hostname> domU ##### vcpus = '1' memory = '256' maxmem = '256' kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin" disk = [ 'phy:<Physical Path to your disk>,xvda1,w', ] name = '<Hostname>' ## ENSURE MAC ADDRESS IS UNIQ!!! vif = [ 'mac=<Unique MAC Address>,bridge=br0' ] on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart'