Xen: NetBSD VM on a Debian Dom0
It’s been a while since I have blogged geekiness, but this one is really necessary: Today I installed a test box with an instance of the Xen Virtual Machine Monitor, with Debian Lenny as the Domain 0 (or Device Driver VM, as the researchers at my university like to call it).
The reason was that I have to run a piece of legacy software that is in SCO Unix binary format, which is incompatible with the (unaltered) Linux kernel. There is the linux-abi project whose kernel patches bring SCO binary compatibility to Linux, but I always try to avoid rebuilding the kernel because I won’t be able to update it anymore with the distro’s means; instead, I have to rebuild the kernel myself when I want to update, and (much worse), before long I am likely to end up in a situation where I am unable to avoid breaking package dependencies — keeping an up-to-date system should just not be that hard.
Thus the idea was born to run several virtual machines on the same hardware, dedicating one of them to the task of running the legacy software, and another one to running the more up-to-date standard services.
However, this still doesn’t change the fact that I would have to build a special linux-abi patched kernel, and this time even worse: It would also have to be modified for running in a Xen domain. To save myself that pain, I looked for alternatives and found the binary compatibility page in the NetBSD docs, stating that it supports UNIX binaries (including SCO) out of the box (and many more). Furthermore, NetBSD has apparently been supporting running on Xen since quite a while now.
Installing NetBSD into a Xen VM (following the howtos) is supposedly quite easy. I created an LVM volume on the harddrive to put the new system into, set up that partition as well as a current NetBSD ISO image as virtual devices, and pointed the config file to a special NetBSD installer kernel image for Xen that NetBSD provides. Then I tried to start installing the VM. But, ouch, Xen claims: “incompatible kernel”. Hm. Wasn’t that easy after all.
As it turns out, the problem is that current Debian kernels are all compiled with Intel’s physical address extensions (PAE) enabled: In short, common 32bit hardware can only address 2^32 bytes of physical RAM, that’s about 4GB. For modern systems, this can be a little short, so extensions where built to support more than that. Modern Linux distributions support them and they usually don’t harm even if you have less RAM than that; sadly, the stable NetBSD distribution does not support PAE yet, and running two systems on the same physical box that have a different understanding of how to talk to physical memory does not work.
But, lucky as I am, just a few weeks ago, NetBSD/Xen hacker Manuel Bouyer has implemented PAE support for NetBSD to an extent that allows it to run on a Xen system with PAE-enabled dom0. Thanks, Manuel!
The respective installation and regular kernel images can be found among the daily builds on the NetBSD FTP server, and if you use these kernel images instead, you’ll easily be able to get a NetBSD instance up and running without touching the stock Debian kernel.
As expected, NetBSD was able to run the SCO binaries, so far without problems. A few iptables rules on the domain0 will soon be in place to transparently forward requests for this service to the NetBSD VM, so clients will never know that it is not the Linux server itself responding to their request, but a little virtual machine running in the background.
February 17th, 2008 at 1:29 pm
Note that PAE is quite a crutch and due to kernel/userland memory split memory performance suffers on 32-bit systems with 2GB or more RAM. So use amd64 if your hardware supports it and you want lots of RAM.
February 17th, 2008 at 2:02 pm
Thanks for the comment, chithanh. I read this too today: people who need tons of RAM and want to run Xen usually just switch to appropriate hardware, which I believe makes sense, as PAE is highly likely to perform worse than a system that can handle more RAM by design.
February 17th, 2008 at 11:22 pm
Wow, that is quite the setup just to avoid building your own kernels which really isn’t that hard. Debian even has make-kpkg so you can easily build debs for your custom kernel. imho it will be a lot easier to maintain a system with a custom kernel than dealing with Xen. Until some day when Xen dom0 support gets merged into mainline (might actually happen now that Redhat is working on it, Xen upstream doesn’t care it seems) you are stuck with either a hopelessly out of date kernel (2.6.18) or a hopelessly forward ported kernel (2.6.21 at best as far as I know). But good luck to you!
February 18th, 2008 at 12:16 am
Haha, okay — what I didn’t mention is that there were other reasons for this setup, mainly security reasons and also the comfort of being able to “make” a new box without actually buying hardware. So yeah, I should have mentioned that the decision to use Xen came first, and afterwards I realized that building a Xen domU Kernel that also supports “foreign” binaries and happens to turn out stable is less than trivial
February 18th, 2008 at 1:10 am
@Mike: By the way I do think Xen has made it into the mainline kernel in 2.6.23: http://kerneltrap.org/node/13917
February 18th, 2008 at 11:42 am
Not really, there is limited support for x86 domU which lacks useful things like live migration and memory resizing. Both x86_64 and dom0 support don’t quite exist yet for mainline’s paravirt ops interface but thankfully redhat recently started this work. The upstream xen people do their development on 2.6.18 which predates paravirt ops.
July 10th, 2008 at 5:14 am
Latests INSTALL_XEN3PAE_DOMU snapshots are crashing on boot.
Is it still working (and i’m doing something wrong) or is it regression?
(XEN) domain_crash_sync called from entry.S (ff15b6a9)
(XEN) Domain 32 (vcpu#0) crashed on cpu#1:
(XEN) —-[ Xen-3.0.3-1 x86_32p debug=n Not tainted ]—-
(XEN) CPU: 1
(XEN) EIP: e019:[]
(XEN) EFLAGS: 00000202 CONTEXT: guest
(XEN) eax: 00000000 ebx: 00000000 ecx: 00000001 edx: c043fa74
(XEN) esi: 00000000 edi: 000024a8 ebp: c0437e44 esp: c0437e08
(XEN) cr0: 8005003b cr4: 000002f0 cr3: 41fe1000 cr2: 00000001
(XEN) ds: e021 es: e021 fs: 0000 gs: 0000 ss: e021 cs: e019
(XEN) Guest stack trace from esp=c0437e08:
(XEN) 00000000 c03972b9 0001e019 00010002 c0397393 00000000 00000000 00000000
(XEN) 00000000 00000000 00000000 00000000 00000000 c042e734 000024a8 c0437e74
(XEN) c02f451f 00000000 00000005 00000000 00000000 c0437e80 00000000 00000000
(XEN) 00000000 c0437e80 00000100 c0437eb4 c02f461e c0422b5b 00000000 c0923f00
(XEN) 00000000 00000000 00000000 00000000 00000000 00000000 c0a2a000 00000000
(XEN) c0437ec0 c0437f48 00007ff0 c0437f64 c0396e22 c042e734 c0a2e000 00001000
(XEN) 00000000 0092e000 00000000 00000000 00000000 00000000 00000008 c0a2a000
(XEN) c0a1d000 c0a2b000 c0a351b0 00000000 c0a2b000 0092f000 00000000 c0e00000
(XEN) c0a30000 c0437000 c0a37000 00000007 000024d8 00a2a000 00000000 00000606
(XEN) c0a37000 0092e000 00000000 00000000 c0a37000 c0a25000 c0a30000 00000000
(XEN) 00000002 0002cff2 00000000 00000000 c0a2a000 00000001 00000007 c0437f94
(XEN) c0396f36 00000007 00000000 00000000 00000008 c0a2a000 c0a1d000 00000007
(XEN) 756e6547 c0a1a200 c09a4740 00000000 c0100041 00000000 00000000 c03f20e0
(XEN) 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN) 00000000 00000000 00000000 00000000 c03f25e0 00000000 c03f2360 c03f2500
(XEN) c03ecce0 00000000 00000000 00000000 00000000 00000000
dom0# uname -a
Linux noisy 2.6.18-6-xen-686 #1 SMP Sat Jun 7 02:07:48 UTC 2008 i686 GNU/Linux