MyComputingArt (circa 1993)

MyComputingArt (circa 1993)

Mon, 26 Nov 2007

Ubuntu Dapper Drake 6.06 amd64 and Windows XP in Xen 3.0

I am coming from a bloody week of wrestling against packages, bugs and "make world"s trying to run Xen in Ubuntu 6.06 amd64 on an AM2 3800+ and to make a Windows XP Pro domU.
I searched on the web some resources that would explain how to run Windows on Xen and I found some very good tutorial that unfortunately don't fit exactly my configuration and my purpose. These are the sites I took as a basis for my setup:

These are all the steps I did and the problems I encountered during this Calvary :-) (I come from Windows with a little linux/unix background):

Note: All the shell commands are executed as root.

Remove unnecessary and maybe problematic packages and install required packages:

apt-get remove exim4 exim4-base lpr nfs-common portmap pidentd pcmcia-cs
apt-get install screen ssh debootstrap python python-twisted-core iproute bridge-utils libcurl3-dev libncurses-dev

I installed also python-dev with Synaptic because apt-get didn't find it
Xen requires also these packages:
- dev86: I compiled bin86-0.16.17.tar.gz sources but it's the same as installing bin86 using Synaptic or apt-get
- sdl and sdl-dev: I compiled SDL-1.2.11 sources because Synaptic and apt-get was reporting a dependency error installing libsdl1.2-dev package (see Ubuntu forum)
- libvncserver: I compiled LibVNCServer-0.8.2 and x11vnc-0.8.2 because I didn't find libvncserver with Synaptic or apt-get; I also installed libvncserver-dev using apt-get
While this happened to me with ubuntu-6.06-alternate-amd64, Nicholas Lee reported no problems installing the following packages with ubuntu-6.06.1-server-amd64 (Thank you Nicholas!):
```
apt-get install libncurses5-dev libncurses5 zlib1g-dev python python-dev python-twisted-core openssl libssl-dev
bridge-utils iproute libcurl3 libcurl3-dev bzip2 module-init-tools tetex-base tetex-extra transfig tgif build-essential
libsdl1.2-dev libvncserver-dev mercurial libjpeg-dev qemu bcc bin86
```
I also installed qemu because I wasn't getting Xen working and I read somewhere to try loading the guest with qemu to check if the problem is in Xen configuration or elsewhere. I don't know if qemu must be installed for Xen to work because I installed qemu, libsdl and libvncserver together and then recompiled Xen; I suppose it's not required because I didn't find any source listing qemu in Xen requirements.
Get Xen 3.0-testing sources.
Don't install Xen 3.0.2 binaries (it doesn't install hvmloader) and don't build Xen 3.0.2 sources (there's a Makefile issue regarding hvmloader and 64-bit). See the troubleshooting section for more info.
Install mercurial (a distributed version control system) and get xen-3.0-testing:
```
# apt-get install mercurial
# cd /usr/src/
# hg clone http://xenbits.xensource.com/xen-3.0-testing.hg
```

Compile the sources (during this process the last kernel will eventually be downloaded from kernel.org):

# cd xen-3.0-testing.hg
# LANG=C; export LANG
# make world
# make install

The LANG=C; export LANG is necessary on my system because there's a bug in dd that generates a "segmentation fault" error (see Ubuntu forum)
At the end I got these files in /boot directory:

-rw-rw-r-- 1 root root  829323 Aug 12 22:32 System.map-2.6.16.13-xen
-rw-rw-r-- 1 root root   55292 Aug 12 22:32 config-2.6.16.13-xen
-rw-r--r-- 1 root root 7239478 Aug 12 22:32 vmlinux-syms-2.6.16.13-xen
lrwxrwxrwx 1 root root      21 Aug 12 22:32 vmlinuz-2.6-xen -> vmlinuz-2.6.16.13-xen
lrwxrwxrwx 1 root root      21 Aug 12 22:32 vmlinuz-2.6.16-xen -> vmlinuz-2.6.16.13-xen
-rw-r--r-- 1 root root 1646379 Aug 12 22:32 vmlinuz-2.6.16.13-xen
-rw-r--r-- 1 root root  251370 Aug 12 22:30 xen-3.0.2-3.gz
lrwxrwxrwx 1 root root      14 Aug 12 22:30 xen-3.0.gz -> xen-3.0.2-3.gz
lrwxrwxrwx 1 root root      14 Aug 12 22:30 xen-3.gz -> xen-3.0.2-3.gz
-rw-r--r-- 1 root root 3190960 Aug 12 22:30 xen-syms-3.0.2-3
lrwxrwxrwx 1 root root      14 Aug 12 22:30 xen.gz -> xen-3.0.2-3.gz

Following the tutorials above I should do
```
# mv /lib/tls /lib/tls.disabled
```
but I don't have tls, so I skipped this step.
On Ubuntu 6.06 i386 I have /lib/tls, so I moved it.
```
# /sbin/depmod -a 2.6.16-xen
```
In /lib/modules I got the directory 2.6.16-xen (2.6.16.13-xen was created when installing xen)
On Ubuntu 6.06 i386 I did /sbin/depmod -a 2.6.16.13-xen because in /lib/modules I only have 2.6.16.13-xen
Edit /etc/mkinitramfs/modules:
```
loop max_loop=64
```
Create an initrd image:
```
# cd /boot
# mkinitramfs -o initrd.img-2.6.16.13-xen 2.6.16.13-xen
# ln -sf initrd.img-2.6.16.13-xen initrd.img-2.6-xen
```
I got /boot/initrd.img-2.6.16.13-xen and initrd.img-2.6-xen.
On Ubuntu 6.06 i386 I only got initrd.img-2.6.16.13-xen and I symlinked it with ln -s initrd.img-2.6.16.13-xen initrd.img-2.6-xen.
Add in /boot/grub/menu.lst before or after automagical section:
```
title Xen 3.0 / XenLinux 2.6 
kernel /boot/xen-3.gz dom0_mem=1024000
module /boot/vmlinuz-2.6-xen root=/dev/sda1 ro
module /boot/initrd.img-2.6-xen
```
/ is on the first partition of my SATA disk (/dev/sda1) and I told Xen to use 1 GB of the 2 GB of RAM available.
Then grub-install /dev/sda because I want grub in my first disk's MBR.

Start xend and xendomains at boot. I did:

# update-rc.d xend start 30 2 3 4 5 . stop 31 0 1 6 .
 Adding system startup for /etc/init.d/xend ...
   /etc/rc0.d/K31xend -> ../init.d/xend
   /etc/rc1.d/K31xend -> ../init.d/xend
   /etc/rc6.d/K31xend -> ../init.d/xend
   /etc/rc2.d/S30xend -> ../init.d/xend
   /etc/rc3.d/S30xend -> ../init.d/xend
   /etc/rc4.d/S30xend -> ../init.d/xend
   /etc/rc5.d/S30xend -> ../init.d/xend
# update-rc.d xendomains start 31 2 3 4 5 . stop 30 0 1 6 .
 Adding system startup for /etc/init.d/xendomains ...
   /etc/rc0.d/K30xendomains -> ../init.d/xendomains
   /etc/rc1.d/K30xendomains -> ../init.d/xendomains
   /etc/rc6.d/K30xendomains -> ../init.d/xendomains
   /etc/rc2.d/S31xendomains -> ../init.d/xendomains
   /etc/rc3.d/S31xendomains -> ../init.d/xendomains
   /etc/rc4.d/S31xendomains -> ../init.d/xendomains
   /etc/rc5.d/S31xendomains -> ../init.d/xendomains

Another workaround for a problem with udev (see Ubuntu forum):

# mv /etc/udev/rules.d/xen-backend.rules /etc/udev/rules.d/92-xen-backend.rules

Edit /etc/init.d/xend after "/proc/xen/capabilities":

if [ ! -d /var/run/xend ] ; then
    mkdir -p /var/run/xend
fi

if [ ! -d /var/run/xenstored ] ; then
    mkdir -p /var/run/xenstored
fi

Edit /etc/init.d/xendomains:
```
LOCKFILE=/var/lock/xendomains
```

Now I rebooted and at grub menu I chose Xen 3.0 / XenLinux 2.6, and I logged in exactly as I did before in Ubuntu. I got no errors, so I checked if Xen was running:

# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0      995     2 r-----   107.5

dom0 (Xen) was running and using 1 GB of memory.

To run Windows from Xen using Pacifica (SVM) hardware virtualization, first check if the cpu supports it and if it's enabled:

# cat /proc/cpuinfo | grep svm
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm cr8_legacy
# xm dmesg | grep SVM
(XEN) AMD SVM Extension is enabled for cpu 0.
(XEN) AMD SVM Extension is enabled for cpu 1.

These are the steps I did to create a Windows XP domU domain:

Create Xen image for Windows XP:

# LANG=C; export LANG
# mkdir /vserver/xen-images
# cd /vserver/xen-images
# dd if=/dev/zero of=winxp.img bs=1k seek=20480k count=1
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 9.3e-05 seconds, 11.0 MB/s
# dd if=/dev/zero of=winxp.img bs=1k count=1 conv=notrunc
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 6.7e-05 seconds, 15.3 MB/s

I haven't put the export LANG=C into my .bashrc yet.
/vserver is the mount point of the ext3 partition I reserved to Xen images (25 GB).
The dd commands create a zeroed file that's large bs * 1024 * seek * 1024 bytes (20 GB).

Configure Xen image to start Windows XP:

# mkdir /etc/xen/vm
# cp /etc/xen/xmexample.hvm /etc/xen/vm/winxp.hvm

and make these changes to /etc/xen/vm/winxp.hvm:

name = "winxp"
memory = 512 
disk = [ 'file:/vserver/xen-images/winxp.img,ioemu:hda,w' ]
on_poweroff = 'destroy'
on_reboot   = 'destroy'
on_crash    = 'destroy'
cdrom='/dev/hdb'
boot='d'
sdl=1
vnc=0 
vncviewer=0

Create Windows XP guest:

# cd /etc/xen/vm
# xm create winxp.hvm
Using config file "winxp.hvm".
Started domain winxp

Immediately a window opened showing Windows setup:

Check the status:

root@am2:/etc/xen/vm# xm list
Name                              ID Mem(MiB) VCPUs State  Time(s)
Domain-0                           0     1000     2 r-----  1139.5
winxp                             13      512     1 ------    18.9

When Windows setup has finished, launch again xm create winxp.hvm: a window with initial configuration should appear and Windows should start.

Troubleshooting

Here are some problems I encountered and the relative solutions, where available.

dd problem
If when doing a dd command, like in
```
dd if=/dev/zero of=winxp.img bs=1k seek=20480k count=1
```
you get something like
```
1+0 records in
1+0 records out
Segmentation fault
```
you may have a malfunctioning dd: it seems there's a bug in the version coming with Ubuntu Dapper Drake 6.06 for amd64 (see for example Ubuntu forum).
Try these solutions:
- run LANG=C; export LANG before executing dd, or LANG=C dd if=....
- take dd from another distribution (I tried debian coreutils_5.97-3_amd64 but it didn't solved)
- I tried also this command, as I read here, but in my case it didn't solved:
```
# touch /usr/share/locale-langpack/en/LC_MESSAGES/coreutils.mo
```
Some other problems due to this bug:
- If you ignored the "Segmentation fault" errors and created the void image for domU with dd if=/dev/zero of=/... maybe you got this error doing xm create ...:
```
Error: Kernel image does not exist: /usr/lib/xen/boot/hvmloader
```
- Compiling of the sources will stop at a dd command
SATA disk problem
If xm create gives an error like this:
```
Using config file "winxp.hvm".
Error: hvm: for qemu vbd type=file&dev=hda~hdd
```
and in the xen image configuration file ("winxp.hvm") the disk= line is like
```
disk = [ 'file:/vserver/xen-images/winxp.img,ioemu:sda,w' ]
```
(note the ioemu:sda), try changing to ioemu:hda; maybe there's a problem with the sata_nv driver in Xen kernel (see here).
hvmloader missing in xen-3.0.2-install-x86_64.tgz
I downloaded xen-3.0.2-install-x86_64.tgz, run install.sh, configured all the files and at the end just as I launched
```
xm create winxp.hvm
```
I got this error:
```
Error: No kernel specified
```
I did not have any file called hvmloader in the directory specified, and after some search I found that file in a deb package that's not available for amd64, so I decided to make Xen from sources
32-bit hvmloader problem with xen-3.0.2-src.tgz
I got xen-3.0.2-src.tgz using torrent, compiled, installed, tried many configurations, and every time I was launching xm create winxp.hvm I was getting an immediate hard reset (and I also lost my swap partition, I had to do mkswap /dev/sda5).
I checked hvmloader with
```
# file /usr/lib/xen/boot/hvmloader
/usr/lib/xen/boot/hvmloader: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, not stripped
```
and I saw it was a 32-bit executable. The problem seems not to be the fact that it's a 32-bit executable on my amd64 platform because even when I got it working it was a 32-bit executable, but that there was a Makefile issue related to 64-bit.
Check with Qemu
There was a moment when I was stuck with libsdl and libvncserver package problems: I installed them and before compiling again Xen I wanted to do some more checks: I read that running Qemu on the image would give a hint about where the problem is: if Qemu works I would have to search in Xen configuration or compile, if Qemu doesn't work I would have to check the required packages. So I installed qemu package and tried:
```
qemu -hda /bkup/winxp.img -cdrom /mmedia/os/winxp_oem.iso -m 512 -boot d
```
I created winxp.img with dd on a EIDE disk partition because I read something about qemu problems on SATA disks, I was not sure but I wanted to avoid any possible problems, so I used /dev/hda5. Instantly a window opened and Windows setup started; it hang up after copying all the files and setting up keyboard, regional settings, etc., I believe at the reboot time, so I shutted it down with xm destroy winxp and tried to boot Windows from the image instead of the cdrom iso:
```
qemu -hda /bkup/winxp.img -cdrom /mmedia/os/winxp_oem.iso -m 512 -boot c
```
A window appeared and Windows loaded without any problems. Then I recompiled Xen, rebooted and reconfigured: now Xen was working!

[/linux] permanent link

Mon, 26 Nov 2007

Troubleshooting

dd problem

SATA disk problem

hvmloader missing in xen-3.0.2-install-x86_64.tgz

32-bit hvmloader problem with xen-3.0.2-src.tgz

Check with Qemu