Remember to backup data in the USB stick before proceeding the method below.

Command line method

Download the Fedora Live ISO image

We can download Fedora 12 Live ISO image by:

$ wget http://download.fedora.redhat.com/pub/fedora/linux/releases/12/Live/i686/Fedora-12-i686-Live.iso

Or if you are using x86_64 system:

$ wget http://download.fedora.redhat.com/pub/fedora/linux/releases/12/Live/x86_64/Fedora-12-x86_64-Live.iso

Other Fedora Live ISO image with different releases version can be found from: http://download.fedora.redhat.com/pub/fedora/linux/releases/

You can even download an boot.iso and netinst.iso with smaller size if you want to install from network of local image. The method in this part works with these images too.

Directly copy the image to USB media

We can use dd to make a direct copy from the image to USB media. Remember to backup your data on the USB media since all data will be lost after dd.

# dd if=./F12-Live-i686.iso of=/dev/sdX bs=8M

/dev/sdX is device name of the USB stick.

Test the Live USB media

We can use qemu or other virtualization tools to try boot from the Live USB media we created.

# umount /dev/sdbX
# qemu -hda /dev/sdX -m 256 -vga std

/dev/sdX is the Live USB media’s device name.

Graphical method with liveusb-creator

This method can be used on both Linux and Windows. For windows users the liveusb-creator application can be downloaded from: http://fedorahosted.org/liveusb-creator

For Fedora users, liveusb-creator is already in the repository.

# yum install liveusb-creator

Run liveusb-creator and select the “Live CD image” path and the “Target Device”, and then click “Create Live USB”. The application will write the Live image to the USB stick.

After it finishes, we can also use qemu as described above to test the Live USB media.

• Setting up Stable Xen DomU with Fedora: Unmodified Fedora 11 with pv_ops Kernel
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.32.13. This is a very stable Dom0 solution for Fedora 12. Lot’s DomUs have been tested on this platform.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

You may need to install packages depended by this. You can try this for solving the dependencies:

#yum install make gcc -y; \
yum groupinstall "Development Libraries" -y; \
yum groupinstall "Development Tools" -y; \
yum install transfig texi2html \
dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial -y

The we can make Xen and Xen tools:

$ make xen
$ make tools

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number.

You may need to install patch if it isn’t installed:

# yum install patch

This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 < >   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 < >   PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -jX

X can be two times of the number of the processor. We use this to let make invoke compilation work in X-way parallel.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

Please refer to
Xen solution
for the DomU solution and more Xen Dom0 solution.

Update history
Sep. 3, 2010. blktap driver is used instead of blktap2; PCI backend driver is disabled.
Sep. 8, 2010. Add install patch; highlight PCI part in make menuconfig.

• Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12

The Dom0 platform on top of which this DomU is set up is introduced also in this site. Please refer to

Xen solution

for the latest stable Xen Dom0 solution.

Create file-backed VBD:

The actual space of VBD will be the amount of disk the virtual machine used. And it will be convenient if the virtual machine will be duplicated since the work is just copying the VBD file and changing some configurations.

Create a 20GB sparse file-backed VBD:

# dd if=/dev/zero of=/lhome/xen/f11install/vmdisk0 bs=1k seek=20480k count=1

Make a ext4 file system in the disk file:

# mkfs -t ext4 vmdisk0

Install Fedora 11 Linux via Internet

First download the pxeboot kernel of Fedora 11 for installation via Internet. Download vmlinuz and initrd.img from here:

http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/x86_64/os/images/pxeboot/

We assume these two files are stored in /lhome/xen/f11install/.

Create an installation profile f11.install:

name="F11INSTALL"
vcpus=2
memory=2048
disk = ['file:/lhome/xen/f11install/vmdisk0,xvda,w' ]
vif = [ 'bridge=eth0' ]
kernel = "/lhome/xen/f11install/vmlinuz"
ramdisk = "/lhome/xen/f11install/initrd.img"
on_reboot = 'restart'
on_crash = 'restart'

The virtual machine’s name is “F11INSTALL”, memory is 2G, CPU number is 2, disk, kernel and ramdisk is prepared in the above steps.

Start this virtual machine and connect to this virtual machine’s console and complete the installation:

# xm create -c f11.install

The console can be released by “Ctrl+]”. And it can be reconnected by:

# xm console F11INSTALL

The installation of Fedora 11 will start. The gateway and DNS server should be set according to the network configuration.

The URL of installation source I used during installation is:

http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/x86_64/os/

After successfully installation of this virtual machine. It can be shut down by:

# xm shutdown F11INSTALL

This virtual machine can be duplicated to get more VMs: How to duplicate Xen DomU virtual machines.

Start DomU

Remember to update your system after the first boot. The 2.6.30 kernel is more stable running as DomU than the default one.

Create a profile vm-10.0.0.123.run for loading the virtual machine:

name="10.0.0.123"
vcpus=2
memory=2048
disk = ['file:/lhome/xen/vm-10.0.0.123/vmdisk0,xvda,w' ]
vif = [ 'bridge=eth0' ]
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the PyGrub (“/usr/bin/pygrub”) as the bootloader. PyGrub starts Linux DomUs with the kernels that lie in the filesystem of the DomU instead of the kernels that lie in the filesystem of the Dom0. That makes the kernel update and management easier.

The blktap2 VBD driver has better performance than blkback backed VBD. The blkback backed driver is used here for safety writing to the disk. The disk driver can be changed to use blktap2 driver later after updating the system. A blktap2 driver can be configured like this:

disk = ['tap:aio:/lhome/xen/f11install/vmdisk0,xvda,w' ]

Then the DomU can be started using this profile:

# xm create vm-10.0.0.123.run

The console of this DomU can be connected to:

# xm console vm-10.0.0.123

The console can be released by “Ctrl+]”.

• Setting up Stable Xen DomU with Fedora: Unmodified Fedora 12 on top of Xenified Fedora 12 Dom0 with Xen 4.0
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 4.0.1

$ wget http://bits.xensource.com/oss-xen/release/4.0.1/xen-4.0.1.tar.gz
$ tar xf xen-4.0.1.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries" -y; \
yum groupinstall "Development Tools" -y; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial -y

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 4.0.1 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-4.0.1.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=16384
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

Problems and solutions can be found here: Problems during Installing Xen Dom0 in Fedora

• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.2

$ wget http://bits.xensource.com/oss-xen/release/3.4.2/xen-3.4.2.tar.gz
$ tar xf xen-3.4.2.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries"; \
yum groupinstall "Development Tools"; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.2 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-3.4.2.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

Dom0 installation and configuration

Stable Dom0 solution (recommended):

Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

Latest Xen 4.0.1 Dom0 solution (Fedora 11 DomUs can’t boot on top of it):

Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

DomU installation and configuration

Unmodified Fedora DomU with pv_ops kernel:

Fedora 12 (recommended):

Setting up Stable Xen DomU with Fedora: Unmodified Fedora 12 on top of Xenified Fedora 12 Dom0 with Xen 4.0.0

Fedora 11:

Setting up Stable Xen DomU with Fedora: Unmodified Fedora 11 with pv_ops Kernel

Dom0 management

Dom0′s CPU and memory configuration advises:

Managing Xen Dom0′s CPU and Memory

DomU management

Create and manage virtual machines on Xen

How to Duplicate Xen DomU Virtual Machines

Automatically backing up Xen File-backed DomU

Unified Xen DomU configuration file

Problems

Some problems that may happen and their solution:

Problems during Installing Xen Dom0 in Fedora

• Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12

Xen solution

for the latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.31.12.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries"; \
yum groupinstall "Development Tools"; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.31.12

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.12.tar.bz2
$ tar xf linux-2.6.31.12.tar.bz2

Download 2.6.31 Xen patches v14

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-14.tar.bz2
$ mkdir xen-patches-2.6.31-14
$ tar xf xen-patches-2.6.31-14.tar.bz2 -C xen-patches-2.6.31-14

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.31-14/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-2.6.31.12-xenified

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 
Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 <*>     Network-device loopback driver (NEW)
 < >   PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 < >   SCSI backend driver (NEW)
 < >   USB backend driver (NEW)
 < > Block-device frontend driver
 < > Network-device frontend driver
 < > SCSI frontend driver (NEW)
 < > USB frontend driver (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (3.0.2 and later)  --->

Build kernel

$ make -j16

-jN: N may be 16 or other numbers depending on the number of processors in the system.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.31.12
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.31.12 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.31.12.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

We may need to enable VT and I/O virtualization in BIOS.

These options can be enabled in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Limited number of loop devices

The default number of loop device in this kernel is 8. When we are using blkback backed VBDs and we need to have more than 8 virtual machines, we should add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

The initramfs image under /root generated by dracut doesn’t work on some servers. If you have the similiar problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.32.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.32.13.img 2.6.32.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.32.13.img

to

module /initrd-2.6.32.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

If Linux kernel can’t boot even nomodeset has been set which may happen when use Intel i915 series hardware, a tricky solution is setting Xen’s vga option to let Xen use gfx console. This method can bypass the drm problem. Just replace “vga=ask” in Xen’s options with:

vga=gfx-1280x1024x32

to use 1280 x 1024 resolution with 32 bit color.

Xen’s graphical video mode codes are different with Linux Kernel’s. This is a list of Xen’s vga option from [1]:

 'vga=<mode-specifier>[,keep]' where <mode-specifier> is one of:

   'vga=ask':
      display a vga menu of available modes

   'vga=text-80x<rows>':
      text mode, where <rows> is one of {25,28,30,34,43,50,60}

   'vga=gfx-<width>x<height>x<depth>':
      graphics mode, e.g., vga=gfx-1024x768x16

   'vga=mode-<mode>:
      specifies a mode as specified in 'vga=ask' menu
      (NB. menu modes are displayed in hex, so mode numbers here must
           be prefixed with '0x' (e.g., 'vga=mode-0x0318'))

 The option 'keep' causes Xen to continue to print to the VGA console even
 after domain 0 starts to boot. The default behaviour is to relinquish
 control of the console to domain 0.

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.

You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

The clocksource=jiffies kernel parameter may be needed.

Error message related to ksm

If you get lots error messages generated by ksm, you can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

Xend conflicts with netplugd

Xend will report error when netplugd is started. Xend will print out a error message after about one minutes:

/etc/xen/scripts/network-bridge: line 240: "sigerr" command not found.

And the network interface of Linux doesn’t change. The new bridge can’t be started and the physical interface will be change name to pethx.

After disable netplugd, xend can start successfully:

# chkconfig netplugd off

References

[1] http://lists.xensource.com/archives/html/xen-changelog/2007-06/msg00206.html

Update history
Aug. 22, 2010. Add drm tricky and xen video mode; add netplugd caused problem.

• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12

IP address, MAC address and netmask

ifconfig will print out all the network interfaces and their information including the IP address and netmask.

$ ifconfig

A typical output is like this:

eth0      Link encap:Ethernet  HWaddr 00:26:22:A1:25:0F
inet addr:143.89.135.175  Bcast:143.89.135.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:305973 errors:0 dropped:0 overruns:0 frame:0
TX packets:337971 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:196287324 (187.1 MiB)  TX bytes:134890044 (128.6 MiB)
Interrupt:30

lo        Link encap:Local Loopback
inet addr:127.0.0.1  Mask:255.0.0.0
UP LOOPBACK RUNNING  MTU:16436  Metric:1
RX packets:274 errors:0 dropped:0 overruns:0 frame:0
TX packets:274 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:37244 (36.3 KiB)  TX bytes:37244 (36.3 KiB)

eth0 is the network interface. lo is the loopback device.

The IP address is shown in inet addr field.

DNS server, hosts file and DNS look up order

The DNS server for Linux to contact is stored in /etc/resolve.conf. The DNS server(s) can be listed by:

$ cat /etc/resolve.conf

A line like this specifies the DNS server:

nameserver 8.8.8.8

The hosts file for Linux is stored in /etc/hosts. The first two lines (127.0.0.1 and ::1) which map the localhost name shouldn’t be changed since localhost is used for interprocess communication.

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.1.8   mysite.mydomain mysite

The order of DNS look up is defined in /etc/nsswitch.conf. Whether the hosts file should be looked up first can be defined. The line in /etc/nsswitch.conf for DNS look up order is:

hosts:          dns [!UNAVAIL=return] files

Gateway

The gateway can be find out by looking at the rules in route table:

$ ip route show

A line like this defines the gateway:

default via 143.89.135.254 dev eth0  proto static

Host name

The host name is specified in /etc/sysconfig/network. The HOSTNAME variable is for the machine’s host name:

HOSTNAME=localhost.localdomain

• Configuration of Linux Kernel Video Mode
• Sending Email from mailx Command in Linux Using Gmail’s Smtp
• Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12

By now, suppose we have install kernel-2.6.32.16-143 in the Linux box and we want to install some older kernel in the repository.

First find out all the kernels in the repository by:

# yum list --showduplicates kernel

All the kernel packages will be listed like this:

Installed Packages
kernel.i686                        2.6.32.16-141.fc12               @updates
Available Packages
kernel.i686                        2.6.31.5-127.fc12                fedora
kernel.i686                        2.6.32.16-150.fc12               updates

We find out that 2.6.31.5-127.fc12 kernel package is available. We can install it now:

# yum install kernel-2.6.31.5-127.fc12

Then yum will install the kernel of older version. For other packages, the method is similar with kernel. We just need to replace kernel with the package name above.

• Problems during Installing Xen Dom0 in Fedora
• Installing Nvidia Driver on Fedora
• Creating Fedora Live USB Media