PKILL.info » dom0 http://pkill.info/b Solutions, tutorials and tips Thu, 09 Sep 2010 02:13:52 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://pkill.info/b/2405/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://pkill.info/b/2405/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Fri, 27 Aug 2010 00:55:23 +0000 Zhiqiang Ma http://pkill.info/b/?p=2405 Read more:]]> This is the latest stable and recommended stable Xen Dom0 solution on Fedora 12. No serious bug found till now and we will fix the bugs by ourselves if some appears.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.32.13. This is a very stable Dom0 solution for Fedora 12. Lot’s DomUs have been tested on this platform.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

You may need to install packages depended by this. You can try this for solving the dependencies:

#yum install make gcc -y; \
yum groupinstall "Development Libraries" -y; \
yum groupinstall "Development Tools" -y; \
yum install transfig texi2html \
dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial -y

The we can make Xen and Xen tools:

$ make xen
$ make tools

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number.

You may need to install patch if it isn’t installed:

# yum install patch

This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 < >   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 < >   PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -jX

X can be two times of the number of the processor. We use this to let make invoke compilation work in X-way parallel.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

Please refer to
Xen solution
for the DomU solution and more Xen Dom0 solution.

Update history
Sep. 3, 2010. blktap driver is used instead of blktap2; PCI backend driver is disabled.
Sep. 8, 2010. Add install patch; highlight PCI part in make menuconfig.

Read more:]]> http://pkill.info/b/2405/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 0 Setting up Stable Xen Dom0 with Fedora: Xen 4.0.1 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://pkill.info/b/2412/setting-up-stable-xen-dom0-with-fedora-xen-4-0-1-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://pkill.info/b/2412/setting-up-stable-xen-dom0-with-fedora-xen-4-0-1-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Thu, 26 Aug 2010 18:51:15 +0000 Zhiqiang Ma http://pkill.info/b/?p=2412 Read more:]]> Please refer to
Xen solution
for the more latest stable Xen Dom0 solution.
—————————————-
How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 4.0.1 from xen.org and Xenified Linux kernel 2.6.32.13. This solution have been tested quite stable in our cluster.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 4.0.1

$ wget http://bits.xensource.com/oss-xen/release/4.0.1/xen-4.0.1.tar.gz
$ tar xf xen-4.0.1.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries" -y; \
yum groupinstall "Development Tools" -y; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial -y

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 4.0.1 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-4.0.1.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=16384
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

Problems and solutions can be found here: Problems during Installing Xen Dom0 in Fedora


Read more:]]> http://pkill.info/b/2412/setting-up-stable-xen-dom0-with-fedora-xen-4-0-1-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 1 Setting up Stable Xen Dom0 with Fedora: Xen 3.4.2 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://pkill.info/b/2388/setting-up-stable-xen-dom0-with-fedora-xen-3-4-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://pkill.info/b/2388/setting-up-stable-xen-dom0-with-fedora-xen-3-4-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Wed, 25 Aug 2010 05:21:24 +0000 Zhiqiang Ma http://pkill.info/b/?p=2388 Read more:]]> Please refer to
Xen solution
for the more latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.2 from xen.org and Xenified Linux kernel 2.6.32.13. This is a very stable Dom0 solution for Fedora 12. Lot’s DomUs have been tested on this platform.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.2

$ wget http://bits.xensource.com/oss-xen/release/3.4.2/xen-3.4.2.tar.gz
$ tar xf xen-3.4.2.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries"; \
yum groupinstall "Development Tools"; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.2 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-3.4.2.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

Read more:]]> http://pkill.info/b/2388/setting-up-stable-xen-dom0-with-fedora-xen-3-4-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 0 Setting up Stable Xen Dom0 with Fedora: Xen 3.4.3 with Xenified Linux Kernel 2.6.31.12 in Fedora 12 http://pkill.info/b/2356/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-31-12-in-fedora-12/ http://pkill.info/b/2356/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-31-12-in-fedora-12/#comments Sun, 15 Aug 2010 16:05:54 +0000 Zhiqiang Ma http://pkill.info/b/?p=2356 Read more:]]> Please refer to

Xen solution

for the latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 3.4.3 from xen.org and Xenified Linux kernel 2.6.31.12.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 3.4.3

$ wget http://bits.xensource.com/oss-xen/release/3.4.3/xen-3.4.3.tar.gz
$ tar xf xen-3.4.3.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries"; \
yum groupinstall "Development Tools"; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.31.12

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.31.12.tar.bz2
$ tar xf linux-2.6.31.12.tar.bz2

Download 2.6.31 Xen patches v14

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.31-14.tar.bz2
$ mkdir xen-patches-2.6.31-14
$ tar xf xen-patches-2.6.31-14.tar.bz2 -C xen-patches-2.6.31-14

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.31-14/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-2.6.31.12-xenified

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 
Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 <*>     Network-device loopback driver (NEW)
 < >   PCI-device backend driver (NEW)
 < >   TPM-device backend driver (NEW)
 < >   SCSI backend driver (NEW)
 < >   USB backend driver (NEW)
 < > Block-device frontend driver
 < > Network-device frontend driver
 < > SCSI frontend driver (NEW)
 < > USB frontend driver (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (3.0.2 and later)  --->

Build kernel

$ make -j16

-jN: N may be 16 or other numbers depending on the number of processors in the system.

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 3.4.3 - Xenified Linux 2.6.31.12
  root (hd0,0)
  kernel /xen-3.4.3.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.31.12 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.31.12.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=2048
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

A list of common problems and tips can be found in Problems during Installing Xen Dom0 in Fedora.

Read more:]]> http://pkill.info/b/2356/setting-up-stable-xen-dom0-with-fedora-xen-3-4-3-with-xenified-linux-kernel-2-6-31-12-in-fedora-12/feed/ 1 Problems during Installing Xen Dom0 in Fedora http://pkill.info/b/2354/problems-during-installing-xen-dom0-in-fedora/ http://pkill.info/b/2354/problems-during-installing-xen-dom0-in-fedora/#comments Sun, 15 Aug 2010 11:58:36 +0000 Zhiqiang Ma http://pkill.info/b/?p=2354 Read more:]]> Here is a list of problem that may occur during installing and configuring Xen Dom0 in Fedora. It is found originally in Fedora systems, but the tips in this post should also be helpful for installing Xen Dom0 on other platforms.

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

We may need to enable VT and I/O virtualization in BIOS.

These options can be enabled in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Limited number of loop devices

The default number of loop device in this kernel is 8. When we are using blkback backed VBDs and we need to have more than 8 virtual machines, we should add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

The initramfs image under /root generated by dracut doesn’t work on some servers. If you have the similiar problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.32.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.32.13.img 2.6.32.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.32.13.img

to

module /initrd-2.6.32.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

If Linux kernel can’t boot even nomodeset has been set which may happen when use Intel i915 series hardware, a tricky solution is setting Xen’s vga option to let Xen use gfx console. This method can bypass the drm problem. Just replace “vga=ask” in Xen’s options with:

vga=gfx-1280x1024x32

to use 1280 x 1024 resolution with 32 bit color.

Xen’s graphical video mode codes are different with Linux Kernel’s. This is a list of Xen’s vga option from [1]:

 'vga=<mode-specifier>[,keep]' where <mode-specifier> is one of:

   'vga=ask':
      display a vga menu of available modes

   'vga=text-80x<rows>':
      text mode, where <rows> is one of {25,28,30,34,43,50,60}

   'vga=gfx-<width>x<height>x<depth>':
      graphics mode, e.g., vga=gfx-1024x768x16

   'vga=mode-<mode>:
      specifies a mode as specified in 'vga=ask' menu
      (NB. menu modes are displayed in hex, so mode numbers here must
           be prefixed with '0x' (e.g., 'vga=mode-0x0318'))

 The option 'keep' causes Xen to continue to print to the VGA console even
 after domain 0 starts to boot. The default behaviour is to relinquish
 control of the console to domain 0.

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.

You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

The clocksource=jiffies kernel parameter may be needed.

Error message related to ksm

If you get lots error messages generated by ksm, you can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

Xend conflicts with netplugd

Xend will report error when netplugd is started. Xend will print out a error message after about one minutes:

/etc/xen/scripts/network-bridge: line 240: "sigerr" command not found.

And the network interface of Linux doesn’t change. The new bridge can’t be started and the physical interface will be change name to pethx.

After disable netplugd, xend can start successfully:

# chkconfig netplugd off

References

[1] http://lists.xensource.com/archives/html/xen-changelog/2007-06/msg00206.html

Update history
Aug. 22, 2010. Add drm tricky and xen video mode; add netplugd caused problem.

Read more:]]> http://pkill.info/b/2354/problems-during-installing-xen-dom0-in-fedora/feed/ 0 Setting up Stable Xen Dom0 with Fedora: Xen 4.0.0 with Xenified Linux Kernel 2.6.32.13 in Fedora 12 http://pkill.info/b/2252/setting-up-stable-xen-dom0-with-fedora-xen-4-0-0-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/ http://pkill.info/b/2252/setting-up-stable-xen-dom0-with-fedora-xen-4-0-0-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/#comments Mon, 26 Jul 2010 16:39:39 +0000 Zhiqiang Ma http://pkill.info/b/?p=2252 Read more:]]> Please refer to
Xen solution
for the more latest stable Xen Dom0 solution.

How to set up Xen Dom0 with Xenified Linux kernel in Fedora 12 will be introduced in this post. We use Xen 4.0.0 from xen.org and Xenified Linux kernel 2.6.32.13. This solution have been tested quite stable in our cluster.

Hardware:

Dom0′s hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64

SELinux is disabled. Please refer here for detail: Disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update the system:

# yum update

The Xen and libvirt packages in Fedora should not be installed to avoid conflict.

# yum erase xen* libvirt

Build and install Xen hypervisor and tools

Download Xen 4.0.0

$ wget http://bits.xensource.com/oss-xen/release/4.0.0/xen-4.0.0.tar.gz
$ tar xf xen-4.0.0.tar.gz

Build Xen and tools

$ make xen
$ make tools

You may need to install packages depended by this. You can try this for solving the dependencies:

# yum groupinstall "Development Libraries" -y; \
yum groupinstall "Development Tools" -y; \
yum install transfig texi2html \
libaio-devel dev86 glibc-devel \
e2fsprogs-devel gitk mkinitrd \
iasl xz-devel bzip2-devel \
pciutils-libs pciutils-devel \
SDL-devel libX11-devel gtk2-devel \
bridge-utils PyXML qemu-common \
qemu-img mercurial -y

Install Xen and tools

$ make install-xen
$ make install-tools

Build and install xenified Linux kernel

Download Linux kernel 2.6.32.13

$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.13.tar.bz2
$ tar xf linux-2.6.32.13.tar.bz2

Download 2.6.32 Xen patches v2

$ wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.32-2.tar.bz2
$ mkdir xen-patches-2.6.32-2
$ tar xf xen-patches-2.6.32-2.tar.bz2 -C xen-patches-2.6.32-2

Apply Xen patches

Apply all the patches downloaded above following the patch number. This patch.sh script can be used (we assume the patch and the kernel are in the same directory):

patch.sh:

#!/bin/bash
for P in `ls ../xen-patches-2.6.32-2/6*.patch1 | sort`
do
    patch -p1 -s -i $P
    if [ $? = 0 ]; then
        echo $P applied
    else
        echo "Error processing "$P
        exit 1
    fi
done

Put this script into Linux source directory and execute:

$ sh ./patch.sh

Configure Xenified Linux kernel

A working configuration file that I used can be downloaded directly from here:

config-for-xenified-linux-2.6.32.13

Just download this file, put it into the kernel source code file directory and rename it to .config .

Other than use my configuration file, you can also configure it by yourself by using “make menuconfig”.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
 [*] Symmetric multi-processing support
 [*] Support sparse irq numbering
 [*] Enable Xen compatible kernel
 Preemption Model (No Forced Preemption (Server))  --->

Device Drivers  --->
 XEN  --->
 [*] Privileged Guest (domain 0)
 <*> Backend driver support (NEW)
 <*>   Block-device backend driver (NEW)
 <*>   Block-device tap backend driver (NEW)
 <*>   Block-device tap backend driver 2 (NEW)
 <*>   Network-device backend driver (NEW)
 (8)     Maximum simultaneous transmit requests (as a power of 2) (NEW)
 [ ]     Pipelined transmitter (DANGEROUS) (NEW)
 < >     Network-device loopback driver (NEW)
 <*>   PCI-device backend driver (NEW)
 PCI Backend Mode (Virtual PCI)  --->
 [ ]     PCI Backend Debugging (NEW)
 < >   TPM-device backend driver (NEW)
 <M>   SCSI backend driver (NEW)
 <M>   USB backend driver (NEW)
 <M> Block-device frontend driver
 <M> Network-device frontend driver
 <M>   Network-device frontend driver acceleration for Solarflare NICs (NEW)
 <M> SCSI frontend driver (NEW)
 <M> USB frontend driver (NEW)
 [*]   Taking the HCD statistics (for debug) (NEW)
 [ ]   HCD suspend/resume support (DO NOT USE) (NEW)
 <*> User-space granted page access driver (NEW)
 <*> Framebuffer-device frontend driver (NEW)
 <*>   Keyboard-device frontend driver (NEW)
 [*] Disable serial port drivers (NEW)
 <*> Export Xen attributes in sysfs (NEW)
 (256) Number of guest devices (NEW)
 Xen version compatibility (no compatibility code)  --->
 [*] Place shared vCPU info in per-CPU storage (NEW)

Build kernel

$ make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry for Xen in /boot/grub/grub.conf. This is an example entry:

title Xen 4.0.0 - Xenified Linux 2.6.32.13
  root (hd0,0)
  kernel /xen-4.0.0.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.32.13.img

The root and other parameters may be different depending on the configuration.

Make Xend and Xendomains services automatically start when system boots

# cd /etc/init.d/
# chkconfig --add xend
# chkconfig --add xendomains

Check whether Xend and Xendomains services are automatically started in level 3-5:

# chkconfig --list | grep xend

It should be like this:

xend               0:off    1:off    2:off    3:on    4:on    5:on    6:off
xendomains         0:off    1:off    2:off    3:on    4:on    5:on    6:off

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

We may need to enable VT and I/O virtualization in BIOS.

These options can be enabled in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Enjoy the fun now!

After booting the system, you can try to use xm to check xen info

# xm info

Then xm command can be used to start up DomUs.

This is one working configuration file for one DomU that I use:

name="10.0.1.201"
vcpus=2
memory=16384
disk = ['tap:aio:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
# disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'

Here we use the blktap backed VBD device which has much better performance than Linux blkback backed VBD device.

Making the performance more stable

Allocating dedicated CPU core and memory for Dom0 may provide more stable performance for the Xen platform. Please refer to Managing Xen Dom0′s CPU and Memory/ for detailed instruction.

Problems

Here is a list of problem that may occur during the configuration.

Limited number of loop devices

The default number of loop device in this kernel is 8. When we are using blkback backed VBDs and we need to have more than 8 virtual machines, we should add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

The initramfs image under /root generated by dracut doesn’t work on some servers. If you have the similiar problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.32.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.32.13.img 2.6.32.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.32.13.img

to

module /initrd-2.6.32.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.32.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.

You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

The clocksource=jiffies kernel parameter may be needed. Please refer to Larry Matter’s reply below. He installed Xen with 32bit kernel successfully.

Error message about ksm

If you get lots error messages generated by ksm, you can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

More problems and solutions can be found here: Problems during Installing Xen Dom0 in Fedora

Update history
Aug. 10, 2010. Add clocksource kernel parameter for 32bit; Add dependency list;

Read more:]]> http://pkill.info/b/2252/setting-up-stable-xen-dom0-with-fedora-xen-4-0-0-with-xenified-linux-kernel-2-6-32-13-in-fedora-12/feed/ 6 Managing Xen Dom0′s CPU and Memory http://pkill.info/b/2258/managing-xen-dom0s-cpu-and-memory/ http://pkill.info/b/2258/managing-xen-dom0s-cpu-and-memory/#comments Mon, 26 Jul 2010 15:12:54 +0000 Zhiqiang Ma http://pkill.info/b/?p=2258 Read more:]]> The performance of Xen’s Dom0 is important for the overall system. The disk and network drivers are running on Dom0. I/O intensive guests’ workloads may consume lots Dom0′s CPU cycles. The Linux kernel calculates various network related parameters based on the amount of memory at boot time. The kernel also allocate memory for storing memory metadata (per page info structures) is also based on the boot time amount of memory. After ballooning down Dom0′s memory, the network related parameters will not be correct. Ballooning down busy Dom0′s memory sometimes cause SSH to die from our observation, which is a nightmare for the administrator since SSH is usually the only way for remote control of the server. Another bed effect is that it’s a waste of memory with a large memory metadata for a smaller memory amount.

Now let’s look at how to menage Xen Dom0′s CPU and memory in a better way.

Dedicate a CPU core for Dom0

Dom0 will have free CPU time to process the I/O requests from the DomUs if it has dedicated CPU core(s). Better performance may be achieved by this since there are less CPU context switches to do in Dom0.

We can dedicate CPU core for Dom0 by passing “dom0_max_vcpus=X dom0_vcpus_pin” options to Xen hypervisor (xen.gz) in /boot/grub/grub.conf. X is the number of vcpus dedicated to Dom0.

As hyperthreading technology is enabled in most modern CPUs, we need to specify two processors to dedicate one CPU core. So the “X” above should usually be 2 for one CPU core.

kernel /xen.gz console=vga vga=ask noreboot dom0_max_vcpus=2 dom0_vcpus_pin

After booting the system, the VCPU list can be got on Dom0 by this command:

# xm vcpu-list

Even after booting the system, the VCPU number can be configured by xm command. We can set Domain-0 have two VCPUs and processor 0 and 1 to be dedicated to Dom0 by these commands:

# xm vcpu-set Domin-0 2
# xm vcpu-pin Domain-0 0
# xm vcpu-pin Domain-0 1

Dedicate memory for Dom0

We should always dedicate fixed amount of memory for Xen Dom0.

We can set the initial memory size of Dom0 by passing “dom0_mem=xxx” (in KB) option to Xen hypervisor (gen.gz) in /boot/grub/grub.conf. “xxx” is the amount of memory for Dom0 in KB.

If we set the initial memory size of Dom0 to 2GB, just change the entry in grub.conf to:

kernel /xen.gz console=vga vga=ask noreboot dom0_max_vcpus=2 dom0_vcpus_pin dom0_mem=2097152

Set lowest permissible memory for Dom0

The option dom0-min-mem in Xend configuration file /etc/xen/xend-config.sxp is used to specify the lowest permissible memory for Dom0.

The value of dom0-min-mem (in MB) is the lowest permissible memory level for Dom0. The default value is 256. If we limit the memory size of Dom0 to 2G, just set:

(dom0-min-mem 2048)

Preventing dom0 memory ballooning

The “enable-dom0-ballooning” option in Xend configuration file is used to specify whether Dom0′s memory can be ballooned out. Setting “enable-dom0-ballooning” to “no” will make sure Xend never takes any memory away from Dom0:

(enable-dom0-ballooning no)
Read more:]]> http://pkill.info/b/2258/managing-xen-dom0s-cpu-and-memory/feed/ 2 Setting up Xen pvops Dom0 on Fedora : Xen 3.4.2 + Kernel 2.6.31 with paravirt_ops in Fedora 12 http://pkill.info/b/1535/setting-up-xen-pvops-dom0-on-fedora-xen-3-4-2-kernel-2-6-31-with-paravirt_ops-in-fedora-12/ http://pkill.info/b/1535/setting-up-xen-pvops-dom0-on-fedora-xen-3-4-2-kernel-2-6-31-with-paravirt_ops-in-fedora-12/#comments Tue, 04 May 2010 08:20:28 +0000 Zhiqiang Ma http://zhiqiangma.info/?p=1535 Read more:]]> Please refer to

Xen solution

for the latest stable Xen Dom0 solution.

How to set up Xen Dom0 with paravirt_ops technology in Fedora 12 will be introduced in this post. The Fedora 12 release has xen 3.4.2 which support paravirt_ops Dom0 and Domu but lack of Dom0 kernel. We use xen 3.4.2 in Fedora 12′s repository and build a paravirt_ops enabled Dom0 kernel by ourseves.

Hardware:

Dom0 hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 8 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Linux system:

Fedora 12 x86_64
SELinux is disabled. Please refer here for detail: disabled SELinux on Fedora.

ext3 is recommended for the file system of disk partition for /boot.

Update system:

# yum update

Install xen

Install xen-3.4.2 from Fedora 12′s repository:

# yum install xen*

This is the packages that we have installed:

[root@localhost xen]# rpm -qa | grep xen
xen-hypervisor-3.4.2-1.fc12.x86_64
xen-3.4.2-1.fc12.x86_64
xenner-0.47-3.fc12.x86_64
xen-libs-3.4.2-1.fc12.x86_64
xenwatch-0.5.4-3.fc12.x86_64
xen-doc-3.4.2-1.fc12.x86_64
xen-runtime-3.4.2-1.fc12.x86_64
xen-devel-3.4.2-1.fc12.x86_64

Install paravirt_ops technology enabled kernel

Download kernel 2.6.31.13

The kernel I used can be downloaded from here:

Dowload Dom0 kernel 2.6.31.13

This kernel is downloaded from Jeremy’s git repository on Apr. 30, 2010:

http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git

Configuraton

The configuration file that I used can be downloaded from here:

Download config-kernel-xen-2.6.31.13

Just download this file, put it into the kernel source code file and rename it to .config .

You can also configure it by yourself using “make menuconfig” if you don’t want to use my configuration file or you need more components to be enabled.

Make sure you build the kernel with these components enabled:

Processor type and features  --->
--- Paravirtualized guest support
[*]   Xen guest support
(32)  Maximum allowed size of a domain in gigabytes
[*]   Enable Xen debug and tuning parameters in debugfs
[*]   Enable Xen privileged domain support
[ ]   Enable support for Xen PCI passthrough devices
[*]   KVM paravirtualized clock
[*]   KVM Guest support
-*-   Enable paravirtualization code
[ ]     Paravirtualization layer for spinlocks

Device Drivers --->
[*] Xen memory balloon driver
[*]   Scrub pages before returning them to system
<*> Xen /dev/xen/evtchn device
[*] Backend driver support
<*>   Block-device backend driver
<*>   Block-device tap backend driver
<*>   Xen backend network device
<*>   PCI-device backend driver
       PCI Backend Mode (Virtual PCI)  --->
         [ ]     PCI Backend Debugging
<*> Xen filesystem
     [*]   Create compatibility mount point /proc/xen
[*] Create xen entries under /sys/hypervisor

Build kernel

# make -j16

Install modules and kernel

# make modules_install install

Configure grub

Add one entry in /boot/grub/grub.conf:

title Xen 3.4.2 - Fedora (2.6.31.13)
  root (hd0,0)
  kernel /xen-3.4.2.gz console=vga vga=ask noreboot
  module /vmlinuz-2.6.31.13 ro root=/dev/mapper/VolGroup-LogVol_root max_loop=32 noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
  module /initramfs-2.6.31.13.img

The root and other parameters may be different depending on the configuration.

Configure xend

# cd /etc/xen
# cp xend-config.sxp xend-config.sxp.bak
# vim xend-config.sxp

Find:

#(network-script network-bridge)
(network-script /bin/true)

Change it to:

(network-script network-bridge)
#(network-script /bin/true)

After changing the configure file, reboot the system and choose xen + fedora entry which I have added in grub.

BIOS configuration

If xen stops at:

“I/O virtualization disabled.”

Enable VT and I/O virtualization in BIOS.

You may enable these in BIOS:

Intel (R) Virtualization Technology
Intel (R) VT for Directed I/O
Interrupt Remapping
Coherency Support
ATS Support

Enjoy it!

After booting the system, you can try to use xm to check xen info

[root@localhost xen]# xm info
host                   : localhost.localdomain
release                : 2.6.31.13
version                : #1 SMP Tue May 4 15:04:27 HKT 2010
machine                : x86_64
nr_cpus                : 16
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 32705
free_memory            : 3158
node_to_cpu            : node0:0-15
node_to_memory         : node0:3158
xen_major              : 3
xen_minor              : 4
xen_extra              : .2
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.4.2 20091027 (Red Hat 4.4.2-7) (GCC)
cc_compile_by          : mockbuild
cc_compile_domain      : (none)
cc_compile_date        : Fri Dec 11 05:58:50 EST 2009
xend_config_format     : 4

Then xm command can be used to set up Domus.

This is the configuration file for one DomU that I use. It works well:

name="vm-10.0.1.201"
memory=16384
disk = ['file:/lhome/xen/vm-10.0.1.201/vmdisk0,xvda,w' ]
vif = ['bridge=eth0']
bootloader = "/usr/bin/pygrub"
vcpus=2
on_reboot = 'restart'
on_crash = 'restart'

Here the loopback backed vbd device is used.

Making the performance more stable

I have test both Dom0 and DomU. The performance is stable but not as stable as the native Linux kernel. I saying not quite stable means that if we run a piece of code and we may find the time for it is a bit different from time to time. The difference may be up to 10%.

If you need a stable, you can configure xend to make Dom0 use one CPU. The Dom0 will change to UP mode. There are less virtual CPUs that physical CPUs. I don’t know the exact reason by now. But this method will make the performance stable.

We have also tested this solution. Both Dom0 and DomU (SMP mode of course) have as stable performance as native Linux.

The configuration:

# vim /etc/xen/xend-config.sxp

Find

(dom0-cpus 0)

Then change it to

(dom0-cpus 1)

Dom0 memory management

Initial memory size of Dom0

We can set the initial memory size of Dom0 by adding option dom0_mem (in KB) to xen in grub.conf.

If we set the initial memory size of Dom0 to 1GB, just change the line in grub entry to:

kernel /xen-3.4.2.gz console=vga vga=ask noreboot dom0_mem=1048576

Lowest permissible memory for dom0

The value dom0-min-mem (in MB) is the lowest permissible memory level for Dom0. The default value is 256. If we limit the memory size of Dom0 to larger than 1G, just set:

(dom0-min-mem 1024)

Problems

Limited number of loop devices

The default number of loop device in this kernel is 8. When we need to have more than 8 virtual machines, we need to add more loop devices. You need to use the first method (pass parameter max_loop=32 to vmlinuz) if you use my kernel configuration file.

initramfs related problem

We have found the initramfs image under /root generated by dracut doesn’t work on one of our servers (Use Xeon 5700 CPU). If you have this problem, you can try to use image generated by mkinitrd:

1) Generate initrd-2.6.31.13.img using mkinitrd

mkinitrd /boot/initrd-2.6.31.13.img 2.6.31.13

2) Edit entry in /boot/grub/grub.conf

Change

module /initramfs-2.6.31.13.img

to

module /initrd-2.6.31.13.img

drm related problem

On one of our servers that uses radeon card we have experienced problem related to drm. The system crashes after the kernel printing out information about drm. We can add nomodeset option to kernel command line to bypass this problem.

The kernel command line in /boot/grub.conf will change to:

module /vmlinuz-2.6.31.13 ro root=/dev/mapper/VolGroup-LogVol_root nomodeset max_loop=32 noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

Build kernel on 32bit platform

You first need to enable PAE support if you’re building 32 bit version of the kernel. Xen only supports 32 bit PAE kernels nowadays. Xen kernel build options won’t show up if you don’t enable PAE for 32 bit builds.
You can enable PAE in “Processor type and features -> High Memory Support (64GB) -> PAE (Physical Address Extension) Support”

I can find Xen options after enable PAE on my laptop. I have never run Xen Dom0 on a 32bit kernel. I can’t say whether it is stable. Please try it and I will appreciate it if you share your result with me ;)

Error message about ksm

We can disable service ksm and ksmtuned to eliminate these error messages:

# chkconfig ksm off
# chkconfig ksmtuned off

Update history:
Updated on Jun 18. 2010. Correct the error in hardware configuration. The memory is 24GB.
Updated on Jun 12. 2010. Add drm related problem.
Updated on May 5. 2010. Add download link of kernel 2.6.31.13.
Updated on May 5. 2010. Add info that SELinux is disabled.
Updated on May 6. 2010. Add info that the /boot partition should in ext3 format rather than ext4.
Updated on May 6. 2010. Add info about dracut generated initramfs’s problem on one of our platform.
Updated on May 6. 2010. Add method to add more loop devices for setting up more than 8 virtual machines.
Updated on May 6. 2010. Add info that PAE should be enabled first before building 32bit Dom0 kernel.
Updated on May 10, 2010. Add the Dom0 memory management part.
Updated on May 16, 2010. Add ksm related problem.

Read more:]]> http://pkill.info/b/1535/setting-up-xen-pvops-dom0-on-fedora-xen-3-4-2-kernel-2-6-31-with-paravirt_ops-in-fedora-12/feed/ 2 Setting up Xen Dom0 on Fedora : Xen 3.4.1 with Linux Kernel 2.6.29 on Fedora 12 http://pkill.info/b/1566/setting-up-xen-dom0-on-fedora-xen-3-4-1-with-linux-kernel-2-6-29-on-fedora-12/ http://pkill.info/b/1566/setting-up-xen-dom0-on-fedora-xen-3-4-1-with-linux-kernel-2-6-29-on-fedora-12/#comments Mon, 12 Apr 2010 13:22:57 +0000 Zhiqiang Ma http://zhiqiangma.info/?p=1566 Read more:]]> Please refer to

Xen solution

for the latest stable Xen Dom0 solution.
==========================================

In this post, the detailed tutorial for setting up Xen 3.4.1 dom0 on top of Fedora 12 with kernel 2.6.29 will be introduced.

Hardware:

Dom0 hardware platform:

Motherboard: INTEL S5500BC S5500 Quad Core Xeon Server Board
CPU: 2 x Intel Quad Core Xeon E5520 2.26G (5.86GT/sec,8M,Socket 1366)
Memory: 4 x Kingston DDR-3 1333MHz 4GB ECC REG. CL9 DIMM w/Parity & Thermal Sensor
HD: 4 x WD WD10EARS 1 TB, SATA II 3Gb/s, 64 MB Cache

Disk Partitions:

[root@localhost vm0]# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-LogVol_root
 ext4     50G   12G   38G  23% /
tmpfs        tmpfs    7.9G     0  7.9G   0% /dev/shm
/dev/sda1     ext3    504M  110M  369M  23% /boot
/dev/mapper/VolGroup-LogVol_lhome
 ext4    858G   21G  794G   3% /lhome

Linux:

Fedora 12 x86_64, updated to Apr. 10, 2010.

No xen and libvirt installed:

# rpm -qa | grep xen
# rpm -qa | grep virt

And SELinux is diabled.

Packages:

Xen 3.4.1: http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz

linux-2.6.29-xen-r4-aka-suse-xenified-2.6.29-62.1: Download here.

download these packages and unzip them.

Build and install Xen

Uncompress xen and put all the source code files of xen to “/usr/src/xen/xen-3.4.1/”:

# mkdir -p /usr/src/xen/
# tar -xf xen-3.4.1.tar.gz -C /usr/src/xen/
# cd /usr/src/xen/xen-3.4.1

Build xen, tools and docs:

# make -j16 dist-xen dist-tools dist-docs

You may need to install some packages that are required and listed.

Install xen:

# cd dist
# sh ./install.sh

Build and install Linux kernel

The .config file I used can be downloaded from here:

https://sites.google.com/site/ericzqma/linux/kernel_config_xen_zma?attredirects=0

My .config file can be used directly.

Or please use menuconfig to create one:

# make menuconfig

When using “make menuconfig“, some options require special attention:

Subarchitecture Type (Enable Xen compatible kernel)
( ) PC-compatible
(X) Enable Xen compatible kernel
( ) Support for ScaleMP vSMP
Device Drivers --->
XEN --->
[*] Privileged Guest (domain 0)
<*>Backend driver support
<*>Block-device backend driver
<*>Block-device tap backend driver
<*> Network-device backend driver

After configuration, build and install this Linux kernel:

# make -j16
# make modules_install install

Create a grub entry

# vim /boot/grub/grub.conf

Then add this entry:

title Fedora (xen 3.4.1 - 2.6.29-xen-r4)
 root (hd0,0)
 kernel /xen-3.4.1.gz console=vga vga=ask noreboot
 module /vmlinuz-2.6.29-xen-r4 ro root=/dev/mapper/VolGroup-LogVol_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
 module /initramfs-2.6.29-xen-r4.img

The “root=” and other options will be different depending on the partitions of the system.

Configure services

Add xend and xendomains to services that automatically start when system boot.

# chkconfig --add xend
# chkconfig --add xendomains
# chkconfig xend on
# chkconfig xendomains on
# chkconfig --list | grep xend
xend            0:off   1:off   2:on   3:on    4:on    5:on    6:off
xendomains      0:off   1:off   2:on   3:on    4:on    5:on    6:off

Disable ksmtuned service.

# chkconfig ksmtuned off

Reboot now

# reboot

After booting the system in the xen environment. the xm command can be used:

[root@localhost ~]# xm info
host                   : localhost.localdomain
release                : 2.6.29-xen-r4
version                : #1 SMP Mon Apr 12 00:41:42 HKT 2010
machine                : x86_64
nr_cpus                : 16
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 16321
free_memory            : 4
node_to_cpu            : node0:0-15
node_to_memory         : node0:4
xen_major              : 3
xen_minor              : 4
xen_extra              : .1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.4.3 20100127 (Red Hat 4.4.3-4) (GCC)
cc_compile_by          : root
cc_compile_domain      : localdomain
cc_compile_date        : Mon Apr 12 01:53:45 HKT 2010
xend_config_format     : 4

Then xm can be used to create and manage virtual machines on the server.

[root@localhost ~]# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 15093    16     r-----    228.2
vm0                                          1  1024     2     -b----      4.4
Read more:]]> http://pkill.info/b/1566/setting-up-xen-dom0-on-fedora-xen-3-4-1-with-linux-kernel-2-6-29-on-fedora-12/feed/ 0 A simple CPU and memory performance test of xen Dom0 and DomU http://pkill.info/b/615/a-simple-cpu-and-memory-performance-test-of-xen-dom0-and-domu/ http://pkill.info/b/615/a-simple-cpu-and-memory-performance-test-of-xen-dom0-and-domu/#comments Thu, 14 Jan 2010 07:25:33 +0000 Zhiqiang Ma http://zhiqiangma.info/a-simple-cpu-and-memory-performance-test-of-xen-dom0-and-domu/ Read more:]]> Please refer to here for the platform of this test.

I have done some simple performance test on DomU and Dom0 and compare with the performance on physical machines. These test are simple, but it can provides some performance factor of xen. I test the CPU bound, memory write and memory read performance of Dom0, DomU and compare them with the result of the test on physical machine. Each test are done for 50 times and the the Average (E), Standard deviation (SD) and SD/E are calculated.

Test method:

I only past the C code here because I think it is clear enough.

1) CPU bound:
The code:

  const int test_limit = 100000;
  cout << "-----------------------------" << endl;
  cout << "cpu bound test begins." << endl;
  clock_t begin_time = clock();
  register int a = 0;
  for (register int i = 0; i < test_limit; i++) {
    for (register int j = 0; j < test_limit; j++) {
      a += i + j;
    }
  }
  clock_t end_time = clock();
  do_nothing(a);
  int time = (end_time - begin_time) / (CLOCKS_PER_SEC / 1000);

2) Memory read:

The code:

  const int test_times = 1000000;
  const int data_size = 1000000000;
  const int data_interval = 1000000;
  int* array_read = new int[data_size];
  cout << "----------------------" << endl
       << "Memory read bound test begins." << endl;

  register int read_value = 0;
  long read_begin =0;
  long read_end = 0;
  int data_range = data_size - test_times;

  read_begin = clock();
  for (register int t = 0; t < test_times; t++) {
    for (register int i = 0; i < data_range; i += data_interval) {
      read_value = array_read[i + t];
    }
  }

  read_end = clock();
  do_nothing(read_value);
  int time = (read_end - read_begin) / (CLOCKS_PER_SEC / 1000);

3) Memory write:

The code:

  const int test_times = 1000000;
  const int data_size = 1000000000;
  const int data_interval = 1000000;

  int* array_write = new int[data_size];

  cout << "----------------------" << endl
       << "Memory write bound test begins." << endl;

  long write_begin = 0;
  long write_end = 0;
  int data_range = data_size - test_times;
  write_begin = clock();

  for (register int t = 0; t < test_times; t++) {
    for (register int i = 0; i < data_range; i += data_interval) {
      array_write[i + t] = 528283;
    }
  }

  write_end = clock();
  int time = (write_end - write_begin) / (CLOCKS_PER_SEC / 1000);

The result:

Performance result on the physical machine:


 Physical machines test fedora kernel:

cpu memory read memory write
Average: 13257.6 21449.6 22243.8
Standard deviation: 4.72 17.2 18.96
SD/E: 0 0 0




Perf/Perf on Phy: 100.00% 100.00% 100.00%

Performance result on Dom0:


 Dom0 suse kernel test

cpu memory read memory write
Average: 13283 23059 23856.6
Standard deviation: 4.58 22.38 18.07
SD/E: 0 0 0




Perf/Perf on Phy: 99.81% 93.02% 93.24%

Performance result on DomU:


 DomU fedora kernel test

cpu memory read memory write
Average: 13307.6 23667.8 24459.2
Standard deviation: 13.2 33.96 37.19
SD/E: 0 0 0




Perf/Perf on Phy: 99.62% 90.63% 90.94%

From the test, we can see that the CPU performance is nearly 100%. But the memory performance is not so good especially for the DomU. This maybe because of the memory validation of Xen. But the CPU and memory performance of Xen VM is pretty good as a virtual machine.

Read more:]]> http://pkill.info/b/615/a-simple-cpu-and-memory-performance-test-of-xen-dom0-and-domu/feed/ 3