Network Boot and Diskless Workstations


On the previous page, we've seen how we can set up a network boot server comprised of a dhcp server and a tftp server. We've set this up to serve a syslinux boot loader to the clients, so the can boot off the server. The example we used boots a Debian installer.

It's also possible to boot an operating system this way, not just an installer. This allows you to run a diskless client that you may want to use as a thin client for server-based computing. An other application might be that you just boot the operating system off the server (for centralized OS management), but run applications locally, or use a local filesystem for specific user files.

Requirements

What follows has been tested with Ubuntu 7.10 Server, starting from a minimal install, and adding software as needed. Using ubuntu server as boot server makes it easier to create diskless ubuntu client systems, because you can use debootstrap to create the client filesystem (see further).

the root filesystem

The root filesystem is the filesystem the client will use after the operating system has booted. This could be any filesystem you can mount after the kernel has booted, but for diskless workstation, there are roughly 2 ways to accomplish this.

The most common approach is to provide a filesystem via NFS (Network File System). This means you have to set up an nfs server that serves a complete filesystem for the clients.

A completely different, and far less common approach is to let the diskless workstations use a RAM drive. When a linux kernel boots, it initially uses a RAM drive with a minimal file system. This filesystem is extracted from an archive, the 'initrd' (initial ram drive). The initrd contains the files the operating system needs to boot, and is abandonned when the root filesystem is mounted. It is possible to modify the initrd or create a custom archive, and let the kernel use that as root file system. In doing so, the entire root file system is served to the clients as an archive file (via tftp), and there is no need to set up nfs. The size of the initrd, however, is limited, so this will only work for minimal client systems, e.g. a client that only needs to start terminal sessions on the server.

We'll look into the nfs approach first, as this is more common and universal.

Setting up a client filesystem with nfs

create, on the server, a directory that will contain the client root filesystem, e.g. /srv/clientrootfs. Then, set up nfs on the server so that it exports this directory to the client network. No need to do any client configuration here. Specify in /etc/exports that this directory will be exported 'no_root_squash' : the kernel will need root access to these files. 'no_subtree_check' increases boot speed because the wait for nfs is shorter. And the filesystem needs to be read/write.

	## /etc/exports ##
	/srv/clientrootfs 192.168.126.0/255.255.255.0(rw,no_subtree_check,no_root_squash)
	

exportfs -ra to activate the export.

Keep in mind that '/srv/clientrootfs' on the server will become '/' to the clients.

The next problem is to get a working operating system inside /srv/clientrootfs. There are a number of ways to do this : You can copy or rsync an existing filesystem into /srv/clientrootfs. You can use debootstrap to create a (minimal) Debian system inside /srv/clientrootfs. You can extract a live CD to /srv/clientrootfs.

As an example, we'll use debootstrap. First, install it with apt-get install debootstrap. Then, run debootstrap with parameters to indicate into which directory the system will be installed, and the mirror server that provides the required packages (I use a local mirror with apt-proxy, but you can also just use http://archive.ubuntu.com/ubuntu/ )

	debootstrap /srv/clientrootfs http://reposrv/ubuntu
	

modifying the client filesystem

'/srv/clientrootfs' on the server will become '/' to the clients. In order to modify it, we use 'chroot' on the server to descend into /srv/clientrootfs and work in it as if it were /. This allows you to run standard tools to modify what will become the root filesystem to the clients.

One of the things you need to add is a kernel. You may want to add some additional software, edit a configuration file such as /etc/apt/sources.list, create a user account .... :

	chroot /srv/clientrootfs/
		apt-get install linux-image					#install a kernel
		echo "disklessclient01" > /etc/hostname		#set future client's hostname
		adduser luser
	exit
	

setting up the bootloader, the kernel, and the initial filesystem

installing linux-image has installed a kernel and created an initial ramdrive image in /srv/clientrootfs/boot/. These need to be copied to /srv/tftp so that the tftp server can serve them to the diskless clients when they try to boot.

	cp /srv/clientrootfs/boot/vmlinuz-2.6.24-16-generic /srv/tftp/vmlinuz
	cp /srv/clientrootfs/boot/initrd.img-2.6.24-16-generic /srv/tftp/initrd.img
	

Next, the pxelinux.0 bootloader needs to be set up to boot this kernel. We'll reuse the pxe bootloader we copied out of the debian netinstall CD, but replace it's configuration file. You can also get pxelinux the /install/netboot directory of an Ubuntu installation CD or from http://archive.ubuntu.com/ubuntu/dists/<release_codename>/main/installer-i386/current/images/netboot/. There's no need to install it, just unpack/copy the pxelinux.0 file and pxelinux.cfg directory into your tftp root directory.

	cd /srv
	rm tftp/pxelinux.0 tftp/pxelinux.cfg		#remove obsolete links
	
	## reuse debian-installer's pxe
	cp -p tftp/debian-installer/i386/pxelinux.0 tftp/
	cp -pr tftp/debian-installer/i386/pxelinux.cfg tftp/
	
	## remove old debian-installer directory, and see the results
	ls tftp/
				initrd.img  pxelinux.0  pxelinux.cfg  vmlinuz
	

The pxelinux.cfg/default file tells pxelinux.0 which kernel to boot, and what parameters to pass to the kernel. Edit the file so it looks like this :

	DEFAULT netboot
	TIMEOUT 0
	PROMPT 0

	LABEL netboot
        kernel vmlinuz
        append initrd=initrd.img ip=dhcp --
    

Power on the diskless client and see if it boots. If all is well, you'll see the kernel boot, but it will hang or fail as it can't mount a root filesystem (you may be dropped to a busybox emergency shell). That's because we haven't told it where to find a root filesystem. Modify pxelinux.cfg/default as follows to fix this.

	DEFAULT netboot
	TIMEOUT 0
	PROMPT 0

	LABEL netboot
        kernel vmlinuz
        append initrd=initrd.img ip=dhcp root=/dev/nfs nfsroot=192.168.1.10:/srv/clientrootfs --
	

Now, reboot the diskless client. You'l see it boot, setup its network interface, and load the nfs filesystem. You can login as root (no password), or with the user account you created earlier.

Notes, caveats and pitfalls

We've established that we can create a diskless workstation that boots of a network server and uses a network filesystem exported by an nfs server. Note that the client system can be modified by chrooting into the exported directory on the server.

The client system will have no swap area. If this is required, it's easiest to use a local hard disk. There exist some patches to enable swapping over nfs (eg http://www.nfs-swap.dot-heine.de/) - JFGI, YMMV.

You may want to prevent the clients of reconfiguring their network settings, because they rely heavily on configuration provided by dhcp and the filesystem served over nfs, and you don't want those things changed. Disabling networking (by removing or renaming [/srv/clientrootfs]/init.d/networking) might be a good idea.

The way we've set things up here, all diskless clients will get to use the same filesystem. You may want to create additional nfs exports for /home or other directories. As Linux uses a unified directory tree where you can mount several filesystems, it is equally possible to mount local filesystems as well. For instance, you may have a separate nfs server that exports user home directories that you'll mount to /home, or mount a /local/usr/bin from the workstation's hard disk in combination with the operating system provided by the server. Other interesting directories to mount separately (nfs or local) are /usr or /var. This can be done by having the clients execute a 'mount' statement after the root filesystem has been mounted (e.g. by including the an appropriate startup script in /srv/clientrootfs/etc/init.d/... with corresponding links in /etc/rcX.d). Avoid remounting the root filesystem. You might want to modify or disable [/srv/clientrootfs]/etc/fstab.

Because all clients share the same root filesystem, they'll have the exact same configuration. The downside is that things you want to be unique, such as host names, aren't anymore. You'll want to work around this, eg by creating separate filesystems for every client, or by using dhcp extensively (things such as reservations based on mac address, assigning hostnames by dhcp or from a combination of /etc/ethers and DNS records), you possibly need separate kernels for different hardware, ....), or using init scripts to set values on the fly in stead of reading them from a configuration file, or using all possible ways of having user preferences overrule systemwide configuration...
It might be a good idea to keep the client system minimal to avoid the need for extensive workarounds, e.g. just let it start a shell to open a session on a server.

If not all of the diskless clients have the same hardware, you may have to provide separate kernels and initrd images, and separate configuration files for software that is hardware-specific (sound, video, ...), and configure dhcp and pxelinux accordingly. pxe can be set up to find a configuration file by client ip address and can be configured to present a boot menu to let the user chose a boot configuration, so that might help.


Network boot without nfs root filesystem

When the kernel boots, it initially uses a ramdrive as its filesystem. This filesystem is later replaced by a more permanent filesystem. In the previous configuration, we used nfs to provide a root filesystem to diskless clients. It is also possible to keep using the initial ramdrive.

The difficulty in this approach lies in the fact that you will need to create a custom initrd that can be used as initial filesystem as well as root filesystem, with at least the minimal functionality to provide a usable system to the user.

Another problem is that the initrd needs to be reasonably small. Too large a file and the tftp server will refuse to serve it.

To get a feel for diskless clients without nfs root filesystems, you could have a look at Damn Small Linux (dsl). Here's a recipe to modify your pxe and tftp configuration so that the client will boot a dsl desktop from RAM.

	wget http://ftp.belnet.be/packages/damnsmalllinux/current/dsl-4.4.10-initrd.iso
	mount -o loop -t iso9660 dsl-4.4.10-initrd.iso /mnt
	cp /mnt/boot/isolinux/minirt24.gz /srv/tftp/minirt24.gz
	cp /mnt/boot/isolinux/linux24 /srv/tftplinux24

	cat >  /srv/tftp/pxelinux.cfg/default <<EOF
	  DEFAULT dsl
	  TIMEOUT 0
	  PROMPT 0

	  LABEL dsl
	  	kernel linux24
      	append initrd=minirt24.gz ip=dhcp ramdisk_size=100000 init=/etc/init BOOT_IMAGE=knoppix --

	EOF
	

The trick is that they have a (small) knoppix Live-CD image in /dev/cdrom of the intitrd filesystem, which is run by the client as if it was a local CD.

You may add additional boot parameters to the 'append' line of /srv/tftp/pxelinux.cfg/default, eg
append initrd=minirt24.gz ip=dhcp ramdisk_size=100000 init=/etc/init BOOT_IMAGE=knoppix lang=be vga=normal atapicd nosound noscsi nousb nopcmcia nofirewire noagp nomce xmodule=vesa

This approach can further be extended by mounting relevant parts of the filesystem (/home, /usr, ...) from nfs shares. The difference here is that the root filesystem is still a ram drive, not an nfs share. Modifying the client system will require mounting/unpacking the initrd, modifying it, and creating a new initrd image.

Note that there are 2 ways of providing an initial filesystem to the kernel : the initrd approach and the initramfs approach. Functionally they're the same, but the technical concept and implementation is different.

Some interesting links on network booting, ram filesystems, and manipulating initrd and initramfs :

Conclusion

Network booting and diskless clients revolves around a server that provides a custom dhcp configuration, is capable of serving a bootable kernel over tftp, and provides a network file system or a ramdisk with a root filesystem to the clients. You can build on this to create your own implementation, although for complex, production-grade implementations, you might want to consider preconfigured packages to set this up, as this will greatly facilitate configuration and maintenance of both the server as a whole as the client filesystem. Examples include:


Koen Noens
January 2008