Preboot Execution Environment - ArchWiki

Jump to content

# Preboot Execution Environment

From ArchWiki

Related articles

- Diskless system

From Wikipedia:Preboot Execution Environment:

  The Preboot eXecution Environment (PXE, most often pronounced as pixie) specification describes a standardized client-server environment that boots a software assembly, retrieved from a network, on PXE-enabled clients. On the client side it requires only a PXE-capable network interface controller (NIC), and uses a small set of industry-standard network protocols such as DHCP and TFTP.

In this guide, PXE is used to boot the installation medium with an appropriate option ROM that supports PXE on the target. This works well when you already have a server set up.

## Preparation

### Overview

It is useful to give an overview of the PXE boot process in order to understand the #Server setup, the #Installation on the client side and the Arch Linux files needed.

The client starts by broadcasting packets asking for a DHCP server and containing specific PXE options. The DHCP server responds with networking information (the IP address assigned to the client) and also provides, by using specific bootstrap protocol (BOOTP) parameters of the DHCP, additional information like the TFTP server address, the path of the initial network bootstrap program (NBP) to download or the boot configuration file name.

The NBP is transferred from the PXE server to the client using TFTP, to be loaded into memory and executed. The kernel and the initramfs are also transferred this way.

Then the root file system is transferred using one of the following protocols: HTTP, NFS or NBD.

Warning PXELINUX does not support any security mechanisms to validate if what it receives (over TFTP or HTTP) is what was expected. See RFC 5071 section8. An alternative is to use Netboot images that can verify signatures against codesigning certificates embedded in the iPXE binary. Mind that Netboot images need to be placed on the local system.

### Boot from install medium

In order to gather the files that will be transferred from the server to the client for booting, get the latest official install image from the download page.

Next mount the image:

    # mount --mkdir -o loop,ro archlinux-release_date-x86_64.iso /mnt/archiso

where `release_date` is the release date in the ISO filename like, e.g., `2022.10.01`.

Note The Arch ISO currently only supports BIOS-style PXE booting. See archlinux/archiso#55 for more information.

### Boot from netboot

Arch Linux netboot images can be used to download the latest Arch Linux release on the fly upon system boot. Download the image compatible with your configuration.

Note The Arch netboot supports both BIOS and UEFI booting.

## Pixiecore

An all-in-one solution is provided by pixiecore

1. Install pixiecore-gitAUR
2. Run `pixiecore quick arch --dhcp-no-bind` as root
3. Boot via PXE

## Server setup

You will need to setup a DHCP server, a TFTP server for transferring the NBP and one of the following services for transferring the root file system: HTTP server, NFS or NBD.

### Network

Bring up your wired network adapter, and assign it an address appropriately.

    # ip link set eth0 up
    # ip addr add 192.168.0.1/24 dev eth0

You can also use one of the network managers to configure the static IP.

### DHCP + TFTP

You will need both a DHCP server and a TFTP server to configure networking on the install target and to facilitate the transfer of files between the server and the client. dnsmasq does both, and is extremely easy to set up.

Install the dnsmasq package.

dnsmasq needs to be configured. See instructions on how to setup a dnsmasq#TFTP server and a dnsmasq#PXE server.

Are provided below some common configuration instructions. tftp_root is the directory where the Arch ISO is mounted (e.g. `/mnt/archiso`) or where the network boot program is located.

    # /etc/dnsmasq.conf

    # Listen only to the specified interface
    interface=eth0

    # Do not function as DNS server
    port=0

    # TFTP server setup
    enable-tftp
    tftp-root=tftp_root

    # Log extra information about DHCP transactions (for debug purposes)
    log-dhcp

To enable the DHCP server and give IPv4 addresses within a range, add to the configuration file a line similar to:

    dhcp-range=192.168.0.50,192.168.0.150

Or in case there is already a DHCP-server running on the network and you want to interoperate with it, see dnsmasq#Proxy DHCP.

Two examples covering different boot style and installation media are provided below.

Once configured according to your needs, start `dnsmasq.service`.

#### BIOS boot from install medium

The path of the initial bootstrap program to be transferred is defined with the `dhcp-boot` option in the configuration file.

    dhcp-boot=/boot/syslinux/lpxelinux.0

In order to send specific bootstrap protocol (BOOTP) parameters, like the configuration file path, the `dhcp-option-force=flag,value` line is used.

    dhcp-option-force=209,archiso_pxe.cfg # this file might be under /mnt/archiso/boot/syslinux
    dhcp-option-force=210,

#### UEFI boot from netboot

To send a file depending on the architecture, here the netboot image for UEFI-style boot, use:

    pxe-service=BC_EFI, "Boot from network BC EFI", ipxe.efi
    pxe-service=X86-64_EFI, "Boot from network X86-64 EFI", ipxe.efi

If using netboot, the rest of the server setup section which focuses on the Arch ISO does not apply.

### Transferring archiso root file system

Thanks to `archiso_pxe_http`, `archiso_pxe_nfs` and `archiso_pxe_nbd` initcpio hooks in archiso, it is possible to boot using HTTP, NFS or NBD. Boot time is approximately the same in all three methods, but HTTP method allows you to watch a state of downloading `airootfs.sfs` in percents.

#### HTTP

Among all alternatives, darkhttpd is by far the most trivial to setup (and the lightest-weight).

First, install the darkhttpd package.

Then start darkhttpd using our `/mnt/archiso` as the document root:

    # darkhttpd /mnt/archiso

    darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic.
    listening on: http://0.0.0.0:80/

Note It is important that the server is running on port `80`. If you start darkhttpd without root access it will default to `8080`, the client will still try to access port 80 and the boot will fail.

#### NFS

You will need to set up an NFS server with an export at the root of your mounted installation medium, which would be `/mnt/archiso` if you followed #Preparation. After setting up the server, add the following line to your `/etc/exports` file:

    /etc/exports

    /mnt/archiso 192.168.0.0/24(ro,no_subtree_check)

If the server was already running, re-export the file systems with `exportfs -r -a -v`.

The default settings in the installer expect to find the NFS at `/run/archiso/bootmnt/`, so you will need to edit the boot options. To do this, press Tab on the appropriate boot menu choice and edit the `archiso_nfs_srv` option accordingly:

    archiso_nfs_srv=${pxeserver}:/mnt/archiso

Alternatively, you can use `/run/archiso/bootmnt` for the entire process.

After the kernel loads, the Arch bootstrap image will copy the root file system via NFS to the booting host. This can take a little while. Once this completes, you should have a running system.

#### NBD

Install the nbd package and configure it:

    /etc/nbd-server/config

    [generic]
    [archiso]
        readonly = true
        exportname = /srv/archlinux-release_date-x86_64.iso

where `release_date` is the release date in the ISO filename like, e.g., `2022.10.01`.

Start `nbd.service`.

### Existing PXE server

If you have an existing PXE server with a PXELINUX system setup (e.g. a combination of DHCP and TFTP), you can add the following menu items to your `/tftpboot/pxelinux.cfg/default` file in order to boot Arch via your preferred method.

Since PXELINUX supports HTTP, only the boot loader needs to be transferred over TFTP, everything else can use HTTP. E.g.:

    LABEL archlinux
            MENU LABEL Arch Linux x86_64
            LINUX http://httpserver/path/to/extracted/Arch/ISO/arch/boot/x86_64/vmlinuz-linux
            INITRD http://httpserver/path/to/extracted/Arch/ISO/arch/boot/x86_64/initramfs-linux.img
            APPEND archisobasedir=arch archiso_http_srv=http://httpserver/path/to/extracted/Arch/ISO/ cms_verify=y
            SYSAPPEND 3
            TEXT HELP
            Arch Linux 2022.10.01 x86_64
            ENDTEXT

For NFS and NBD, the kernel and initramfs must be downloaded from TFTP. E.g., for NFS:

    LABEL archlinux
            MENU LABEL Arch Linux x86_64
            LINUX /path/to/extracted/Arch/ISO/arch/boot/x86_64/vmlinuz-linux
            INITRD /path/to/extracted/Arch/ISO/arch/boot/x86_64/initramfs-linux.img
            APPEND archisobasedir=arch archiso_nfs_srv=pxeserver:/run/archiso/bootmnt cms_verify=y
            SYSAPPEND 3
            TEXT HELP
            Arch Linux 2022.10.01 x86_64
            ENDTEXT

The `LINUX` and `INITRD` paths are relative to TFTP root. For NBD, replace `archiso_nfs_srv` with `archiso_nbd_srv` in the above example. See usage examples in `boot/syslinux/archiso_pxe.cfg` file resided on Arch Linux ISO.

Whichever method you choose, you must pass `ip=` parameter to instruct the kernel to bring up the network interface before it attempts to mount the installation medium over the network. Passing `BOOTIF=` is required when there are several wired interfaces on the client side and/or you want `resolv.conf` to be already configured inside booted archiso. You can use sysappend mask `3` (which is `1`+`2`) to pass these parameters automatically. For available boot parameters see README.bootparams.

## Installation

For this portion you will need to figure out how to tell the client to attempt a PXE boot; in the corner of the screen along with the normal post messages, usually there will be some hint on which key to press to try PXE booting first. On an IBM x3650 `F12` brings up a boot menu, the first option of which is Network; on a Dell PE 1950/2950 pressing `F12` initiates PXE booting directly.

### Boot

Looking at journald on the PXE server will provide some additional insight to what exactly is going on during the early stages of the PXE boot process:

    # journalctl -u dnsmasq.service -f

    dnsmasq-dhcp[2544]: DHCPDISCOVER(eth1) 00:1a:64:6a:a2:4d
    dnsmasq-dhcp[2544]: DHCPOFFER(eth1) 192.168.0.110 00:1a:64:6a:a2:4d
    dnsmasq-dhcp[2544]: DHCPREQUEST(eth1) 192.168.0.110 00:1a:64:6a:a2:4d
    dnsmasq-dhcp[2544]: DHCPACK(eth1) 192.168.0.110 00:1a:64:6a:a2:4d
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/pxelinux.0 to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/whichsys.c32 to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso_pxe_choose.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/ifcpu64.c32 to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso_pxe_both_inc.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso_head.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso_pxe32.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso_pxe64.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/archiso_tail.cfg to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/vesamenu.c32 to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/boot/syslinux/splash.png to 192.168.0.110

After you load `pxelinux.0` and `archiso.cfg` via TFTP, you will (hopefully) be presented with a syslinux boot menu with several options, where you can select Boot Arch Linux (x86_64) (HTTP).

Next the kernel and initramfs (appropriate for the architecture you selected) will be transferred, again via TFTP:

    dnsmasq-tftp[2544]: sent /mnt/archiso/arch/boot/x86_64/vmlinuz to 192.168.0.110
    dnsmasq-tftp[2544]: sent /mnt/archiso/arch/boot/x86_64/initramfs-linux.img to 192.168.0.110

If all goes well, you should then see activity on darkhttpd coming from the PXE-target; at this point the kernel would be loaded on the PXE-target, and in init:

    1348347586 192.168.0.110 "GET /arch/aitab" 200 678 "" "curl/7.27.0"
    1348347587 192.168.0.110 "GET /arch/x86_64/root-image.fs.sfs" 200 107860206 "" "curl/7.27.0"
    1348347588 192.168.0.110 "GET /arch/x86_64/usr-lib-modules.fs.sfs" 200 36819181 "" "curl/7.27.0"
    1348347588 192.168.0.110 "GET /arch/any/usr-share.fs.sfs" 200 63693037 "" "curl/7.27.0"

After the root file system is downloaded via HTTP, you will eventually end up at the normal live system root zsh prompt.

### Post-boot

Unless you want all traffic to be routed through your PXE server (which will not work anyway unless you set it up properly), you will want to stop `dnsmasq.service` and get a new lease on the install target, as appropriate for your network layout.

You can also kill darkhttpd; the target has already downloaded the root file system, so it is no longer needed. While you are at it, you can also unmount the installation image:

    # umount /mnt/archiso

At this point you can follow the Installation guide.

### Low memory systems

The `copytoram` initramfs option can be used to control whether the root file system should be copied to ram in its entirety in early-boot.

It highly recommended to leave this option alone, and should only be disabled if entirely necessary (systems with less than ~256MB physical memory). Append `copytoram=n` to your kernel line if you wish to do so.

Note As this requires loop-mounting squashfs from a mounted remote file system, `copytoram=n` and `archiso_pxe_http` are mutually exclusive.

### Sharing internet with PXE clients

If your network for PXE clients is private (for example, 192.168.1.0/24), and you want them to be able to access internet (for example, for packages installation), you should configure masquerade/source nat properly. Your PXE server must have a separate NIC connected to the internet. You can use such command to pass through the internet to clients:

    iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE

To make this rule persistent after reboot, run the following command:

    iptables-save -f /etc/iptables/iptables.rules

and enable `iptables.service`.

See Simple stateful firewall#Setting up a NAT gateway and Internet sharing#Enable NAT for more information.

## Troubleshooting

### DHCP interface rename bug

FS#36749 causes default predictable network interface renaming to fail and then DHCP client to fail because of it. A workaround is to add the kernel boot parameter `net.ifnames=0` to disable predictable interface names.

### VirtualBox cannot boot while real machines can

When using VirtualBox to test your configuration, the virtual machine may get stuck at:

    Probing EDD (edd=off to disable)... ok

While PXE booting with a real machine works fine. The problem may be because you have set several CPU cores to your client machine, and you set its type as Other and version as Other/Unknown (64 bit). So VirtualBox does not know which paravirtualization interface to use by default.

Adding `loglevel=7` to the kernel parameters lets you see where it actually got stuck:

    [    0.063697] smp: Bringing up secondary CPUs...
    [    0.103768] x86: Booting SMP configuration:

To resolve this, either use one CPU core, or go to Machine > Settings > System > Acceleration and set one of the following paravirtualization interface: Minimal, Hyper-V or KVM.

Categories:

- Installation process
- Networking
- Boot process

Preboot Execution Environment
Add topic