Automated Installs, the old fashioned way

 I've used TheForeman and Canonical MaaS (and have heard of Cobbler), and while they can and do work, they can be a bit fiddly to understand, and if something goes wrong everything is so abstracted it can be hard to understand how things should be working and what you need to do to fix things.

So, I decided to take a step back and look at how you would configure automated installs the old fashioned way, by hand. This gives a better understanding on what's going off under the hood and helps debug things if you're having a problem with your pre-packaged bare metal deploy automation.

DHCP

First things, I'm using ISC DHCPD (I guess about time I swapped to something newer), and have a host entry like:

        host rockytest {
          hardware ethernet AB:CD:EF:D4:A2:BB;
          fixed-address 192.168.2.64;
          next-server 192.168.2.1;
          filename "rhel8/redhat/EFI/BOOT/BOOTX64.EFI";
        }

So this identifies my new system by MACADDR (hardware ethernet), gives it an IP address, and then next-server defines the TFTP server and the filename that should be retrieved for booting. I'm using EFI on this system, not legacy BIOS, so I identify the EFI boot file.

Powering on this box results in it doing a TFTP to 192.168.2.1 downloading and running that particular file.

TFTP

The tftp service by default doesn't log actual file transfers which is too bad because seeing these in the logs actually gives you more info on what things are trying, which also show where you have options for configuring things.

So, create a custom local override /etc/systemd/system/tftp.service by doing systemctl edit --full tftp, and add in "-v" to the tftp invocation so that things look like:

[Unit]

Description=Tftp Server

Requires=tftp.socket

Documentation=man:in.tftpd

[Service]

ExecStart=/usr/sbin/in.tftpd -v -s /var/lib/tftpboot

StandardInput=socket

[Install]

Also=tftp.socket

Do the usual systemctl daemon-reload and systemctl restart tftp to get things going.

So now after powering on the VM rockytest, we will see in journalctl -f the log entry for retrieving the EFI file:

RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/BOOTX64.EFI

Note, evidently journalctl only logs the successful transfers and not any failed transfer, so you are probably better off looking in /var/log/messages to see what else tftp is logging, particularly with the files that aren't found. Those are interesting since it shows some inner workings, more on that later...

Configs, configs, and more configs.

So now we have to put all the right files into the /var/lib/tftpboot directory. In our DHCP config we specified the EFI file was in the "rhel8" subdirectory, so the layout looks like:

rhel8/redhat/EFI/BOOT:

BOOTX64.EFI  grub.cfg  grubx64.efi  README.md

rhel8/images/pxeboot:

initrd.img  README.md  vmlinuz

For the EFI/BOOT directory, these files were retrieved as per my README.md:

wget https://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/EFI/BOOT/BOOTX64.EFI

wget https://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/EFI/BOOT/grubx64.efi

wget https://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/EFI/BOOT/grub.cfg

So based on the /var/log/messages tftp entries, we can see things being retrieved:

in.tftpd[3400683]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/BOOTX64.EFI

in.tftpd[3400683]: Client ::ffff:192.168.2.64 finished rhel8/redhat/EFI/BOOT/BOOTX64.EFI

in.tftpd[3400685]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/revocations.efi

in.tftpd[3400685]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/revocations.efi

in.tftpd[3400686]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grubx64.efi

in.tftpd[3400686]: Client ::ffff:192.168.2.64 finished rhel8/redhat/EFI/BOOT/grubx64.efi

So it retrieves the RHEL 8 BOOTX64.EFI, tries for some revocations.efi file, IDK, then downloads grubx64.efi.

Then things get a bit interesting as it looks for the grub.cfg file:

in.tftpd[3400687]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-01-ab-cd-ef-d4-a2-bb

in.tftpd[3400687]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-01-ab-cd-ef-d4-a2-bb

in.tftpd[3400688]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0A80240

in.tftpd[3400688]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0A80240

in.tftpd[3400689]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0A8024

in.tftpd[3400689]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0A8024

in.tftpd[3400690]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0A802

in.tftpd[3400690]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0A802

in.tftpd[3400691]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0A80

in.tftpd[3400691]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0A80

in.tftpd[3400692]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0A8

in.tftpd[3400692]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0A8

in.tftpd[3400693]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0A

in.tftpd[3400693]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0A

in.tftpd[3400694]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C0

in.tftpd[3400694]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C0

in.tftpd[3400695]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg-C

in.tftpd[3400695]: Client ::ffff:192.168.2.64 File not found rhel8/redhat/EFI/BOOT/grub.cfg-C

in.tftpd[3400696]: RRQ from ::ffff:192.168.2.64 filename rhel8/redhat/EFI/BOOT/grub.cfg

in.tftpd[3400696]: Client ::ffff:192.168.2.64 finished rhel8/redhat/EFI/BOOT/grub.cfg

So what's interesting here is it first tries a grub.cfg with 01-MACADDR, so we can have a custom grub.cfg for our specific system. If that's not found, it tries grub.cfg-IPADDRINHEX to get a specific config for the IP (listed in HEX), and then walks back on the filename to allow you to set up grub.cfg files for various network ranges.

In my case, experimenting, I just have a single grub.cfg, but ultimately, you're likely to want to have a grub.cfg per system.

The grub.cfg for the EFI looks like:

set default="1"

function load_video {

  insmod efi_gop

  insmod efi_uga

  insmod video_bochs

  insmod video_cirrus

  insmod all_video

}

load_video

set gfxpayload=keep

insmod gzio

insmod part_gpt

insmod ext2

set timeout=60

### END /etc/grub.d/00_header ###

search --no-floppy --set=root -l 'Rocky-8-10-x86_64-dvd'

### BEGIN /etc/grub.d/10_linux ###

menuentry 'Install Rocky Linux 8.10' --class fedora --class gnu-linux --class gnu --class os {

        linuxefi rhel8/images/pxeboot/vmlinuz inst.repo=http://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/

        initrdefi rhel8/images/pxeboot/initrd.img

}

I've trimmed this file down and changed from the default grub.cfg that was downloaded as per my README.md, but the interesting parts are the linuxefi and initrdefi entries:

        linuxefi rhel8/images/pxeboot/vmlinuz inst.repo=http://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/

        initrdefi rhel8/images/pxeboot/initrd.img

Here we define where the vmlinuz and initrd.img files are coming from, so basically our TFTP directory, this time in rhel8/images/pxeboot/, where we downloaded the images as per note in my README.md:

wget https://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/images/pxeboot/initrd.img

wget https://mirrors.rit.edu/rocky/8/BaseOS/x86_64/kickstart/images/pxeboot/vmlinuz

So after we choose menu entry 1 from the install screen, this results in TFTP activity:

in.tftpd[3400704]: RRQ from ::ffff:192.168.2.64 filename rhel8/images/pxeboot/vmlinuz

in.tftpd[3400704]: Client ::ffff:192.168.2.64 finished rhel8/images/pxeboot/vmlinuz

in.tftpd[3400706]: RRQ from ::ffff:192.168.2.64 filename rhel8/images/pxeboot/initrd.img

in.tftpd[3400706]: Client ::ffff:192.168.2.64 finished rhel8/images/pxeboot/initrd.img

So now the PXE/EFI installer has the vmlinuz kernel image and the initialrd disk image loaded, it passes control to the installer. Note that I updated the linuxefi line to specify inst.repo, which means that all the packages and other installer stuff is going to come from http://mirrors.rit.edu.

If we were fully automating this install, the grub.cfg would probably be custom for our IP address, and besides specifying inst.repo, it would also specify inst.ks=http://[whatever] so that we get a custom kickstart config for our particular system install that would specify defaults needed, etc.

Note also that ROCKY8 and ROCKY9 systems can use cloud-init for configuring rather than kickstart configs.

iPXE

So that's all cool for a FEDORA based system, and is all logical. What if you just wanted to boot some random ISO, like the ProxMox installer? This is where iPXE (https://ipxe.org) comes in to play. It provides a more enhanced PXE firmware that you can boot into, and optionally boot an ISO image using sanboot http://[whatever]/stuff.iso

Verizon FIOS and IPv6

tl;dr - if you want to run IPv6 on an internal network with an internal router behind a FIOS G1100 router, you must carve up a different /64 network from the /64 the FIOS gives you on the LAN interface, and then also update the FIOS G1100 route table to have a static route to this different /64 network, passing it to the "WAN" interface of your internal router which is really just connected to the LAN interface of the FIOS G1100 router.

IPv6, it's been around for a while. Verizon supports it and if I look at my Verizon Fios-G1100 router, I can also turn on IPv6:

and Verizon gives me a /56 network. Fun Fact! That /56 network gives me 4,722,366,482,869,645,213,696 possible IP addresses. The number is four sextillion, seven hundred and twenty-two quintillion, three hundred and sixty-six quadrillion, four hundred and eighty-two trillion, eight hundred and sixty-nine billion, six hundred and forty-five million, two hundred and thirteen thousand and six hundred and ninety-six. That's a lot of IP addresses.

What's interesting is the Fios-G1100 is configured to act as a DHCP server and so gives out a /64 address on the LAN:

This /64 turns gives me 18,446,744,073,709,551,616 IP addresses which is only eighteen quintillion, four hundred forty-six quadrillion, seven hundred forty-four trillion, seventy-three billion, seven hundred nine million, five hundred fifty-one thousand, six hundred and sixteen.

So I have a singular Linux router behind my Fios-G1100 so I can have more options with experimenting with stuff, and on the Linux router on my public interface, I see:

3: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1c:1b:0d:03:bb:ec brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.2/24 brd 192.168.1.255 scope global noprefixroute enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 2600:1234:5678:abcd:1e1b:dff:fe03:bbec/64 scope global
       valid_lft 1258sec preferred_lft 1258sec
    inet6 fe80::1e1b:dff:fe03:bbec/64 scope link
       valid_lft forever preferred_lft forever

So it gives me out a singular /64, which is fine here. A bit strange that to Verizon I have a singular machine on my network. With the IPv4, I'm running NAT so get 192.168.1.2 here, with the Verizon router being 192.168.1.1. With IPv6, I get a public IP on the /64 network carved out from the /56 network. So that's one /64 network out of 255 possible in the /56. What was the challenge for me was that internally I wanted to have the /64 network available to all my boxes being my Linux router. That didn't work as I wanted to have the same 2600:1234:5678:abcd::/64 network on my LAN side of the Linux router that was the same as the /64 network on the WAN side of my Linux router, using the same /64 network assigned on the LAN side of the FIOS router. That confused routing of various things on the Linux side after setting up IP forwarding.

Here's how I was trying to define the LAN interface, which is wrong:

2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 1c:1b:0d:03:bb:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global noprefixroute enp8s0
       valid_lft forever preferred_lft forever
    inet6 2600:1234:5678:abcd:1e1b:dff:fe03:bbee/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::1e1b:dff:fe03:bbee/64 scope link
       valid_lft forever preferred_lft forever

After a fruitless week of googling, AI assistance, and generally learning about IPv6 and routing, getting all sorts of advice ranging from this should be just working, to I can turn off the majority of functionality in my FIOS router which should then enable a hidden "bridge mode" on the router which would in effect give the entire /56 to my Linux router WAN interface which would then let it carve that /56 into a /64 on the LAN interface. This may have worked, but then I would lose functionality of the FIOS router, including any out of the box firewall protection that I'm not yet confident enough I would be able to do in my Linux router.

The solution, as I stumbled across it, is to actually carve up a different /64 network for my internal Linux router LAN address and then configure the FIOS router to have a static route to that network:

2: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 1c:1b:0d:03:bb:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.1/24 brd 192.168.2.255 scope global noprefixroute enp8s0
       valid_lft forever preferred_lft forever
    inet6 2600:1234:5678:abce:1e1b:dff:fe03:bbee/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::1e1b:dff:fe03:bbee/64 scope link
       valid_lft forever preferred_lft forever

The subtle part in there being my Linux router WAN interface (defined automatically by the FIOS router LAN side) is 2600:1234:5678:abcd:1e1b:dff:fe03:bbec/64, but my carve out for the internal LAN is 2600:1234:5678:abce:1e1b:dff:fe03:bbee/64. The notable difference being the "abcd" on the WAN side and the "abce" on the LAN side, so one address range higher in my /56 range given out by Verizon.

Then on the FIOS router, under the routing table:

I add a route entry with a Destination network of 2600:1234:5678:abce::/64 being sent to the Gateway of 2600:1234:5678:abcd::/64, which is the Verizon FIOS assigned LAN address that I am using for the WAN address on my Linux router. This makes sure that any packets that I have sent out that get returned to the FIOS router, it knows to just pass them off to my Linux router, which can then send them to my devices on my local IPv6 network.

None of this was obvious to me, and I suspect not many people are running their own custom Linux router behind the FIOS router and so don't need to do this additional route. I suspect people who are smart enough to run their own Linux router just inherently understand this route on the FIOS router thing, and so don't need guides on how to set it up. I am in this between world where I do need the guide, but also have the custom stuff internally that I sort of (?) understand. So here we are, me writing down this note since it took a week of figuring out and I wanted to immortalize the solution.