OpenWRT With a Slice of Pi

Note: These instructions are superseded with my CLI that I wrote to ease the configuration. I am retaining these instructions so that the curious among you can see the details behind what the Lab CLI abstracts.

Setting up a Raspberry Pi as your lab bastion host

For your bastion host, you will need a Raspberry Pi 4b with 8GB of RAM. I am using a Vilros kit from Amazon. You will also need an SD Card of at least 128GB capacity. Get one with fast read/write speeds as it will be used to serve up RPMs for host installations and images from your Nexus registry.

We are going to use the edge router that we set up in the previous step to configure the OS for the Raspberry Pi.

1. Insert the SD Card into the edge router.

2. SSH into the router

   ssh root@router.${LAB_DOMAIN}
   

3. Install some additional packages

   opkg update && opkg install wget sfdisk rsync resize2fs
   

4. Retrieve the OpenWRT image for the Pi 4b

   wget https://downloads.openwrt.org/releases/21.02.1/targets/bcm27xx/bcm2711/openwrt-21.02.1-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz
   gunzip openwrt-21.02.1-bcm27xx-bcm2711-rpi-4-ext4-factory.img.gz
   

5. Unmount the SD Card and flash it with the OS image

   umount /dev/mmcblk1p1
   dd if=openwrt-21.02.1-bcm27xx-bcm2711-rpi-4-ext4-factory.img of=/dev/mmcblk1 bs=4M conv=fsync
   rm openwrt-21.02.1-bcm27xx-bcm2711-rpi-4-ext4-factory.img
   

6. Resize the Root volume to 20GB and create a /usr/local filesystem using the rest of the drive.

   PART_INFO=$(sfdisk -l /dev/mmcblk1 | grep mmcblk1p2)
   let ROOT_SIZE=41943040
   let P2_START=$(echo ${PART_INFO} | cut -d" " -f2)
   let P3_START=$(( ${P2_START}+${ROOT_SIZE}+8192 ))
   sfdisk --delete /dev/mmcblk1 2
   sfdisk -d /dev/mmcblk1 > /tmp/part.info
   echo "/dev/mmcblk1p2 : start= ${P2_START}, size= ${ROOT_SIZE}, type=83" >> /tmp/part.info
   echo "/dev/mmcblk1p3 : start= ${P3_START}, type=83" >> /tmp/part.info
   umount /dev/mmcblk1p1
   sfdisk /dev/mmcblk1 < /tmp/part.info
   
   e2fsck -f /dev/mmcblk1p2
   resize2fs /dev/mmcblk1p2
   mkfs.ext4 /dev/mmcblk1p3
   

7. Mount the new root filesystem to a temporary mount point

   mkdir /tmp/pi
   mount -t ext4 /dev/mmcblk1p2 /tmp/pi/
   

8. Configure the bastion network settings

   read -r -d '' FILE << EOF
   config interface 'loopback'\n
   \toption device 'lo'\n
   \toption proto 'static'\n
   \toption ipaddr '127.0.0.1'\n
   \toption netmask '255.0.0.0'\n
   \n
   config device\n
   \toption name 'br-lan'\n
   \toption type 'bridge'\n
   \tlist ports 'eth0'\n
   \n
   config interface 'lan'\n
   \toption device 'br-lan'\n
   \toption proto 'static'\n
   \toption ipaddr '${BASTION_HOST}'\n
   \toption netmask '${NETMASK}'\n
   \toption gateway '${ROUTER}'\n
   \toption dns '${ROUTER}'\n
   EOF
   
   echo -e ${FILE} > /tmp/pi/etc/config/network
   

9. Disable password login so that only SSH key access is allowed

   read -r -d '' FILE << EOF
   config dropbear\n
   \toption PasswordAuth 'off'\n
   \toption RootPasswordAuth 'off'\n
   \toption Port '22'\n
   EOF
   
   echo -e ${FILE} > /tmp/pi/etc/config/dropbear
   

10. Set the hostname

    read -r -d '' FILE << EOF
    config system\n
    \toption timezone 'UTC'\n
    \toption ttylogin '0'\n
    \toption log_size '64'\n
    \toption urandom_seed '0'\n
    \toption hostname 'bastion.${DOMAIN}'\n
    \n
    config timeserver 'ntp'\n
    \toption enabled '1'\n
    \toption enable_server '0'\n
    \tlist server '0.openwrt.pool.ntp.org'\n
    \tlist server '1.openwrt.pool.ntp.org'\n
    \tlist server '2.openwrt.pool.ntp.org'\n
    \tlist server '3.openwrt.pool.ntp.org'\n
    EOF
    
    echo -e ${FILE} > /tmp/pi/etc/config/system
    

11. Add SSH keys to the bastion host

    cat /etc/dropbear/authorized_keys >> /tmp/pi/etc/dropbear/authorized_keys
    dropbearkey -y -f /root/.ssh/id_dropbear | grep "^ssh-" >> /tmp/pi/etc/dropbear/authorized_keys
    

12. Disable dnsmasq

    rm -f /tmp/pi/etc/rc.d/*dnsmasq*
    

13. Unmount the SD Card

    umount /dev/mmcblk1p1
    umount /dev/mmcblk1p2
    umount /dev/mmcblk1p3
    

Remove card from router, put it in the Pi, and boot it up.

Finish configuring the bastion host:

1. From your workstation, SSH into the bastion host

   createEnvScript.sh -e 
   cat ${OKD_LAB_PATH}/work-dir/edge-router | ssh root@bastion.${LAB_DOMAIN} "cat >> /root/.profile"
   rm -rf ${OKD_LAB_PATH}/work-dir
   

2. Log onto the Pi:

   ssh root@bastion.${LAB_DOMAIN}
   

3. Set a complex root password. Keep it safe, but remember that we are using SSH keys to log in. So you shouldn’t need this password.

   passwd
   

4. Install the necessary packages

   opkg update && opkg install ip-full uhttpd shadow bash wget git-http ca-bundle procps-ng-ps rsync curl libstdcpp6 libjpeg libnss lftp block-mount
   
   opkg list | grep "^coreutils-" | while read i
   do
       opkg install $(echo ${i} | cut -d" " -f1)
   done
   

5. Create an SSH Key Pair

   mkdir -p /root/.ssh
   dropbearkey -t rsa -s 4096 -f /root/.ssh/id_dropbear
   

6. Mount the /usr/local filesystem:

   let RC=0
   while [[ ${RC} -eq 0 ]]
   do
     uci delete fstab.@mount[-1]
     let RC=$?
   done
   PART_UUID=$(block info /dev/mmcblk0p3 | cut -d\" -f2)
   MOUNT=$(uci add fstab mount)
   uci batch << EOI
   set fstab.${MOUNT}.target=/usr/local
   set fstab.${MOUNT}.uuid=${PART_UUID}
   set fstab.${MOUNT}.enabled=1
   EOI
   uci commit fstab
   block mount
   

7. Create folders for the installation files:

   mkdir -p /usr/local/www/install/kickstart
   mkdir /usr/local/www/install/postinstall
   mkdir /usr/local/www/install/fcos
   

8. Create a script for synching a CentOS Stream repository mirror:

   mkdir -p /root/bin
   
   cat << EOF > /root/bin/MirrorSync.sh
   #!/bin/bash
   
   for i in BaseOS AppStream PowerTools extras
   do 
     rsync  -avSHP --delete \${REPO_MIRROR}8-stream/\${i}/x86_64/os/ /usr/local/www/install/repos/\${i}/x86_64/os/ > /tmp/repo-mirror.\${i}.out 2>&1
   done
   EOF
   
   chmod 750 /root/bin/MirrorSync.sh
   

9. Create the repo mirror tree:

   for i in BaseOS AppStream PowerTools extras
   do 
     mkdir -p /usr/local/www/install/repos/${i}/x86_64/os/
   done
   

10. Go to https://centos.org/download/mirrors/ and select a mirror new you that supports rsync:

    Add the mirror’s rsync link as an environment variable to /root/.profile

    export REPO_MIRROR=rsync://your.centos.mirror.com/centos/
    echo "export REPO_MIRROR=${REPO_MIRROR}" >> /root/.profile
    

11. Start the mirror. This will take a while:

    nohup /root/bin/MirrorSync.sh ${REPO_MIRROR} &
    

12. Create a repo file for the mirror:

    cat << EOF > /usr/local/www/install/postinstall/local-repos.repo
    [local-appstream]
    name=AppStream
    baseurl=http://${BASTION_HOST}/install/repos/AppStream/x86_64/os/
    gpgcheck=0
    enabled=1
    
    [local-extras]
    name=extras
    baseurl=http://${BASTION_HOST}/install/repos/extras/x86_64/os/
    gpgcheck=0
    enabled=1
    
    [local-baseos]
    name=BaseOS
    baseurl=http://${BASTION_HOST}/install/repos/BaseOS/x86_64/os/
    gpgcheck=0
    enabled=1
    
    [local-powertools]
    name=PowerTools
    baseurl=http://${BASTION_HOST}/install/repos/PowerTools/x86_64/os/
    gpgcheck=0
    enabled=1
    EOF
    

13. Setup uhttpd for hosting a CentOS Stream repo mirror and host installation files

    uci del_list uhttpd.main.listen_http="[::]:80"
    uci del_list uhttpd.main.listen_http="0.0.0.0:80"
    uci del_list uhttpd.main.listen_https="[::]:443"
    uci del_list uhttpd.main.listen_https="0.0.0.0:443"
    uci del uhttpd.defaults
    uci del uhttpd.main.cert
    uci del uhttpd.main.key
    uci del uhttpd.main.cgi_prefix
    uci del uhttpd.main.lua_prefix
    uci add_list uhttpd.main.listen_http="${BASTION_HOST}:80"
    uci add_list uhttpd.main.listen_http="127.0.0.1:80"
    uci set uhttpd.main.home='/usr/local/www'
    uci commit uhttpd
    /etc/init.d/uhttpd restart
    

14. Enable NTP server:

    uci set system.ntp.enable_server="1"
    uci commit system
    /etc/init.d/sysntpd restart
    

15. Create a Chrony configuration file for KVM Hosts:

    cat << EOF > /usr/local/www/install/postinstall/chrony.conf
    server ${BASTION_HOST} iburst
    driftfile /var/lib/chrony/drift
    makestep 1.0 3
    rtcsync
    logdir /var/log/chrony
    EOF
    

16. Create a script to trigger a host reinstall:

    cat << EOF > /usr/local/www/install/postinstall/rebuildhost.sh
    #!/bin/bash
    
    REBOOT=true
    
    for i in "$@"
    do
      case ${i} in
        -s|--shutdown)
        REBOOT=false
        shift
      ;;
        *)
          echo "USAGE:"
          echo "\nTo initiate an immediate rebuild:"
          echo "rebuildhost.sh"
          echo "\nTo shutdown and rebuild on the next boot:"
          echo "rebuildhost.sh -s"
      ;;
      esac
    done
    
    P1=\$(lsblk -l | grep /boot/efi | cut -d" " -f1)
    P2=\$(lsblk -l | grep /boot | grep -v efi | cut -d" " -f1)
    MAJ=\$(lsblk -l | grep \${P1} | tr -s " " | cut -d" " -f2 | cut -d: -f1)
    BOOT_DISK=\$(lsblk -l | grep "\${MAJ}:0" | cut -d" " -f1)
    
    umount /boot/efi
    umount /boot
    wipefs -a /dev/\${P1}
    wipefs -a /dev/\${P2}
    dd if=/dev/zero of=/dev/\${BOOT_DISK} bs=512 count=1
       
    if [[ ${REBOOT} == "true" ]]
    then
      shutdown -r now
    else
      shutdown -h now
    fi
    EOF
    

17. Copy the bastion host SSH public key to a file for host installation:

    dropbearkey -y -f /root/.ssh/id_dropbear | grep "ssh-" > /usr/local/www/install/postinstall/authorized_keys
    

18. Log off of the bastion host:

    exit