Cobbler provisioning via L3 and DHCP Relay, a PoC

Posted on In

Enable cobbler provisioning via L3 DHCP relay

I would like to setup a minimal PoC for Cobbler provisioning via DHCP relay, I will use virtualbox to spawn:

Three VMs:

  • Cobbler server in subnet A, let’s call it cobbler_server.
  • A machine to be provisioned in subnet B, it’s named as machine_0.
  • vRouter wired above machines in two subnets, it’s named as vrouter.

To create two subnets:

  • subnet_A, vboxnet0: 192.168.56.0/24
  • subnet_B, vboxnet1: 192.168.57.0/24

And it should be like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
+-------------+        +-------------+        +-------------+
|             |        |     vRouter |        |   machine_0 |
|             |        |192.168.56.10|        |             |
|           +-+        +-+           |        |             |
|           | |00000000| rr          |        |             |
|           +-+        +-+r          |        |             |
|192.168.56.11|        |  r          |        |             |
|             |        |  r          |        |             |
|             |        |  r          |        |             |
|             |        |  r        +-+        +-+           |
| Cobbler     |        |  rrrrrrrrr| |11111111| |           |
| Server      |        |           +-+        +-+           |
|             |        |192.168.57.10|        |             |
+-------------+        +-------------+        +-------------+


0000000 subnet A: vboxnet0     +-+
                               | | network interface
1111111 subnet B: vboxnet1     +-+

rrrrrrr DHCP Relay

Then we could use cobbler_server to provision a OS to machine_0.

Setup minimal env

Download a ubuntu bionic image for virtualbox

1
wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.ova

Create three VMs

1
2
3
4
❯ VBoxManage list vms
"cobbler_server" {aa4c06ba-139d-4e1e-a248-33d5cb4763bd}
"vrouter" {092add9c-0b03-4230-adf9-465f18bafde1}
"machine_0" {97041fe8-a547-4f84-a9fe-1190664100b7}

Create vNICs for all VMs

Note to enable tftp working in virtualbox in default configuration, use nic type:

PCnet-PCI II(Am79C970A)

reference: https://www.webscalability.com/blog/2013/03/testing-out-cobbler-with-virtuabox/

1
2
3
4
5
6
7
8
9
10
11
12
13
❯ VBoxManage showvminfo cobbler_server | grep NIC | grep -v disabled
NIC 1:  MAC: 0800270CCFBD, Attachment: Bridged Interface 'en1: Wi-Fi (AirPort)'
NIC 2:  MAC: 0800275425ED, Attachment: Host-only Interface 'vboxnet0'


❯ VBoxManage showvminfo vrouter | grep NIC | grep -v disabled
NIC 1:  MAC: 0800274A2FAB, Attachment: Bridged Interface 'en1: Wi-Fi (AirPort)'
NIC 2:  MAC: 080027381AB1, Attachment: Host-only Interface 'vboxnet0'
NIC 3:  MAC: 08002736BD7E, Attachment: Host-only Interface 'vboxnet1'


❯ VBoxManage showvminfo machine_0 | grep NIC | grep -v disabled
NIC 1:  MAC: 08002715BD34, Attachment: Host-only Interface 'vboxnet1', Cable connected: on, Trace: off (file: none), Type: Am79C970A, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth group: none

Configure machine_0 boot order as Network first

Change it here: (System -> Motherboard -> Boot Order) in virtualbox if you don’t like to get into BIOS.

1
2
3
4
5
6
❯ VBoxManage showvminfo machine_0 | grep "Boot "
Boot menu mode:              message and menu
Boot Device 1:               Network
Boot Device 2:               HardDisk
Boot Device 3:               Not Assigned
Boot Device 4:               Not Assigned

Configure Cobbler, vRouter and DHCP Relay

Install cobbler in cobbler_server and install dhcprelay in vrouter

We have to complie it ourselves :( 

1
2
3
4
5
6
7
ubuntu@cobbler_server:~$ wget https://github.com/cobbler/cobbler/archive/v3.0.1.tar.gz
ubuntu@cobbler_server:~$ wget tar -xzvf *.tar.gz
ubuntu@cobbler_server:~$ cd cobbler-3*
ubuntu@cobbler_server:~$ sudo apt install python3-pip apache2 apache2-dev -y
ubuntu@cobbler_server:~$ sudo python3 -m pip install -r requirements.txt
ubuntu@cobbler_server:~$ sudo make install
ubuntu@cobbler_server:~$ sudo apt-get install xinetd python-pykickstart rsync

There are issues on Ubuntu 18.04, switched to CentOS epel-release stock cobbler instead…

Install Cobbler following https://cobbler.github.io/quickstart

Enable vrouter dhcp relay to forward dhcp

1
2
3
4
5
ubuntu@vrouter:~$ sudo apt install isc-dhcp-relay

ubuntu@vrouter:~$ ps -ef | grep dhcrelay
root      1483     1  0 00:59 ?        00:00:00 dhcrelay -i enp0s8 192.168.56.11
root      1485     1  0 00:59 ?        00:00:00 dhcrelay -i enp0s9 192.168.56.11

configure networks

  • configure nics IP for cobbler_server and vrouter
  • in cobbler: add route for vboxnet1 via vrouter
  • in vrouter: enable ipv4 forwarding

cobbler_server

1
2
3
4
5
6
7
8
9
10
11
12
root@cobbler_server:/home/ubuntu/# cat /etc/netplan/50-cloud-init.yaml
network:
    ethernets:
        enp0s3:
            dhcp4: true
        enp0s8:
            dhcp4: false
            addresses:
              - 192.168.56.11/24
    version: 2
root@cobbler_server:/home/ubuntu/# netplan apply
root@cobbler_server:/home/ubuntu/# ip route add 192.168.57.0/24 via 192.168.56.10

vrouter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
root@vrouter:/home/ubuntu# cat /etc/netplan/50-cloud-init.yaml

network:
    ethernets:
        enp0s3:
            dhcp4: true
        enp0s8:
            dhcp4: false
            addresses:
              - 192.168.56.10/24
        enp0s9:
            dhcp4: false
            addresses:
              - 192.168.57.10/24
    version: 2

root@vrouter:/home/ubuntu/# netplan apply

root@vrouter:/home/ubuntu# grep forward /etc/sysctl.conf
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1

Ping test from cobbler_server to vrouter

1
2
3
4
5
6
7
root@cobbler_server:/home/ubuntu# ping 192.168.57.10 -c 1
PING 192.168.57.10 (192.168.57.10) 56(84) bytes of data.
64 bytes from 192.168.57.10: icmp_seq=1 ttl=64 time=0.437 ms

--- 192.168.57.10 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.437/0.437/0.437/0.000 ms

Configure cobbler_server

I switched OS for cobbler_server from Ubuntu 18.04 to CentOS 7 as cobbler worked fine in CentOS out-of-box

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@cobbler-server ~]# grep "server: 192" /etc/cobbler/settings
next_server: 192.168.56.11
server: 192.168.56.11

[root@cobbler-server ~]# openssl passwd -1 -salt 'passwd' 'cobbler'
$1$passwd$uNhicNpING2yTBiLmcMOY.

[root@cobbler-server ~]# grep ^default_pass /etc/cobbler/settings
default_password_crypted: "$1$passwd$uNhicNpING2yTBiLmcMOY."

[root@cobbler-server ~]# grep manage_dhcp /etc/cobbler/settings
manage_dhcp: 1

[root@cobbler-server ~]# cobbler check
The following are potential configuration items that you may want to fix:

1 : debmirror package is not installed, it will be required to manage debian deployments and repositories
2 : fencing tools were not found, and are required to use the (optional) power management features. install cman or fence-agents to use them

enable cobbler to dhcp subnet B

reference: https://docs.cumulusnetworks.com/cumulus-linux-31/Configuring-and-Managing-Network-Interfaces/Configuring-DHCP-Relays-and-Servers/

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@cobbler-server ~]# vi /etc/cobbler/dhcp.template

shared-network Combined-pools {

   subnet 192.168.56.0 netmask 255.255.255.0 {
       # subnet A
   }

   subnet 192.168.57.0 netmask 255.255.255.0 {
       # subnet B
        option routers             192.168.57.10;
        option subnet-mask         255.255.255.0;
        range dynamic-bootp        192.168.57.100 192.168.57.254;
        default-lease-time         21600;
        max-lease-time             43200;
        next-server                $next_server;
        class "pxeclients" {
             match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
             if option pxe-system-type = 00:02 {
                     filename "ia64/elilo.efi";
             } else if option pxe-system-type = 00:06 {
                     filename "grub/grub-x86.efi";
             } else if option pxe-system-type = 00:07 {
                     filename "grub/grub-x86_64.efi";
             } else if option pxe-system-type = 00:09 {
                     filename "grub/grub-x86_64.efi";
             } else {
                     filename "pxelinux.0";
             }
        }

   }

}

import an image

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
[root@cobbler-server ~]# mount -o loop CentOS-7-x86_64-Minimal-1908.iso /mnt/CentOS-7
mount: /dev/loop0 is write-protected, mounting read-only

[root@cobbler-server ~]# cobbler import --arch=x86_64 --path=/mnt/CentOS-7/ --name=CentOS7
task started: 2020-03-26_144343_import
task started (id=Media import, time=Thu Mar 26 14:43:43 2020)
Found a candidate signature: breed=suse, version=opensuse15.0
Found a candidate signature: breed=suse, version=opensuse15.1
Found a candidate signature: breed=redhat, version=rhel6
Found a candidate signature: breed=redhat, version=rhel7
Found a matching signature: breed=redhat, version=rhel7
Adding distros from path /var/www/cobbler/ks_mirror/CentOS7-x86_64:
creating new distro: CentOS7-x86_64
trying symlink: /var/www/cobbler/ks_mirror/CentOS7-x86_64 -> /var/www/cobbler/links/CentOS7-x86_64
creating new profile: CentOS7-x86_64
associating repos
checking for rsync repo(s)
checking for rhn repo(s)
checking for yum repo(s)
starting descent into /var/www/cobbler/ks_mirror/CentOS7-x86_64 for CentOS7-x86_64
processing repo at : /var/www/cobbler/ks_mirror/CentOS7-x86_64
need to process repo/comps: /var/www/cobbler/ks_mirror/CentOS7-x86_64
looking for /var/www/cobbler/ks_mirror/CentOS7-x86_64/repodata/*comps*.xml
Keeping repodata as-is :/var/www/cobbler/ks_mirror/CentOS7-x86_64/repodata
*** TASK COMPLETE ***
[root@cobbler-server ~]# cobbler signature update
task started: 2020-03-26_144402_sigupdate
task started (id=Updating Signatures, time=Thu Mar 26 14:44:02 2020)
Successfully got file from https://cobbler.github.io/signatures/2.8.x/latest.json
*** TASK COMPLETE ***


[root@cobbler-server ~]# cobbler distro list
   CentOS7-x86_64

[root@cobbler-server ~]# cobbler distro report --name=CentOS7-x86_64
Name                           : CentOS7-x86_64
Architecture                   : x86_64
TFTP Boot Files                : {}
Breed                          : redhat
Comment                        :
Fetchable Files                : {}
Initrd                         : /var/www/cobbler/ks_mirror/CentOS7-x86_64/images/pxeboot/initrd.img
Kernel                         : /var/www/cobbler/ks_mirror/CentOS7-x86_64/images/pxeboot/vmlinuz
Kernel Options                 : {}
Kernel Options (Post Install)  : {}
Kickstart Metadata             : {'tree': 'http://@@http_server@@/cblr/links/CentOS7-x86_64'}
Management Classes             : []
OS Version                     : rhel7
Owners                         : ['admin']
Red Hat Management Key         : <<inherit>>
Red Hat Management Server      : <<inherit>>
Template Files                 : {}


[root@cobbler-server ~]# cobbler system add --name=test --profile=CentOS7-x86_64


[root@cobbler-server ~]# cobbler system report --name=test
Name                           : test
TFTP Boot Files                : {}
Comment                        :
Enable gPXE?                   : <<inherit>>
Fetchable Files                : {}
Gateway                        :
Hostname                       :
Image                          :
IPv6 Autoconfiguration         : False
IPv6 Default Device            :
Kernel Options                 : {}
Kernel Options (Post Install)  : {}
Kickstart                      : <<inherit>>
Kickstart Metadata             : {}
LDAP Enabled                   : False
LDAP Management Type           : authconfig
Management Classes             : <<inherit>>
Management Parameters          : <<inherit>>
Monit Enabled                  : False
Name Servers                   : []
Name Servers Search Path       : []
Netboot Enabled                : True
Owners                         : <<inherit>>
Power Management Address       :
Power Management ID            :
Power Management Password      :
Power Management Type          : ipmitool
Power Management Username      :
Profile                        : CentOS7-x86_64
Internal proxy                 : <<inherit>>
Red Hat Management Key         : <<inherit>>
Red Hat Management Server      : <<inherit>>
Repos Enabled                  : False
Server Override                : <<inherit>>
Status                         : production
Template Files                 : {}
Virt Auto Boot                 : <<inherit>>
Virt CPUs                      : <<inherit>>
Virt Disk Driver Type          : <<inherit>>
Virt File Size(GB)             : <<inherit>>
Virt Path                      : <<inherit>>
Virt PXE Boot                  : 0
Virt RAM (MB)                  : <<inherit>>
Virt Type                      : <<inherit>>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[root@cobbler-server ~]# cobbler system edit --name=test --gateway=192.168.57.10 --hostname=machine_0

[root@cobbler-server ~]# cobbler system report
Name                           : test
TFTP Boot Files                : {}
Comment                        :
Enable gPXE?                   : <<inherit>>
Fetchable Files                : {}
Gateway                        : 192.168.57.10
Hostname                       : machine_0
Image                          :
IPv6 Autoconfiguration         : False
IPv6 Default Device            :
Kernel Options                 : {}
Kernel Options (Post Install)  : {}
Kickstart                      : <<inherit>>
Kickstart Metadata             : {}
LDAP Enabled                   : False
LDAP Management Type           : authconfig
Management Classes             : <<inherit>>
Management Parameters          : <<inherit>>
Monit Enabled                  : False
Name Servers                   : []
Name Servers Search Path       : []
Netboot Enabled                : True
Owners                         : <<inherit>>
Power Management Address       :
Power Management ID            :
Power Management Password      :
Power Management Type          : ipmitool
Power Management Username      :
Profile                        : CentOS7-x86_64
Internal proxy                 : <<inherit>>
Red Hat Management Key         : <<inherit>>
Red Hat Management Server      : <<inherit>>
Repos Enabled                  : False
Server Override                : <<inherit>>
Status                         : production
Template Files                 : {}
Virt Auto Boot                 : <<inherit>>
Virt CPUs                      : <<inherit>>
Virt Disk Driver Type          : <<inherit>>
Virt File Size(GB)             : <<inherit>>
Virt Path                      : <<inherit>>
Virt PXE Boot                  : 0
Virt RAM (MB)                  : <<inherit>>
Virt Type                      : <<inherit>>

lines to debug/verify tftp and dhcp

1
2
tftp -m binary "192.168.56.11" -c get "pxelinux.0"
dhclient

Verify the PXE boot via L3, DHCP Relay

Start machine_0, and it worked ;-).

Start machine_0, and it worked ;-).

From log we could see machine_0 PXE nic got its DHCP IP and default route asssigned in subnet B (192.168.57.0/24) via cobbler-server in subnet A (192.168.56.0/24).

Then the PXE boot succeeded in L3 network routed via our v_router.

cobbler_server

1
2
3
4
5
6
7
8
9
10
11
12
[root@cobbler-server ~]# tail /var/log/messages

Mar 29 02:16:01 cobbler-server dhcpd: DHCPDISCOVER from 08:00:27:15:bd:34 via 192.168.57.10
Mar 29 02:16:02 cobbler-server dhcpd: DHCPOFFER on 192.168.57.101 to 08:00:27:15:bd:34 via 192.168.57.10
Mar 29 02:16:02 cobbler-server dhcpd: DHCPREQUEST for 192.168.57.101 (192.168.56.11) from 08:00:27:15:bd:34 via 192.168.57.10
Mar 29 02:16:02 cobbler-server dhcpd: DHCPACK on 192.168.57.101 to 08:00:27:15:bd:34 via 192.168.57.10
Mar 29 02:18:58 cobbler-server in.tftpd[12800]: Client ::ffff:192.168.56.10 finished pxelinux.0

[root@cobbler-server ~]# tail /var/log/cobbler/cobbler.log

Sun Mar 29 02:16:07 2020 - INFO | REMOTE generate_kickstart; user(?)
Sun Mar 29 02:16:07 2020 - INFO | generate_kickstart

vrouter

1
2
3
4
5
6
7
8
root@vrouter:/home/ubuntu# tcpdump -i enp0s8 -n port 67 and port 68 and -i enp0s9 -n port 67 and port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s9, link-type EN10MB (Ethernet), capture size 262144 bytes
06:43:21.321035 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:15:bd:34, length 380
06:43:22.305331 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:15:bd:34, length 380
06:43:22.323428 IP 192.168.57.10.67 > 192.168.57.100.68: BOOTP/DHCP, Reply, length 300
06:43:24.282418 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 08:00:27:15:bd:34, length 392
06:43:24.297630 IP 192.168.57.10.67 > 192.168.57.100.68: BOOTP/DHCP, Reply, length 300

screen captured for machine_0

Issue notes

  1. I tried to complie cobbler in Ubuntu 18.04 while failed in several phases, to save time switched to epel-release build Cobbler binary package in CentOS 7

  2. default vNIC type not working properly on tftp, corrected for machine_0

  3. dhcp server configuration, I shouldn’t have put DHCP for subnet A(was removed already), in which case, machine_0 will be with IP address from subnet A while gateway in subnet B and it led L3 network failure, see below failure: