I’ve been working on having an Opnsense backup after experiencing a catrastophic failure recently. I wrote enough about it here and here, but I need to now document my main plan of recovery in the event of it happening again.

Setting up a dormant emergency Opnsense VM

The aim of this is just to have an up-to-date Opnsense VM install ready to go if (when?) the main firewall box dies. The idea is that this is booted every week or two and updates are installed. The VM then goes back to being dormant… The updates are really so that if the config.xml is restored it will have a higher chance of not messing something up if the Opnsense VM version isn’t months or years behind the machine which created the backup.

I figure it takes all of 5mins to run an update every now and then, rather than going to find a WiFi network (or use mobile) to download a Opnsense install iso and fresh install when I’m already stressed and network-less.

However, the VM needs to be on an isolated network so that the DHCP server doesn’t start messing with anything. I could enable a DHCP server later, but I figure I’ll be a bit frantic and just want something that makes sense to my brain. Here’s what I did to set up the VM.

Step 1 - Prepare Proxmox network adapters

To avoid issues with DHCP, three network bridges are made in Proxmox.

vmbr0 - Connected to the WAN Opnsense VM to get an internet connection for updates. Ordinarily the network LAN.
vmbr1 - Second NIC of the Proxmox host. Disconnected until catastropic failure.
vmbr2 - An isolated, temporary LAN just in use until catastrophic failure.

vmbr2 can’t be on the same subnet as the rest of the network, so the interface has to be changed. Follow the steps below for Set interface IP address in the Opnsense console. This can be largely anything, as long as it’s not the same as the other subnet. E.g. Opnsense IPv4 address at 192.168.250.1 and IPv4 range of 192.168.250.100 - 192.168.250.200.

Step 2 - Install Opnsense VM

Simple enough. Keep as much stock as possible. Assign vmbr0 as WAN and vmbr2 as LAN. It’s also a reasonable idea to pre-install the same plugins as the main router install so that the config.xml restore is as smooth as possible.

The plan then is just boot the Opnsense VM every week or so, login via the console in Proxmox and select option 12) Update from console and then shut it down once done.

Recovering from catastrophic failure

Step 1 - Getting into Proxmox

Ok, so shit just got real. Go and get a coffee, a dongle and a LAN cable. For anyone else reading this, it is very much written as me talking to my frantic self.

Now connect a LAN cable from the modem to the spare NIC on the Proxmox box. Disconnect the cable connecting the Proxmox box to the switch and plug it directly into your MacBook.

Set a static IP & subnet mask on your MacBook to connect directly to Proxmox, e.g, 192.168.1.* and a subnet mask of 255.255.255.0. Wait for the connection to be live and then connect in a web browser to the static Proxmox IP address. Breath, it’s straightforward from here.

Step 2 - Opnsense tweaks

vmbr0 has been used as a wan port to keep the VM updated. In the Opnsense VM hardware settings, select the network interfaces and ensure that they are disconnected in the VM (there’s a disconnect checkbox). You can connect them once the interface and DHCP range has been set.

Boot the Opnsense VM & login to the console. The login is the default (root/opnsense), but will change once the configuration is restored.

First set the correct interfaces. Select Assign interfaces (option 1) and:

Set vmbr0 as LAN
Set vmbr1 as WAN
Don't worry about vmbr2 - that was the temporary isolated LAN

The MAC addresses of the adapters are in the VM hardware page in Proxmox to make sure it’s the right bindings.

Now select option 2 Set interface IP address as we have to setup DHCP.

- Enter the number of the interface to configure: `1` (LAN!)
- Configure  IPv4 address LAN interface via DHCP?: `N`
- Enter the new LAN IPv4 address:  `192.168.1.1`
- Enter the new LAN IPv4 subnet bit count (1 to 32): `24`
- For a LAN, press <ENTER> for none:  <Enter>
- Configure IPv6 address LAN interface via WAN tracking: "N"
- Configure IPv6 address LAN interface via DHCP6?: "N"
- Enter the new LAN IPv6 address. Press <ENTER> for none:  <Enter>
- Do you want to enable the DHCP server on LAN?: "Y"
- Enter the start address of the IPv4 client address range: "192.168.1.2"
- Enter the end address of the IPv4 client address range: "192.168.1.254"
- Do you want to change the GUI protocol from HTTPS to HTTP: "N"
- Do you want to generate a new self-signed web GUI certificate?: "Y"
- Restore web GUI access defaults?: "N"

Now reconnect the interfaces by unchecking the ‘disconnect’ box in the VM Proxmox hardware settings. Hopefully the Opnsense interface is now reachable as the laptop is on the same subnet as Opnsense.

Reboot!

Step 3 - Restore the configuraton

The Opnsense VM should also now have internet connectivity and the DHCP server should be issuing IP addresses. On your still-connected Macbook, head to 192.168.1.1, login with root/opnsense and restore your config.xml from the last backup.

The restore will fail, but only because it needs the interface assignments to be reset from the dead firewall to the Opnsense mappings. Do this and then restore config.xml again. Also check if plugins are all restored and working ok.

Lastly, change the Dnsmasq DHCP option of Pihole DNS server to 192.168.1.1 (Unbound) and re-enable blocklists if necessary. PiHole is fabulous, but there were issues with it when Opnsense was previously virtualised on the same host Proxmox machine. Unbound works just fine.

Step 4 - Connect Proxmox back to switch

Disconnect your Macbook from the Proxmox host and plug it into the switch. Hopefully you now have a working network again, although many devices will need to be power cycled to receive the new DNS server address.

Breath, sip that coffee and look for any problems. They will probably be DNS. It’s always DNS.

Hopefully these instructions should save me from some pain in the future and maybe be helpful to anyone else looking to have a VM backup of their router. May your routers be secure and functional and your networks be up and stable.