Luckily, it's an easy fix
Apparently the installer adds both interfaces to the netplan, so it happily waits for DHCP to assign a network address for the full two minutes (even if there's no ethernet cable attached)
To fix, edit /etc/netplan/00-installer-config.yaml and remove the interface you're not using.
For example, the file contents will look like the following:
# This is the network config written by 'subiquity'
network:
ethernets:
enp2s0:
dhcp4: true
enp3s0:
dhcp4: true
version: 2
Since there is only a network cable connected to 'enp2s0', we can simply remove the 'enp3s0' lines. (note: only changing the dhcp4 value to 'false' does NOT resolve the issue, removing the lines does)