A crazy server recovery

If you’re a seasoned server administrator then this post will boring beyond oblivion to you; the ingredients:

a headless server (no screen, keyboard)
years of personal backups on a linux software RAID
a botched system upgrade
happy ending

The pretext

Several years ago I bought an HP Proliant microserver on discount and set up a NAS with Ubuntu 14.04 LTS. The Proliant is a 64 bit x86 architecture with some fancy HP management BIOS which is accessible over HTTP. The NAS consists of an SSD with Ubuntu installed and three rotational HDDs which make up an mdadm RAID 1 (mirrors). I remember that getting this to run in the first place was a day long task since I didn’t have any screen or keyboard, there was a lot of trial and error and the Proliant server wouldn’t boot from a GTP partitioned harddisk but required an MBD partitioned harddisk. So I avoided upgrading that system for a long time. Until yesterday, when I typed “apt-get upgrade” into the wrong window.

The journey

After upgrading Ubuntu the server wouldn’t boot anymore. So I booted up VirtualBox with an Ubuntu 18 server image on my laptop, connected the NAS SSD to the laptop and installed Ubuntu 18 server on the NAS’ SSD. After reinstalling the SSD in the NAS, the server still doesn’t boot. Worse, the SSD doesn’t boot in my laptop because of a weird error:

Timed out waiting for device ….

It turns out that most of the time Ubuntu won’t boot over the USB SSD, but sometimes it will, which is just enough to fix a few things, namely adding “rootdelay=20” to /etc/default/grub and running a grub-update after that.

After that, the SSD would at least boot on my laptop over USB.

I re-installed the SSD into the Proliant and rebooted. Then I connected to ILO with a web browser which opens a console (like a screen in the browser) to the server. Unfortunately that requires the Java plugin which isn’t available in modern Ubuntu/Firefox combinations (or other browsers).

So I need an old browser 🙂 I installed an old Windows XP ISO into VirtualBox and tried to access ILO with IE6, but IE6 would connect to Bing only and no other site. I then downloaded Firefox 52 and Java 6 into a folder on the host computer, shared it with the virtual machine and installed both in the VM. FF52 was able to access ILO and run the remote console.

ifconfig shows no ethernet cards (the Proliant has 2 + 1 management port) so my first fear is a driver issue, but “lshw -C network” showed both cards as disabled. Ubuntu 18 uses netplan for network management, so I took a peak at /etc/netplan/50-cloud-init.yaml, which listed a card that wasn’t installed. So I added both cards which lshw showed:

network:
    ethernets:
        enp2s0:
            dhcp4: true
        eno1:
            dhcp4: true
        eno2:
            dhcp4: true
    version: 2

(Note that I don’t own an ILO license, so the remote console closes every 30 seconds. It’s much easier to only run exploratory commands in the console, save output to files and then mount the SSD to my laptop and edit files there).

After a reboot the server shows up with an IP in the router management interface and I was able to SSH to it 🙂