Tuesday, 11 Oct, 2016

Recently, one of my backup servers started complaining - during its latest reboot, I ended up with an emergency boot prompt loop!

The server was a Fedora (but the workaround should also apply to almost every CentOS, Ubuntu and maybe to other distros too!).

Why no regular boot up?

Here's a 'shot of what I got:

I'll paste the error for the search engines:

*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type control-D to continue):

By interpreting the above error, it seems there's an issue while checking the filesystems.

The md arrays looks alright ("clean"), although there's a problem with /dev/sdc1 ("No such file or directory while opening /dev/sdc1").

IMHO, it's a good practice to keep an external HDD plugged (and maybe also rsync'ed).

As I discovered with my work-mates, that was the case, 'though (for reasons unknown), it seems the external HDD had been removed.

I also discovered the external HDD automount was (manually) hardcoded in fstab.

In conclusion, to fix the issue, I had to patch fstab.

'Thoug I couldn't edit fstab since the "slash" was mounted Read-Only.

After searching through endless posts and forums (OK, I may have used the wrong terms...), I just found what I needed:

mount -n -o remount,rw /

The above command mounted the ROOT (aka slash), as a READ-WRITE filesystem.

After mounting slash in RW-mode, I was able to manually patch fstab, namely, by commenting out the offending drive.

MOUNT-command details.

The "-n" parameter prevents the mount command to populate the mtab lines (while mounting devices).

The "-o" parameter is a switch to specify additional, comma-separated options (similar to the options used inside fstab - ie. 'defaults', 'sync', 'noatime', etc.).

The "remount" option tries to remount the fs as-is (since it is already mounted RO).

Finally, the "rw" option enables read & write mode for the desired filesystem.

Once done, I rebooted the backup box and ta-daaa! (Windows 3.1 welcome sound): onsite backup server back online.

2.7/5 - (4 votes)