I went through this experience recently. First of all, don’t panic! I panicked, and because of this, I made a mistake: I didn’t wait long enough for it to come online. Had I waited up to 60 minutes, it would probably have come online (see reason below). The story:
I had broken packages on my Ubuntu 10.04 server and decided to fix them by
apt-get update apt-get upgrade
While updating, I noticed that the package grub-pc also was upgraded, apparently a new bootloader (or bootloader configuration) was installed. This made me feel uncomfortable, since I didn’t know if the server would reboot after this upgrade. So, because of the saying “The devil you know is better than the devil you don’t know” (and a desire to sleep peacefully at night!) I decided to reboot the server and see what would happen. To my big dismay, it did not come online. SSH connections failed with “Port 22: Connection refused” .
I panicked and asked the Hetzner Support (which is very responsive and supportive btw!) to install a LARA Remote Console so that I could see the text output of the booting screen. After some regular startup text, the screen became blank. I panicked even more and took immediate steps to move all data to a new server. It took me 6 hours to complete the most important parts, and another full week of restless work to finish it. We will see that it was not necessary.
Most regular servers at the Hetzner data center are running Software RAID. It seems that after a reboot (especially if you send a Hardware Reset) the OS needs some time to re-sync or check the file system. I am not sure what caused the delay in my case. Re-syncing the entire RAID array can take up to 1-2 hours, depending on your hardware and disc space.
So, wait at least 1 hour for it to come online, especially after a hardware reset! When you activate Hetzner’s Rescue system (which is very good btw!) it will stay active for a minimum of 1 hour, so your server will be down for 1 hour at least, in any event. So you are not losing much by waiting a bit longer.
Now, in my case, I assumed that Grub2 was broken. So I activated the Hetzner Rescue System, booted into it, and reinstalled Grub2. I have found the following method here and it worked for me. First you have to mount the regular RAID filesystem under /mnt :
mount /dev/md2 /mnt mount /dev/md1 /mnt/boot mount -t dev -o bind /dev /mnt/dev mount -t proc -o bind /proc /mnt/proc mount -t sys -o bind /sys /mnt/sys chroot /mnt
At this point, you are in your regular root directory. To reinstall Grub2 the Debian Way, I did:
apt-get install --reinstall grub-pc
To make really sure, I reconfigured the package:
It will ask you where to install the bootloader. I selected:
[*] /dev/sda [*] /dev/sdb
No errors were reported. I rebooted again and it did not come online immediately, for the reasons previously mentioned. I waited long enough (in my case, 15 minutes) and it did come online. So, rule number one is: Don’t panic!
35 thoughts on “Do not Panic! Remote Server (Hetzner) not rebooting any more – A Solution”
Thanks a lot, that saved my day!
Thank you very much, you saved me a weekend full of headaches!
Thank you and thank you again!!!!
Great great great tutorial! Thanks a lot Michael.
Thank you very much for this excellent tutorial!!!
I think I have a similar problem with Hetzner. After reboot the server will be down, not sure for how long.
Do this method works on Ubuntu 12.04 LTS 64bit? and on RAID 0?
It should also work on Ubuntu, because Ubuntu is based on Debian. Not sure about RAID 0 though…
Thank you. You saved my day.
You provided me more details than the Hetzner’s wiki, Hetzener’s wiki some times is not detailed at all, so it doesn’t help much.
BTW, I used this method to reinstall Grub over Centos 6 and it worked, from Debian rescue mode of Hetzner.
Thank you! You saved me a lot of time and probably money!!!
Thank you !!! saved me a lot of time and probably money !
I have an issue, I only can access from rescue mode, after do apt-upgrade
anyone can help me? because when I reboot my pc, I can’t access
My server needed 45 min to boot. I already got a LARA up and running. But i couldn’t find out what took so long. fsck is disabled and raid resync should not prevent booting. Thanks a lot for the tip!
WOOHOO!! Thanks for this!
I had a server with Ubuntu + Software RAID 1. Drive sda died and was replaced, but the machine wouldn’t boot. Host temporarily set it to boot from sdb, which worked. After resync, I just did the steps starting at:
Picked both the new sda + sdb for GRUB to install on, and rebooted. DONE! 🙂
I tried this and waited but it did not help. Is there any other public software which can block the boot?
Thank you, you saved my night!
Thanks much appreciated. my harddrive died and i didnt eject it out of the sytem in time (hetzner support was damn quick to replace it).
I booted the rescue system and partitioned the hard drive as explained in the hetzner wiki, but wasn’t sure about the whole chroot part at the end. I followed your chroot instructions and am currently resyncing the drives. Fingers crossed it works.
If anyone reads this: Make sure when your drive dies to eject it from the system before calling support. it saves you a lot of headache as you won’t need to fiddle around with the rescue system etc….
This was awesome. You sir have my gratitude !
Totally saved my bacon on this one. The Hetzner Wiki is not clear on this point, as it makes it sound like the rescue system goes away when you reboot. Also, there is the added issue of having to sync the raid drives if you made the mistake of using a hardware reboot. Always try the soft reboot first!
You saved my life – or at least my server!
kudos man….really helpfull.
Thanks. Works fine. Saved me hours…
Thanks! This saved my day. Again (iirc this was the second time already).
Thanks that worked, thanks again
Another day saved by you. I was not mounting /boot. Thanks!
WOW! COOL! Saved a lot of time for me as well!
Great guide, thanks! It gave me a starting point but I then ran into “grub-install: error: disk `md1′ not found.”
Turns out mdadm wasn’t installed. That was all. Hopefully my server will come up now 😐
This is where I got the mdadm idea: https://bugs.launchpad.net/ubuntu/+source/grub-installer/+bug/1315344
Just Perfect !!! Saved my night !!! Can’t believe it still works after so many years !!!
Thank you so much!
Aah. Thank you. Mine took 4 hours to boot.
I upgraded the kernel and I got the same problem. I did all the steps here, grub updates and installs without problems but when I restart the system does not come back. I wrote to hetzner support and they attached a kvm console and I saw that the system is halted saying: “Operating system not found… etc.”. When I press enter, the system boots normally.
I checked the disks with fdisk -l and saw that no partition is selected as boot partition so I flagged the /boot partition as boot but still did not solve my problem. I think it might be something about the MBR but could not find a solution yet.
My current situation is that if I want to restart my system I will have to call hetzner support and ask them to attach a KVM console so I can press enter. Anyone has any ideas about a fix?
i have the same problem
Thanks – this helped a lot. I needed an additional step to get it back running:
# apt-get install mdadm
Before reinstalling grub.
Hope this helps someone.