Vagrant hangs with message “Waiting for domain to get an IP address…”

The problem may be (it was in my case) that there are firewall rules preventing NAT firewall rules for the virbr0 network device created by Vagrant via libvirt, which may look like the following (excerpt of iptables -L -n):

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            192.168.121.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.121.0/24     0.0.0.0/0

A possible fix is to disable those restrictive rules (for example, clear all iptables rules before starting the Vagrant machine).


Wake-on-LAN (WoL) in Linux is disabled by default. How to enable it without ethtool.

Accomplishing the simple often takes a day or two.

According to countless tutorials on the internet, in order for a working WoL, you need to fulfill the following preconditions (where point 4 below is rarely mentioned [that’s why this article exists]):

  1. An ethernet hardware/card supporting WoL
  2. Enabling WoL in the BIOS
  3. Enabling the WoL feature on the ethernet card (usually using the user space tool ethtool)
  4. Enabling the WoL feature in the Linux kernel (and this was the major pitfall for me)

Support of WoL can be checked for a given ethernet card in the following way:

ethtool <card_label> | grep Wake-on

According to my own experiments on my local hardware, WoL seems to always be enabled by default:

Supports Wake-on: pumbg
Wake-on: g 

Because it already prints Wake-on: g, signalling that WoL is enabled (for Magic Packets), one is tempted to not run ethtool -s <card_label> wol g to achieve the same.

However, this ethtool command does at least one other thing, being equivalent to the following:

echo enabled > /sys/class/net/<card_label>/device/power/wakeup

And the main pitfall is: After a reboot, cat /sys/class/net/<card_label>/device/power/wakeup will return disabled.

That is why attempts to make WoL work without ethtool may fail. The feature will stay disabled in the Linux kernel, which will completely shut off the ethernet card in most powersave and power-off states. (I verified this observing the LEDs of a connected ethernet switch).

Conclusion

On modern Linux systems, given supporting hardware, it seems to be enough to write the string enabled to /sys/class/net/<card_label>/device/power/wakeup to make WoL work. ethtool doesn’t seem to be required any more for that, since Ethernet cards seem to already be configured properly by default. The Ethernet card will stay powered in most power-save and power-off states, so that it can receive Wake-on-LAN packets.

Resources

https://www.kernel.org/doc/html/v4.14/driver-api/pm/devices.html
https://www.kernel.org/doc/Documentation/power/states.txt

“Green Energy” complication: gdm3 suspends machine after 20 minutes: A solution

I just posted a bug report on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=896083

The solution to the problem is further down in the bug report.

Dear Maintainer,

gdm3’s default dconf energy settings suspend the machine after 20 minutes.

This is independent of the power settings made by an unprivileged user within a Gnome login session.

While this could be forgiven on a locally accessible desktop machine, it also suspends remote/headless machines (e.g. in a data center). Activity on a SSH terminal or VNC connection does not prevent this issue. Having no easy way to re-wake remote machines, this may create highly inconvenient situations for administrators. In addition, unexpected suspension may also have disastrous consequences, depending on the use of the machine.

To reproduce, install task-gnome-desktop and wait for 20 minutes on a machine which supports power management.

The offending settings can be printed to the console. As superuser:

su -s /bin/bash Debian-gdm

unset XDG_RUNTIME_DIR

dbus-launch gsettings get org.gnome.settings-daemon.plugins.power sleep-inactive-ac-type

dbus-launch gsettings get org.gnome.settings-daemon.plugins.power sleep-inactive-ac-timeout

This prints ‘suspend’ and ‘1200’, respectively.

For quicker reproduction of the problem, reduce the timeout to 2 minutes:

dbus-launch gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-ac-timeout 120

Then reboot and wait 2 minutes.

To turn off suspension, set:

dbus-launch gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-ac-type nothing

Regards,
Michael Franzl

Obscure error: dd: failed to open: No medium found

I got this error message when I wanted to use dd  to copy a raw image to a MicroSD card. You will see this message if you have unmounted the card with a file manager like Nautilus. I guess it unmounts not fully, or in a special way.

The fix is easy:

Unmount the card via the terminal:

umount /dev/sdd1

The dd  command will work now.

Another variant of this error can be seen when using bmaptool :

bmaptool: ERROR: cannot open destination file '/dev/sdd': [Errno 123] No medium found: '/dev/sdd'

Symlinks within shared folders in VirtualBox: Operation not permitted and Read Only Filesystem

This is not a bug, it is a security feature of Virtual Box. Nevertheless it is annoying when you want to use your virutal system as a build system which needs symlinks in shared folders. To enable symlink creation, do the following:

Shut down your virtual machine, close all VirtualBox windows, including the main GUI window

VBoxManage setextradata MACHINE_LABEL VBoxInternal2/SharedFoldersEnableSymlinksCreate/SHARE_LABEL 1

For example, if your virtual machine is called “Debian Wheezy” (with a space), you have to escape that space with a backslash. In my case, I had to do:

VBoxManage setextradata Debian\ Wheezy VBoxInternal2/SharedFoldersEnableSymlinksCreate/Public 1

To check if your setting has been taken, run

VBoxManage getextradata MACHINE_LABEL enumerate

Restart your virtual machine and creating of symlinks should work. I found that creating hard links still doesn’t work with this method. If anyone knows a fix for hard links, please let me know!

A solution for MySQL Assertion failure FIL_NULL

A defective RAM module recently caused data corruption in MySQL tables. MySQL would log the following to /var/log/syslog  in regular intervals, about every few minutes:

140125 5:04:41 InnoDB: Assertion failure in thread 140046518261504 in file fut0lst.ic line 83
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA
InnoDB: We intentionally generate a memory trap.

Reading MySQL documentation and various blogs didn’t help much. I ran CHECK TABLES  on all the tables and they all reported OK. Then I ran

mysqlcheck --all-databases

and still all tables reported OK. Nevertheless the Assertion Failures continued. Then I stumbled across this excellent blog post, which suggested to dump evertying into a .sql file, wipe /var/lib/mysql (not without making a backup, mind you!), and re-import everything from scratch. This is what I did and it worked.

I recorded the passwords for each database and collected it into a SQL script like this:

GRANT ALL PRIVILEGES ON database_name1 TO 'username1'@'localhost' IDENTIFIED BY 'password1';
GRANT ALL PRIVILEGES ON database_name2 TO 'username2'@'localhost' IDENTIFIED BY 'password2';
etc.

When you dump all databases into a .sql file, it will not dump the permissions, so you will need to restore them later with this script. Next, the dumping part, then removal and reinstallation of mysql (Danger here: When you remove mysql-server, all packges which depend on it also will be removed!):

cd ~
mysqldump -u root -ppass --all-databases > alldbs.sql
cd /var/lib
cp -vpr mysql mysql-backup
apt-get remove --purge mysql-server-5.5
apt-get install mysql-server

Here, I had to reset the MySQL admin password because it didn’t work any more, so I ran:

dpkg-reconfigure mysql-server-5.5

Then, I re-imported all databases from the dump file:

cd ~
mysql -u root -ppass < alldbs.sql

Then I run the SQL permission script that I mention above.

For me, this resulted in no more Assertion Failures. Yay!

 

 

Adventures with various Segfaults due to defective RAM

If you get various segfaults on your Linux server, like these:

spamd child[2656]: segfault at 200251c208 ip 00007fa039223684 sp 00007fff77953680 error 4 in libperl.so.5.14.2[7fa03916a000+177000]

or:

clamd[3311]: segfault at 1000000008 ip 00007f00200b3751 sp 00007fff3e2cef60 error 4 in libclamav.so.6.1.17[7f001fff1000+988000]

or

php5[14914]: segfault at 7fff7d2939c8 ip 00000000006bf04d sp 00007fff6d293860 error 6 in php5[400000+6f3000]

or

PassengerHelper[11644]: segfault at ffffffffca4ef420 ip 0000000000492fea sp 00007f5b81e991d0 error 7 in PassengerHelperAgent[400000+203000]

etc. etc., then, no, your system is not suddenly crazy. Nor are you. It is highly likely that you RAM is defective. You should reboot your server and run the  RAM test from your boot manager (Grub always has such a test) to see if it can detect faulty RAM.

If you are operating a server that you can’t reboot because you can’t tolerate downtime, there is an excellent tool calledmemtester , which is a memory test for a running system. It is part of the Debian distribution, installit with apt-get install memtester  Check top to see how much free RAM there is available. Say you have 10GB RAM free, then ask memterst to test 8GB of it (so that 2GB are remaining free for the running system to operate). In my case, memtester indeed detected faults.

I ran

memtester 8000 3

It outputted stuff like this:

Loop 1/3:
Stuck Address : ok 
Random Value : ok
Compare XOR : ok
Compare SUB : ok
Compare MUL : ok
Compare DIV : ok
Compare OR : ok
Compare AND : ok
Sequential Increment: ok
Solid Bits : testing  30FAILURE: 0xffffffffffffffff != 0xfffffffbffffffff at offset
0x36e77910.
Block Sequential : ok 
Checkerboard : ok 
Bit Spread : ok 
Bit Flip : ok 
Walking Ones : ok 
Walking Zeroes : ok 
8-bit Writes : ok
16-bit Writes : ok

So, when I replaced the RAM, the Segfaults stopped. You can runmemtester  regularly to make sure the RAM is okay. Healty RAM is a very crucial part of your successful hosting operation!

In my case however, the segfaults corrupted MySQL tables, which I had to clean up. All’s well that ends well!