How to compile ezstream from source

Debian Stretch’s version of ezstream is currently a bit out of date. Here is how you compile ezstream from source to get the latest improvements and bugfixes. Not even the INSTALL file in the ezstream repo has all the steps:

Note that the configuration file structure has changed from what can be found on older blog posts on the internet. For example, to pipe OGG Vorbis data into ezstream without re-encoding, you can use something like teststream.xml:

Then, to stream 30 seconds of brown noise with a sine sweep to an Icecast server for testing purposes:

 

Zero Client: Boot kernel and root filesystem from network with a Raspberry Pi2 or Pi3

A Zero Client is a computer that has nothing on its permanent storage but a bootloader. Rather, it loads everything from the network.

With the method presented in this article, you will be able to boot a Raspberry Pi into a full Debian OS with nothing more on the SD card other than the Raspberry firmware files and the u-boot bootloader on a FAT file system. The Linux kernel and the actual OS will be served over the local ethernet network.

We will only focus on the Raspberry Pi 3, but the instructions should work with minor adaptations also on a Pi 2.

The following instructions assume that you have already built…

  1. a full root file system for the Raspberry
  2. a u-boot binary, and
  3. a Linux kernel

… based on my previous blog post. Thus, you should already have the following directory structure:

We will do all the work inside of the ~/workspace directory.

Preparation of the SD card

You will only need a small SD card with a FAT filesystem on it. The actual storage of files in the running OS will be transparently done over the network. Mount the filesystem on /mnt/sdcard and do the following:

Copy firmware

Copy u-boot bootloader

Create config.txt

config.txt is the configuration file read by the Raspberry firmware blobs. Most importantly, it tells the firmware what kernel to load. “Kernel” is a misleading term here, since we will boot u-boot rather than the kernel.

Create /mnt/sdcard/config.txt with the following contents:

 

Make an universal boot script for the u-boot bootloader

To achieve maximum flexibility — to avoid the repetitive dance of manually removing the SD card, copying files to it, and re-inserting it — we will make an universal u-boot startup script that does nothing else than loading yet another u-boot script from the network. This way, there is nothing specific about the to-be-loaded Kernel or OS on the SD card at all.

Create a file boot.scr.mkimage  with the following contents:

Replace the server IP with the actual static IP of your server. Note that this script does nothing else other than loading yet another script called netboot-${serial#}.scr  from the server. serial# is the serial number which u-boot extracts from the Raspberry Pi hardware. This is usually the ethernet network device HW address. This way, you can have separate startup scripts for several Raspberry Pi’s if you have more than one. To keep the setup simple, set the file name to something predictable.

Compile the script into an u-boot readable image:

Copy boot.scr to the SD card:

The SD card preparation is complete at this point. We will now focus on the serving of the files necessary for boot.

Preparation of the file server

Do all of the following as ‘root’ user on a regular PC running Debian 9 (“Stretch”). This PC will act as the “server”.  This server will serve the files necessary to network-boot the Raspberry.

The directory /srv/tftp will hold …

  • an u-boot start script file
  • the kernel uImage file
  • and the binary device tree file.

… to be served by a TFTP server.

The directory /srv/rootfs_rpi3 will hold our entire root file system to be served by a NFS server:

You will find installation instructions of both TFTP and NFS servers further down.

 

Serve the root file system

Let’s copy the pre-built root file system into the directory from where it will be served by the NFS server:

(notice the slash at the end of the source directory)

 

Fix the root file system for network booting

Edit  /srv/rootfs_rpi3/etc/fstab  and comment out all lines. We don’t need to mount anything from the SD card.

When network-booting the Linux kernel, the kernel will configure the network device for us (either with a static IP or DHCP). Any userspace programs attempting to re-configure the network device will cause problems, i.e. a loss of conncection to the NFS server. Thus, we need to prevent systemd-networkd from managing the Ethernet device. Make the device unmanaged by removing the folowing ethernet configuration file:

If you don’t do that, you’ll get the following kernel message during boot:

That is because systemd has shut down and then re-started the ethernet device. Apparently NFS transfers are sensitive to that.

In case you want to log into the chroot to make additional changes that can only be done from within (e.g. running systemctl scripts etc.), you can do:

 

Serve Kernel uImage

In this step, we create a Linux kernel uImage that can be directly read by the u-boot bootloader. We read Image.gz directly from the Kernel source directory, and output it into the /srv/tftp directory where a TFTP server will serve it to the Raspberry:

 

Serve device tree binary

The u-boot bootloader will also need to load the device tree binary and pass it to the Linux kernel, so copy that too into the /srv/tftp directory.

 

Serve secondary u-boot script loading the kernel

Create a file netboot-rpi3.scr.mkimage with the following contents:

Replace the server IP with the static IP of your server PC. Then compile this script into an u-boot readable image and output it directly to the /srv/tftp directory:

Make sure that the filename of the .scr file matches with whatever file name you’ve set in the universal .scr script that we’ve prepared further above.

 

Install a NFS server

The NFS server will serve the root file system to the Raspberry and provide transparent storage.

Edit /etc/exports and add:

To apply the changed ‘exports’ configuration, run

Useful to know about the NFS server:

You can restart the NFS server by running service nfs-kernel-server restart

Configuration files are /etc/default/nfs-kernel-server  and /etc/default/nfs-common

 

Test NFS server

If you want to be sure that the NFS server works correctly, do the following on another PC:

Mount the root file system (fix the static IP for your server):

 

 

Install a TFTP server

To install:

After installation, check if the TFTP server is running:

This command will tell you the default serving directory (/srv/tftp):

Here is another command that tells you if the TFTP server is listening:

To get help about this server: man tftpd

Test TFTP

If you want to be sure that the TFTP server works correctly, do the following on another PC:

Then see if the server serves the Linux kernel we’ve installed before:

You now should have a local copy of the linux-rpi3.uImage file.

 

Complete

If you’ve done all of the above correctly, you can insert the prepared SD card into your Raspberry Pi and reboot it. The following will happen:

  1. The Raspberry Pi GPU will load the firmware blobs from the SD card.
  2. The firmware blobs will boot the image specified in config.txt. In our case, this is the u-boot binary on the SD card.
  3. The u-boot bootloader will boot.
  4. The u-boot bootloader loads and runs the universal boot.scr script from the SD card.
  5. The boot.scr downloads the specified secondary boot script from the network and runs it.
  6. The secondary boot script …
    • downloads the device tree binary from the network and loads it into memory.
    • downloads the Linux kernel from the network and loads it into memory
    • passes the device tree binary to the kernel, and boots the kernel
  7. the Linux kernel will bring up the ethernet device, connect to the NFS server, and load the regular OS from there.

Many things can go wrong in this rather long sequence, so if you run into trouble, check the Raspberry boot messages output on an attached screen or serial console, and the log files of the NFS and TFTP servers on your server PC.

 

Resources

https://www.raspberrypi.org/documentation/linux/kernel/building.md

http://www.whaleblubber.ca/boot-raspberry-pi-nfs/

https://cellux.github.io/articles/moving-to-nfs-root/

http://billauer.co.il/blog/2011/01/diskless-boot-nfs-cobbler/

https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt

http://wiki.linux-nfs.org/wiki/index.php/General_troubleshooting_recommendations

https://wiki.archlinux.org/index.php/NFS

How to turn the Raspberry Pi into a Gateway to mobile phone internet

Your DSL internet connection is too slow? Want to set up an improvised office? You do not want to pay for a DSL internet plan when you already have a fast 4G mobile plan? If yes to one of the above, it is quite easy to configure a Raspberry Pi to share one mobile internet connection to an Ethernet network.

Strictly speaking, you don’t have to use a Raspberry Pi to do this. A laptop or desktop computer with any Operating System would work too, but the Raspberry is so small and consumes only 2-3 W of electrical power, and is so cool (quite literally!), so will will make use of this awesomeness!

 

Prerequisites

The following step-by-step guide is based on a pure Debian 9 (“Stretch”) distribution with a mainline/vanilla/unpatched Linux kernel built according to my previous blog post:

https://blog.michael.franzl.name/2016/10/31/raspberry-pi-debian-stretch/

  • We will not focus on the Raspbian OS nor on any other distribution, because documentation for these other setups exists in abundance.
  • You should not have a graphical interface installed. GUIs also install the NetworkManager service for systemd (Debian package “network-manager”), and I have not tested how NetworkManager interacts with the methods presented below. In addition, a bare-bone system is the preferred choice because it saves RAM and CPU resources.
  • In any case, you should attach a keyboard and screen to the Raspberry because you may temporarily lose network connectivity during the setup.
  • You also need a smart phone with an internet plan, supporting USB tethering. I have only tested recent Android based smartphones. Keep in mind during the following steps that, with most smart phones, you need to re-enable USB tethering after reboots or USB cable reconnects.

 

Goals

  • Computers in the LAN will be able to set the Raspberry Pi’s static IP address as internet Gateway and DNS server.
  • The Raspberry Pi will prefer a smart phone connection (tethered USB) to forward traffic.
  • If the smart phone is disconnected, the Rasbperry Pi will automatically fall back to an already existing gateway if present (i.e. a DSL modem)

 

Step 1: Install a DNS server

This ensures that cached DNS lookups are very fast when a DNS query has already been fetched.

Tell “bind” to use Google’s public DNS servers (they are good). Edit /etc/bind/named.conf.options and change the “forward” block to:

Restart “bind”:

 

Step 2: Configure a static IP address for the Ethernet adapter

If you already have a DHCP server running in your local network (we will use the subnet 192.168.0.0 in this guide), give the Raspberry Pi a free static IP address in this existing subnet, e.g. 192.168.0.250.

If you don’t have an existing DHCP server running in your local network, we will set one up on the Raspberry (see Step 8 below).

In both cases, we will give our Rasberry the static IP address 192.168.0.250. Using systemd, change the config file of your ethernet connection /etc/systemd/network/eth.network:

 

If your LAN already has an internet gateway, e.g. a DSL modem with address 192.168.0.1, add the following (optional) section to the same config file:

The large positive integer value of “Metric” ensures that other configured gateways with a lower Metric will be preferred. This will come in handy in the next step where the smart phone will be our preferred gateway with a Metric value of 1024.

Now reboot the Raspberry or run systemctl restart systemd-networkd.  You may lose network connectivity at this point if you are logged in via ssh.

Now, check that networkctl status eth0 matches our wanted static IP address:

Next, check the output of route -n (the kernel routing table). It should show:

If you have added the optional  [Route] section, you should also see the following as first line, which is our current default route to the internet:

 

 

Step 3: Set the smart phone connection as gateway

Plug in your phone’s USB cable into one of the Raspberry’s USB connectors. Then turn on USB tethering in the Settings UI of your smart phone.

Run networkctl. You should see the following entry amongst the other network connections (notice “off” and “unmanaged”).

 

To have the “systemd-networkd” service manage the “usb0” network device, create a file /etc/systemd/network/mobile.network with the following contents:

To apply this config file, run systemctl restart systemd-networkd .  After a few seconds,  networkctl should output (notice the “routable” and “configured” parts):

You also can check networkctl status usb0  to see the dynamic IP address obtained from the DHCP server on the smart phone. For Android phones this is usually in the subnet 42.

Next, check the output of route -n. Now, the phone connection “usb0” should be on the top of the list thanks to the lower metric of 1024:

 

Step 4: Check internet connectivity

With this routing table, we already can connect to the internet via the smart phone. To make sure that we are routed via the smart phone, we will ask the Linux kernel which gateway it would take first for traffic. ip route get 8.8.8.8  should ouput the IP address of the smart phone (192.168.42.129, subnet 42):

Let’s ping Google’s server a few times: ping 8.8.8.8  to see if we have an actual working route to the internet:

The answer: Yes!

Check phone’s DNS server

 

Now let’s check if the phone’s DNS server is working. Type  dig google.com (install Debian package “dnsutils” if not yet installed), and make sure that you’ve got an “ANSWER SECTION”:

Note that the response came from the phone’s IP. So, “systemd” has correctly configured the phone’s IP address as DNS server for the Raspberry (that information came from the phone’s DHCP server).

Run  dig google.com again. This time the result should be cached and returned much faster (just 1ms):

Check local DNS server

Type  dig @localhost google.com:

Note that this time, the response came from the “bind” DNS server which we have installed in Step 1. It, in turn, forwards queries via the phone connection. This server will be used for all requests via Ethernet.

Step 5: Turn on IP protocol forwarding for the Linux kernel

By default, this feature is turned off. Check the current status of this feature:

sysctl -a | grep net\.ipv4\.ip_forward  will output:

To permanently set this variable to 1, create /etc/sysctl.d/30-ipforward.conf and add the following:

Reload all settings by typing  sysctl --system. Now, and also after a reboot, the “ip_forward” variable should stay enabled.

 

Step 6: Turn on Network address translation (NAT) aka. “Masquerading” between Ethernet and USB Smart Phone network links

Create a shell script  /usr/bin/startgateway.sh with the following contents and make it executable ( chmod a+x):

This will masquerade IP packets coming in through the Ethernet adapter as if they were coming from the Raspberry itself, forward them to the USB smart phone connection, and the incoming answers (from remote servers) will be re-written and forwarded back to whereever in the LAN they came from. That is the central purpose of the problem we’re trying to solve in this tutorial.

Run this script. Check the output of iptables -L -n -v:

 

To run this shell script at system boot, right after the network links have been brought up, create the following systemd service file:

Add the following:

 

 

Step 7: Test the Raspberry Gateway!

On another machine in your LAN (can be Linux, Windows or Mac), configure the Ethernet connection manually. Set the following:

  • Static IP Address: 192.168.0.10 (or any other freely available address on this subnet)
  • Gateway: 192.168.0.250
  • DNS: 192.168.0.250

Then run traceroute 8.8.8.8  on that other machine. Truncated output:

The route is correctly resolved. First traffic goes to the Raspberry Pi, then to the smart phone, and from there to the internet.

If you can’t run traceroute on that other machine, using a regular browser to browse the internet should work at this point!

 

Step 8: Running a DHCP server on the Raspberry

TODO

 

Conclusion

This tutorial may seem long, but the commands are few, and with a bit of practice you can turn your Raspberry Pi into a mobile phone Gateway in 10 minutes to enjoy faster 4G internet when your other modems are too slow.

Hardening WordPress against hacking attempts

The WordPress Codex states:

Security in WordPress is taken very seriously

This may be the case, but in reality, you yourself have to take some additional measures so that you won’t have a false sense of security.

With the default settings of WordPress and PHP, the minute you host Wordpress and give access to one non-security-conscientious administrative user, your entire hosting environment should be considered as compromised.

The general problem with WordPress and PHP is that rather than thinking about which few essential features to turn on (whitelisting), you have to think about dozens of insecure features to turn off (blacklisting).

This excellent article (“Common WordPress Malware Infections”) gives you an overview what you’re up against when it comes to protecting WordPress from Malware.

Below are a couple of suggestions that should be undertaken, starting with the most important ones.

Disable WordPress File Editing

WordPress comes with the PHP file editor enabled by default. One of the most important rules of server security is that you never, ever, allow users to execute arbitrary program code. This is just inviting desaster. All it takes is the admin password to be stolen/sniffed/guessed to allow the WordPress PHP code to be injected with PHP malware. Then, if you haven’t taken other restricting measures in PHP.ini (see section below), PHP may now

  • Read all readable files on your entire server
    • Include /etc/passwd and expose the names of all user accounts publicly
    • Read database passwords from wp-config.php of all other WordPress installations and modify or even delete database records
    • Read source code of other web applications
    • etc.
  • Modify writable files
    • Inject more malware
    • etc.
  • Use PHP’s curl functions to make remote requests
    • Turns your server into part of a botnet

So amongst the first things to do when hosting WordPress, is to disable file editing capabilities:

But that measure assumes that WordPress plus third-party Plugins are secure enough to improve their own security, which one cannot assume, so it is better to…

The “Big Stick”: Remove Write File Permissions

I’ll posit here something that I believe to be self-evident:

It is safer to make WordPress files read-only and thus disallow frequent WordPress (and third-party Plugin) upgrades than it is to allow Wordpress (and third-party Plugins) to self-modify.

Until I learn that this postulate is incorrect, I’ll propose that you make all WordPress files (with the obvious exception of the uploads directory) owned by root, and writable only to root, interpreted by a non-root user. This will leverage the security inherent in the Linux Kernel:

Note that you still can upgrade WordPress from time to time manually. You could even write a shell script for it.

Restrict serving of files

Disable direct access to wp-config.php which contains very sensitive information which would be revealed should the PHP not be processed correctly. In Nginx:

Disable PHP execution in the uploads directory. In Nginx:

Restrict PHP

I’ll refer the reader to already written excellent external articles –  please do implement the suggestions therein:

Hardening PHP from PHP.ini

25 PHP Security Best Practices

Host WordPress in a Virualization Environment

In addition to all of the above, any kind of publicly exposed web application (not just WordPress) should really be hosted in an isolated environment. Docker seems promising for this purpose. I found the following great external tutorial about generating a LAMP Docker image:

WordPress Developer’s Intro To Docker

WordPress Developer’s Intro To Docker, Part Two

 

no.php – Transparent reverse proxy written in PHP that allows you to not have to write PHP any more

This little project will probably be my only contribution to the world of PHP.

The code is at https://github.com/michaelfranzl/no.php

This short, single-file, 80-line PHP script is a simple and fully transparent HTTP(S) reverse proxy written in PHP that allows you to never have to use PHP again for a new project, if you feel so inclined, for example if you are forced to host on a fully 3rd-party-managed server where you can’t do more than run PHP and upload files via FTP. The PHP script simply reads all requests from a browser pointed to it, forwards them (via PHP’s curl library) to a web application listening at another URL (e.g. on a more powerful, more secure, more private, or more capable server in a different data center), and returns the responses transparently and unmodified.

Supports:

  • Regular and XMLHttpRequests (AJAX)
  • All HTTP headers without discrimination
  • GET and POST verbs
  • Content types (HTTP payload) without discrimination
  • Redirects (internal redirects are rewritten to relative URIs)

Does not support (or not tested):

  • HTTP verbs other than GET and POST (but these are usually emulated anyway)
  • HTTP greater than version 1.1 (e.g. reusable connections)
  • Upgrade to websocket (persistent connections)
  • Multipart content type

Usage illustrated by the standard example

You have a non-PHP web application (called the “backend”) listening on https://myapp.backend.com:3000 but due to constraints you must make it available on a shared hosting server called https://example.com/subdir which only supports PHP and can’t be configured at all. On latter server, Apache (or Nginx, doesn’t matter) will usually do the following:

  1. If a URI points to a .php file, this file will be interpreted
  2. If a URI points to a file that is not existing, a 404 status will be returned.

Using no.php, to accomodate the second case, all URIs of the proxied web app (including static files) must be appended to the URI https://example.com/subdir/no.php. For example:

If your backend app supports that extra /subdir/no.php prefix to all paths, you are all set and ready to use no.php. Then:

  1. Simply copy no.php into the subdir directory of example.com
  2. Change $backend_url in no.php to "https://myapp.backend.com:3000"
  3. Point a browser to https://example.com/subdir/no.php

Project status

Experimental. Use only if you know what you are doing.

Apparent diameter plot for Jupiter 2017-2026

The apparent diameter of Jupiter varies with the Earth-Jupiter distance. Depending on the distance, the apparent angular diameter of Jupiter is between approx. 0.5 and 0.8 arcminutes. For best astrophotography results, one should pick a time when Jupiter is closest. Similar to my plot of the apparent size of Venus, I wrote a short program to plot Jupiter’s apparent diameter for the time period 2017-2026 (see featured image).

Apparent diameter plot for the Moon 2017-2018

The apparent diameter of the Moon varies with the Earth-Moon distance. Depending on the distance, the apparent angular diameter of the Moon is between 29.4 and 33.5 arcminutes. For best astrophotography results, one should pick a time when the moon is closest. Similar to my plot of the apparent size of Venus, I wrote a short program to plot the Moon’s apparent diameter (see featured image).

Venus apparent diameter plot for 2017-2020

The apparent size of Venus varies dramatically with the Earth-Venus distance, depending on both orbits around the sun. The apparent angular diameter of Venus is between 0.175 and 1 arcminutes.

Wikipedia has nice images on that and says that…

“the extreme crescent phase of Venus can be seen without a telescope by those with exceptionally acute eyesight, at the limit of human perception.”

I will definitely try next time with my naked eyes, but if you have an entry-level telescope, you will be certainly be better able to see the crescent phase during closest approach.

The question is, when does Venus come closest to Earth for best observation? Not finding anything on the interwebs, I wrote a quick plotting program with Python’s ephem package and gnuplot.

Venus apparent diameter plot for 2017-2020

Graph of apparent diameter of Venus in arcminutes for the time period 2017-2020
Graph of apparent diameter of Venus in arcminutes for the time period 2017-2020

Here is the very short Python program that calculates the apparent diameter of Venus for 4 years beginning with January 2017:

Reading Raspberry Pi chip temperature with mainline Linux kernel

This tutorial is based on a previous article where we installed pure Debian 9 with a recent mainline/vanilla Linux kernel on a Raspberry Pi, and so differs from what would be done on a Raspbian Distribution with a Raspbian kernel. In this article, we will read the Raspberry Pi chip temperature. Here is my previous article:

https://blog.michael.franzl.name/2016/10/31/raspberry-pi-debian-stretch/

As of mainline Linux Kernel 4.9.0-rc3, the sysfs entry /sys/class/thermal for the Broadcom BCM283x chip found on Raspberry Pi’s is empty. You can apply the following patch to Linux kernel 4.9.0-rc3, even though it will soon be superfluous because it seems that currently there is ongoing work by Linux Kernel developers to add in the missing functionalities.

The patch which will get you the standard sysfs temperature node which you can read like this:

This is the chip temperature in thousandths of degrees cenigrade, i.e. 40.084 degrees Celsius.

The following patch will give you a new entry in the mainline Kernel config, under Drivers -> SoC -> BCM -> Raspberry Pi thermal sysfs driver, which you have to enable, then recompile your kernel.

I have actually submitted this patch to a Linux kernel developer, but the process to get code into the Linux kernel is quite elaborate, and he said that they are already working on it, so I let it drop and decided to write this blog post instead.

 

Setting I2C bus speed on a Raspberry Pi via Device Tree

This tutorial is based on my previous article where we installed pure Debian 9 with a recent mainline/vanilla Linux kernel, and so differs from what would be done on a Raspbian Distribution with a Raspbian kernel. In this article, we will set the I2C bus speed on a Raspberry Pi. Here is my previous article:

https://blog.michael.franzl.name/2016/10/31/raspberry-pi-debian-stretch/

Device Trees

The I2C bus on the Broadcom BCM283x chips found on Raspberry Pi’s is well and directly supported by the mainline/vanilla Linux kernel. Since with the Raspberry Pi we’re dealing with a System on a Chip (SoC), and not a regular PC, the hardware is configured with so-called device trees, which is a low-level description of the chip hardware compiled from text into binary format.

The rpi23-gen-image script mentioned in my previous tutorial installs the binary device tree into /boot/firmware/bcm2836-rpi-2-b.dtb. The U-Boot bootloader can read this file and pass it to the Linux kernel which interprets it and enables all the mentioned features in it.

The clock frequency for the I2C bus is configured in this .dtb file, and the default is 100kHz. There is a tool which allows you to inspect the .dtb file, outputting regular text. With this tool you also can make changes to the device configuration. Nowadays, this is the proper way to configure low-level devices on SoC’s!

 

Read the device tree

This will output the decoded device tree as text. Regarding I2C, you will find i2c@-entries like this:

Change the device tree

The clock-frequency value is what we want to change. The value is a raw binary unsigned 32-bit int stored big-endian, unreadable for humans.

But you can use another tool fdtget to read just this value decoded:

This will output 100000.

In my case, I wanted to set the I2C bus to the slowest frequency, to compensate for long cable lengths. I found that one of the lowest supported I2C clock frequencies is 4kHz. With fdtput you can set the clock-frequency property for each i2c device (there are 3 on the RPi):

And that’s it. You have to reboot for these settings to take effect.

I used an oscilloscope to verify that the SCL pin of the GPIO of the RPi indeed toggled with 4kHz, and it did!