Tag: server

As a server administrator, I recently discovered a severe oversight of mine, one that was so big that I didn’t consciously see it for years.

What can Unprivileged Unix Users do on your server?

Any so-called “unprivileged Unix users” who have SSH access to a server (be it simply for the purpose of rsync’ing files) is not really “unprivileged” as the word suggests. Due to the world-readable permissions of many system directories, set by default in many Linux distributions, an “unprivileged” user can read a large percentage of directories and files existing on the server, many of which can and should be considered a secret. For example, on my Debian system, the default permissions are:

/etc: world-readable including most configuration files, amongst them passwd which contains plain-text names of other users
/boot: world-readable including all files
/home: world-readable including all subdirectories
/mnt: world-readable
/src: world-readable
/srv: world-readable
/var: world-readable
etc.

Many questions are asked about how to lock a particular user into their home directory. User “zwets” on askubuntu.com explained that this is besides the point and even “silly”:

A user … is trusted: he has had to authenticate and runs without elevated privileges. Therefore file permissions suffice to keep him from changing files he does not own, and from reading things he must not see. World-readability is the default though, for good reason: users actually need most of the the stuff that’s on the file system. To keep users out of a directory, explicitly make it inaccessible. […]

Users need access to commands and applications. These are in directories like /usr/bin, so unless you copy all commands they need from there to their home directories, users will need access to /bin and /usr/bin. But that’s only the start. Applications need libraries from /usr/lib and /lib, which in turn need access to system resources, which are in /dev, and to configuration files in /etc. This was just the read-only part. They’ll also want /tmp and often /var to write into. So, if you want to constrain a user within his home directory, you are going to have to copy a lot into it. In fact, pretty much an entire base file system — which you already have, located at /.

I agree with this assessment. Traditionally, on shared machines, users needed to have at least read access to many things. Thus, once you give someone even just “unprivileged” shell access, this user — not only including his technical knowledge but also the security of the setup of his own machine which might be subject to exploits — is still explicitly and ultimately trusted to see and handle confidentially all world-readable information.

The problem: Sometimes, being “unprivileged” is not enough. The expansion towards “untrusted” users.

As a server administrator you sometimes have to give shell access to some user or machine who is not ultimately trusted. This happens in the very common case where you transfer files via rsync  (backups, anyone?) to/from machines that do not belong to you (e.g. a regular backup service for clients), or even those machines which do belong to you but which are not physically guarded 24/7 against untrusted access. rsync  however requires shell access, period. And if that shell access is granted, rsync  can read out all world-readable information, and for that matter, even when you have put into place rbash  (“restricted bash”) or rssh  (“restricted ssh”) as a login shell. So now, in our scenario, we are facing a situation where someone ultimately not trusted can rsync all world-readable information from the server to anywhere he wants to simply by doing:

rsync user@host:/ .

One may suggest to simply harden the file permissions for those untrusted users, and I agree that this is a good practice in any case. But is it practical? Hardening the file permissions of dozens of configuration files in /etc  alone is not an easy task and is likely to break things. For one obvious example: Do I know, without investing a lot of research (including trial-and-error), which consequences chmod o-rwx /etc/passwd  will have? Which programs for which users will it break? Or worse, will I even be able to reboot the system?

And what if you have a lot of trusted users working on your server, all having created many personal files, all relying on the world-readable nature as a way to share those files, and all assuming that world-readable does not literally mean ‘World readable’? Grasping the full extent of the user collaboration and migrating towards group-readable instead of world-readable file permissions likely will be a lot of work, and again, may break things.

In my opinion, for existing server machines, this kind of work is too expensive to be justified by the benefits.

So, no matter from which angle you look at this problem, having ultimately non-trusted users on the system is a problem that can only be satisfactorily solved by jailing them into some kind of chroot directory, and allowing only those tasks that are absolutely neccessary for them (in our scenario, rsync  only). Notwithstanding that, and to repeat, users who are not jailed must be considered as ultimately trusted.

The solution: Low-level system utilities and a minimal jail

For above reasons regarding untrusted users, ‘hardening’ shell access via rbash  or even rssh  is just a cosmetic measure that still doesn’t prevent world-readable files to be literally readable by the World (you have to assume that untrusted users will share data liberally). rssh  has a built-in feature for chroot’ing, but it was originally written for RedHat and the documentation about it is vague, and it wouldn’t even accept a chroot environment created by debootstrap.

Luckily, there is a low-level solution, directly built into the Linux kernel and core packages. We will utilize the ability of PAM to ‘jailroot’ a SSH session on a per-user basis, and we will manually create a very minimal chroot jail for this purpose. We will jail two untrusted system users called “jailer” and “inmate” and re-use the same jail. Each user which will be able to rsync  files, but either will not be able to escape the jail, nor see the files of the other.

The following diagram shows the directory structure of the jail that we will create:

/
|-- bin
|-- home
|   |-- jailer
|   \-- inmate
|-- lib
|   \-- i386-linux-gnu
|       \-- cmov
|-- usr
    \-- bin

The following commands are based on Debian and have been tested in Debian Wheezy.

First, create user accounts for two ultimately untrusted users, called “jailer” and “inmate” (note that the users are members of the host operating system, not the jail):

adduser jailer
adduser inmate

Their home directories will be /home/jailer  and /home/inmate  respectively. They need home directories so that you can set up SSH keys (via ~/.ssh/authorized_keys ) for passwordless-login later.

Second, install the PAM module that allows chroot’ing an authenticated session:

apt-get install libpam-chroot

The installed configuration file is /etc/security/chroot.conf. Into this configuration file, add

jailer /home/minjail
inmate /home/minjail

These two lines mean that after completing the SSH authentication, the users jailer and inmate will be jailed into the directory /home/minjail of the host system. Note that both users will share the same jail.

Third, we have to enable the “chroot” PAM module for SSH sessions. For this, edit /etc/pam.d/sshd  and add to the end

session required pam_chroot.so

After saving the file, the changes are immediately valid for the next initiated SSH session — thankfully, there is no need to restart any service.

Making a minimal jail for chroot

All that is missing now is a minimal jail, to be made in /home/minjail . We will do this manually, but it would be easy to make a script that does it for you. In our jail, and for our described scenario, we only need to provide rsync  to the untrusted users. And just for experimentation, we will also add the ls  command. However, the policy is: Only add into the jail what is absolutely neccessary. The more binaries and libraries you make available, the higher the likelihood that bugs may be exploited to break out of the jail. We do the following as the root user:

cd /home
mkdir minjail
cd minjail

Next, create the home directories for both users:

mkdir -p ./home/jailer
mkdir -p ./home/inmate

Next, for each binary you want to make available, repeat the following steps (we do it here for rsync , but repeat the steps for ls ):

1. Find out where a particular program lives:

which rsync
$ /usr/bin/rsync

2. Copy this program into the same location in the jail:

mkdir -p ./usr/bin
cp /usr/bin/rsync ./usr/bin

3. Find out which libraries are used by that binary by running ldd /usr/bin/rsync:

linux-gate.so.1 => (0xb7798000)
libacl.so.1 => /lib/i386-linux-gnu/libacl.so.1 (0xb7774000)
libpopt.so.0 => /lib/i386-linux-gnu/libpopt.so.0 (0xb7767000)
libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xb7603000)
libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xb75fd000)
/lib/ld-linux.so.2 (0xb7799000)

4. Copy these libraries into the corresponding locations inside of the jail (linux-gate.so.1 is a virtual file in the kernel and doesn’t have to be copied):

mkdir -p ./lib/i386-linux-gnu/i686/cmov
cp /lib/i386-linux-gnu/libacl.so.1 ./lib/i386-linux-gnu
cp /lib/i386-linux-gnu/libpopt.so.0 ./lib/i386-linux-gnu
cp /lib/i386-linux-gnu/i686/cmov/libc.so.6 ./lib/i386-linux-gnu/cmov
cp /lib/i386-linux-gnu/libattr.so.1 ./lib/i386-linux-gnu
cp /lib/ld-linux.so.2 ./lib

After these 4 steps have been repeated for each program, finish the minimal jail with proper permissions and ownership:

chown root:root ./bin ./lib ./usr
chown jailer:jailer ./home/jailer
chown inmate:inmate ./home/inmate
chmod 751 ./home
chmod 750 ./home/jailer
chmod 750 ./home/inmate

The permission 751 of the ./home  directory (drwxr-x--x  root  root) will allow any user to enter the directory, but forbid do see which subdirectories it contains (information about other users is considered private). The permission 750 of the user directories (drwxr-x--- ) makes sure that only the corresponding user will be able to enter.

We are all done!

Test the jail from another machine

As stated above, our scenario is to allow untrusted users to rsync  files (e.g. as a backup solution). Let’s try it, in both directions!

Testing file upload via rsync

Both users “jailer” and “inmate” can rsync a file into their respective home directory inside the jail. See:

rsync -avz myfile.tar jailer@host:~

sending incremental file list
sent 75 bytes received 11 bytes 8.19 bytes/sec
total size is 24,070,782 speedup is 279,892.81

To allow password-less transfers, set up a public key in /home/jailer/.ssh/authorized_keys  of the host operating system.

Testing file download via rsync

This is the real test. We will attempt do download as much as possible with rsync (we will try to get the root directory recursively):

rsync -avz jailer@host:/ .

receiving incremental file list
rsync: opendir "/home" failed: Permission denied (13)
./
bin/
bin/bash
bin/ls
home/
lib/
lib/ld-linux.so.2
lib/i386-linux-gnu/
lib/i386-linux-gnu/libacl.so.1
lib/i386-linux-gnu/libattr.so.1
lib/i386-linux-gnu/libc.so.6
lib/i386-linux-gnu/libdl.so.2
lib/i386-linux-gnu/libpopt.so.0
lib/i386-linux-gnu/libpthread.so.0
lib/i386-linux-gnu/librt.so.1
lib/i386-linux-gnu/libselinux.so.1
lib/i386-linux-gnu/libtinfo.so.5
usr/
usr/bin/
usr/bin/rsync

sent 285 bytes received 1,639,733 bytes 142,610.26 bytes/sec
total size is 3,515,881 speedup is 2.14
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [generator=3.1.1]

Here you see that all world-readable files were transferred (the programs ls and rsync and their libraries), but nothing from the home directory inside of the jail.

However, rsync  succeeds to grab the user’s home directory. This is expected and desired behavior:

rsync -avz jailer@host:~ .

receiving incremental file list
jailer/
jailer/.bash_history
jailer/myfile.tar

sent 53 bytes received 23,773,208 bytes 1,901,860.88 bytes/sec
total size is 24,070,805 speedup is 1.01

Testing shell access

We have seen that we cannot do damage or reveal sensitive information with rsync . But as stated above, rsync  cannot be had without shell access. So now, we’ll log in to a bash shell and see which damage we can do:

ssh jailer@host "/bin/bash -i"

Put /bin/bash -i  as an argument to use the host system’s bash in interactive mode, otherwise you would have to set up special device nodes for the terminal inside of the jail, which makes it more vulnerable for exploits.

We are now dumped to a primitive shell:

bash-4.2$

At this point, you can explore the jail. Try to do some damage (Careful! Make sure you’re not in your live host system, prefer an experimental virtual machine instead!!!) or try to read other user’s files. However, you will likely not succeed, since everything you have available is Bash’s builtin commands plus rsync  and ls , all chroot’ed by a system call to the host’s kernel.

If any reader of this article should discover exploits of this method, please leave a comment.

Conclusion

I have argued that the term “unprivileged user” on a Unix-like operating system can be misunderstood, and that the term “untrusted user” should be introduced in certain use cases for clarity. I have presented an outline of an inexpensive method to accomodate untrusted users on a shared machine for various purposes with the help of the low-level Linux kernel system call chroot()  through a PAM module called pam_chroot.so  as well as a minimal, manually created jail. This method still is experimental and has not entirely been vetted by security specialists.