Cover V13, i05

Article
Sidebar

may2004.tar

Secure File Transfer with chroot'ed SFTP-Only Accounts

Ralph Durkee

Popular open source ftp servers, such as ProFTPD (proftpd.linux.co.uk) and WU-FTPD (wuftpd.org), have built-in security features so that accounts can be set up for transferring files with restricted ftp-only access to a specific assigned (chroot'ed) directory. Yet, the ftp protocol doesn't provide encryption of the data transfer, doesn't protect the user name and password from disclosure over the Net, and is subject to a variety of attacks, such as IP and DNS spoofing. The SSH protocol provides encryption of the authentication and file transfers while mitigating the risk of IP- and DNS-level spoofing attacks by authenticating the server's host key. So, how can we get the combined security of the chroot'ed ftp-only environment along with the security of the ssh and sftp protocols? In this article, I'll show how it is possible to get similar chroot'ed restricted account functionality from OpenSSH with open source software.

Security Includes the Big Picture

I'll step through the process of setting up a chroot'ed sftp-only server using Red Hat 9 and Solaris 9 as examples. Of course, setting up the chroot'ed environment for the sftp server is just one part of the overall security for the server and for your network. The server must be secured before installing the sftp-only chroot functionality and should be re-scanned for vulnerabilities and tested afterward for proper security controls.

Online resources for securing Linux or Solaris servers -- such as the scoring tools provided by the Center for Internet Security, SANS Institute, and the Bastille Linux project -- are especially applicable here. Some additional recommendations for configuration of the SSH server include: the SANS OpenSSH Step by Step titled OpenSSH -- A Survival Guide for Secure Shell Handling (to which I was a contributing author) and, of course, SSH, The Secure Shell: The Definitive Guide from O'Reilly & Associates.

In particular, the following server-side configurations are recommended in the sshd_config file, typically found in /etc/ssh/ or /usr/local/etc/:

Protocol 2
The protocols allowed should be only 2, rather than "Protocol 2,1" which prefers protocol 2, but allows protocol 1. The older protocol 1 has some known weaknesses and is not recommended. (See http://www.openssh.com/security.html for details.) Also, the ssh clients must be configured to allow only protocol 2, because allowing multiple protocols may make it easier for man-in-the-middle attacks to succeed. If the MITM attack forces protocol 1, while previous connections were made only via protocol 2, then instead of the user receiving an appropriate alarm message that the host keys do not match, the user will receive a message indicating a connection to a new host with a new key.

The scponly Software

To start, you'll need the OpenSSH software installed -- it now comes bundled with most current Unix operating systems. There were several OpenSSH vulnerabilities in 2003, so be sure you have all the appropriate patches. The OpenSSH Security vulnerabilities are very briefly documented at:

http://www.openssh.com/security.html
Visit the OpenSSH Web site and your OS vendor for patches and latest software downloads. I recommend starting with an up-to-date, fairly vanilla version of OpenSSH, once you have the chroot functionality working, it's easy to add customization and hardening.

We'll use a restricted shell, which is open source software called "scponly". It is written and maintained by Joe Boyle and others at:

http://www.sublimation.org/scponly
The scponly shell has been around for more than a year now, and Joe Boyle, with the help of few others, has done an admirable job of updating the software. I would expect scponly to gain popularity and for the functionality eventually to be incorporated into or bundled with the OpenSSH server. The commercial SSH product from www.ssh.com includes a chroot manager for sftp users.

Next, download the software and verify the md5 hash, since a PGP signature is not yet provided. The 3.9 release is the current release used in these examples. These examples were tested on a Linux Red Hat 9 system and a Solaris 9 Sparc system. Examples should work on either except where noted (the md5sum command shown above is just "md5" on Solaris):

$ wget http://www.sublimation.org/scponly/scponly-3.9.tgz
$ md5sum scponly-3.9.tgz
e18410e7d49f171e711954da836107ea  scponly-3.9tgz
How It Works

The scponly software provides a restricted shell, which replaces the normal user shell (such as /bin/bash or /bin/csh) in the /etc/passwd. This shell doesn't allow arbitrary shell commands to be run. Instead, the scponlyc shell allows only the sftp-server to be run when configured and built as below. The sftp-server then allows the specific commands of the sftp protocol to be executed. The sftp protocol is technically independent of the SSH protocol. The SSH protocol ensures proper authentication and then sets up the encrypted channel, which the sftp then uses for file transfers. This ensures a high degree of independence between SSH protocols and subsystem protocols such as sftp.

The default sftp-server includes a pretty wide range of commands (including cd, lcd, chgrp, chmod, chown, help, get, lls, ln, lmkdir, lpwd, ls, lumask, mkdir, progress, put, pwd, exit, quit, rename, rmdir, rm, symlink, version). Additional changes can be made to the sftp-server source code to limit the commands allowed and to increase security. In particular, chown and chgrp are normally not necessary or desirable to allow restricted sftp-only accounts to change group or owner for files to another group or user.

Also, the scponly shell and the sftp-server generally have a default umask of 000 (on the platforms tested). Since they do not execute normal login initialization scripts (e.g., /etc/profile or /etc/login), there remains no means to easily change the umask value for sftp-only users. Fortunately, there is a patch available by Michael Martinez at:

http://sftplogging.sourceforge.net/
which addresses this issue and provides additional logging. Unfortunately, time and space don't allow coverage of the patch in this article.

Building and Installation

To build the scponly software, do the usual tar extraction and change to the directory as shown below. Reading the README and INSTALL files is highly recommended. For this usage of scponly, we are running the configure script with the --enable-chrooted-binary option to provide the chroot'ed environment, the --disable-scp-compat option disables usage of several shell commands and allows only the sftp protocol to be used.

In this mode, the only shell command that the client may place is to request the sftp-server, which is typically /usr/libexec/openssh/sftp-server. The final options, --disable-winscp-compat and --disable-wildcards, should in theory be redundant and wouldn't apply to the sftp protocol. However, they do change how the code is compiled and may be desired for the paranoid and the careful. This configuration is compatible with recent WinSCP releases (tested with 3.3.0 & 3.4.2) as long as the sftp protocol is chosen in the WinSCP client GUI.

In the steps below, the configure script is run as a root user, which is generally not recommended. However, the configure needs to find administrative programs like useradd, which may not be located in your path or may not be executable if you are not a root (user id=0) user. The useradd program is required for the setup_chroot.sh script, which attempts to find the necessary files for the chroot environment:

$ tar xzf scponly-3.9.tgz
$ cd scponly-3.9
$ su  # or sudo 

# ./configure  --enable-chrooted-binary --disable-scp-compat \
  --disable-winscp-compat --disable-wildcards
# make
# make install
Next, add the path to scponlyc /usr/local/sbin/scponlyc to the /etc/shells file, so that it is recognized as a valid shell.

Setting Up the Chroot Environment

Setting up the virtual environment for a chroot is often platform specific and can be a bit difficult to debug. The scponly tar file contains a script to help with the setup. The script may be run as "make jail", which performs a "make install" and then makes the script ./setup_chroot.sh executable and runs the script. It's okay to use the ./setup_chroot.sh script as a starting point, but you must be aware of exactly what is placed in the chroot as well as the permissions placed on the directories and files.

The chrooted environment should be tailored to provide the minimal needs of the specific system and security policies. It's easy to make a mistake here that could compromise the system security. It's important therefore, that the final configuration be tested thoroughly to ensure the security controls are effective. The new 3.9 release of setup_chroot.sh script is much improved over the 3.8 version and now has a means for using pre- and post-processing scripts specific to each platform. There is a pre-processing script for Linux, which was contributed as an outcome of writing this article and should be in build_extras/arch/ Linux.pre.sh. If the configure script detected "Linux" as the platform, then this script would be used to ensure that the /etc/ld.so.cache /etc/ld.so.conf files are included in the chroot directory.

The setup_chroot.sh script should run without errors on Red Hat 9, and probably most recent Linux systems. Some minor changes were needed to run on Solaris 9, which are documented below. When run, the script issues an important warning that the user's home directory must not be owned by the user and must not be writeable. This is done to prevent the users from changing the environment via ~/.ssh/environment, although this feature is now also disabled by default via the sshd configuration of "PermitUserEnvironment no". You may also want to consider the impact of having users place entries in files such as the ~/.ssh/authorized_keys file. This would allow the adding of public keys that could be accepted rather than using a password.

The use of public keys may be disabled if desired via sshd configuration "PubkeyAuthentication no", or better yet, the path to the public key file may be changed out of the users home directory via "AuthorizedKeysFile /etc/ssh/authorized_keys/%u". You don't want the sftp user to be able to move or replace any of the directories that contain libraries and other files required for the virtual root. Allowing write access to only a specific directory intended for incoming files is the best practice. Having a read-only directory for outbound files is also common. The rest of the directory structure should have minimal access:

# chmod u+x ./setup_chroot.sh
# ./setup_chroot.sh
The setup_chroot.sh will use the default system user skeleton directory and will likely include some unnecessary files, such as .bash_profile or .profile or .login, etc. Remove all extra files that were copied from the default skeleton directory (typically from /etc/skel).

The directory to be the virtual root must contain all of the files and shared libraries that the sftp-server requires. If a library or file is missing, the exec of the sftp-server will fail, and in many cases there will be no error message indicating the trouble. Some tips for debugging are included later.

Check for the following requirements:

1. The sftponly user id must have read access to parent directories in its home path. For example, for /home/sftpusers/testsftp, all three directories (including /home and /home/sftpusers) must be readable (r) and searchable (x) for the testsftp user.

2. The Linux pre-script should have caused the dynamic loader configuration and cache to be copied. Check for the files /etc/ld.so.conf and /etc/ld.so.cache in the testsftp directory.

3. Remove any .profile or .login or other files created by the default skeleton. (See the sidebar for specific changes to setup_chroot.sh required for Solaris 9.)

Adding SFTP-Only Users

Once the chroot'ed functionality works for a single test user, how do we add additional users? I recommend setting up the chroot'ed directory tree as a user skeleton directory to be used with the -k option of useradd. However, the useradd command will not set up the restricted permissions that we need, so let's first consider the correct permissions needed for the user directory tree. The best answer is, of course, the minimal required permissions, which depends on the specific usage. Check that the scponly chroot functionality is working correctly before making additional changes to the permissions, so you'll know whether a problem was caused by the permissions hardening or something else.

Below is a partial script with line numbers for setting the permissions. Discussion and comments follow the script, where $targetdir is the sftp-users virtual root, and $targetdir/$incoming/ is the one writeable incoming directory:

1: chown -R 0.0 $targetdir
2: chmod -R u=rwX,g=rX,o=  $targetdir
3: chgrp -R $newgid   $targetdir
4: chmod -R g+w  $targetdir/$incoming/
Line 1 sets everything to be owned by root (i.e., user 0) and group = 0. The -R option applies the ownership recursively to all subdirectories and files. You may instead want to set the ownership to an operator/sftp admin account, which has the administrative duties of placing or replacing outbound files and cleaning up old incoming files. In that case, replace the zeros with the appropriate user and group.

The point is that the sftp-only user does not need ownership of any directories and should never own any directory or file outside of the "incoming" directory. Also note that $newgid should be a group specific to the user. Some Unix systems have useradd(1M) set up to do this as the default while others (e.g., Solaris and some Linux systems) do not. Check your useradd(1M) man page for details.

Line 2 recursively sets the default permission for the entire hierarchy. The capital X is a conditional form of the small x, and it sets the search bit if it is a directory, or if it is a file and already has execute permission. The capital X with the = argument is not always supported on all Unix systems, although it works fine in Solaris 9 and Red Hat 9. In any case, you could do the same thing with a few find commands. The first find below sets the permissions on all directories, the second on all executable regular files, and the third on all regular files that are not executable. If you have any other non-files, such as devices, links, or sockets, include an additional line to set the desired permissions on them:

find $targetdir -type d  -print | xargs chmod u=rwx,g=rx,o=
find $targetdir -type f -perm +111 -print | chmod u=rwx,g=rx,o=
find $targetdir -type f  \! -perm +111 -print | chmod u=rw,g=r,o=
find $targetdir \! \( -type d -o -type f  \) | chmod u=rw,g=r,o=
Line 3 sets everything to have the group id of the sftpuser. This allows read access for the user without giving ownership or giving read access to the world.

Line 4 sets the files and directories in the incoming directory to have write access for the user.

Note that the sftp user is allowed to create new sub-directories within the incoming directory since it is writeable, and the sftp-user will have ownership of these new directories. If a non-root account is used to administer the sftpusers directories, then some means of allowing the sftp-admin to clean out these directories is desirable. One simple approach would be to reset the ownership and permissions of the sftp-user directory with the above script at an appropriate time as part of the directory maintenance. You should also consider what happens to the system when a user accidentally transfers too many files or a too large file. The sftpusers directory should be mounted on a non-critical file system, but usage of disk quotas must also be considered.

Once the skeleton user directory is tested, any disk quotes are in place, and the script is tested for setting the permissions, a user may be added with a script that uses the useradd command. Something like:

# useradd -d /home/sftpusers/rdurkee -c "Ralf Durkee"  -k \
  /home/sftpusers/skeleton -s /usr/local/sbin/scponlyc rdurkee
There is also a very useful but not well-documented feature in scponly to have the home directory be a subdirectory within the chroot'ed directory. This is done by specifying the home directory in the passwd file with two slashes where the chroot directory path ends. So, by changing the testsftp user's home directory to /home/sftpusers/testsftp//incoming, the sftp user will start off in the incoming directory rather than in the virtual root directory. This is more convenient and is much less confusing for novice users. However, now we have violated the big warning about the sftp user not having write access to the home directory, because the incoming directory is writeable.

The writeable home directory is a significant concern if the user is able to have the server use the modified environment. However, if usage of the user environment is disabled via the sshd_config directive "PermitUserEnvironment no", and with appropriate testing to ensure the environment can't be modified, you may prefer the convenience of the writeable home directory. For the paranoid and the careful, I still recommend not having a writeable home directory as a defense-in-depth measure.

So far, this approach requires a virtual root for each user. The virtual root on the Solaris 9 was 1.1 MB and on the tested Red Hat 9 system including dynamic Kerberos libraries was 3.6 MB, which may add up for systems with a lot of sftp-only users. Compiling out unneeded libraries, such as the Kerberos support from the sftp-server, would trim about half a megabyte off that space.

There are, as usual, a large number of solutions for reducing the space requirements. One simple approach is to use the double slash notation to have a group of related users share a virtual chroot. Something like:

/home/sftpusers//testsftp/
could have all user accounts in the sftpusers directory use the same virtual root. This implies that all users must be able to read the /home/sftpusers directory, and could therefore see the directory names of other accounts, but permissions could limit users from reading the names or contents of files in other user's directories. Also note that it is not necessary to have user directory names correspond to valid user names just because it's traditional.

Testing and Debugging the Chroot'ed SFTP Environment

1. Check the system logs for errors, typically /var/log/messages and/or /var/log/secure.

2. Make sure sftp is working for users who are not chroot'ed. You can easily change the shell of the test account from /usr/local/sbin/scponlyc to a normal shell such as /bin/bash.

3. If the shell works, but scponlyc does not, check whether the scponly (without the c for chroot) works. It will need to be included in the /etc/shells file, of course. This test ensures that the problem is with the chroot rather than the scponly software installed.

4. There's also a debug feature of scponly that may be helpful in some cases. It is enabled by placing a 1 or 2 in the configured file. The default is /usr/local/etc/scponly/debuglevel. Something like:

echo 2 > /usr/local/etc/scponly/debuglevel
will enable additional syslog output when the scponlyc is executed. You may see something like:

Dec  6 10:13:58 net3 sshd[11547]: Accepted password for \
  testsftp from 10.0.0.12 port
1463 ssh2
Dec  6 10:13:58 net3 sshd[11549]: subsystem request for sftp
Dec  6 10:13:58 net3 [11550]: chrooted binary in place, will chroot()
Dec  6 10:13:58 net3 [11550]: 3 arguments in total.
Dec  6 10:13:58 net3 [11550]: ^Iarg 0 is scponlyc
Dec  6 10:13:58 net3 [11550]: ^Iarg 1 is -c
Dec  6 10:13:58 net3 [11550]: ^Iarg 2 is /usr/libexec/sftp-server
Dec  6 10:13:58 net3 [11550]: opened log at LOG_AUTHPRIV, opts 0x00000029
Dec  6 10:13:58 net3 [11550]: retrieved home directory of "/home/sftpusers/testsftp"
for user "testsftp"
Dec  6 10:13:58 net3 [11550]: chrooting to dir: "/home/sftpusers/testsftp"
Dec  6 10:13:58 net3 [11550]: setting uid to 531
Dec  6 10:13:58 net3 [11550]: processing request: "/usr/libexec/openssh/sftp-server"
Dec  6 10:13:58 net3 [11550]: running: /usr/libexec/openssh/sftp-server (username:
testsftp(531), IP/port: 10.0.0.12 1463 22)
From the logs, it looks as if everything worked: we got authentication, the chroot was set up, the sftp-server was invoked, and there were no additional logs. Yet it failed. Why? With a failure of a dynamic loading library, the exec call of the sftp-server may not be able to return a failure to the calling process because it is being overwritten by the exec call. A clean failure of an exec, such as not being able to execute the sftp-server file, will be logged; however, a missing dynamic library may be detected too late.

5. If you reach this point, review the dynamic libraries reported by ldd for the sftp-server. This is how the setup_chroot detects the needed dynamic libraries. Make sure all of these files are readable and executable for the user. For example:

$ ldd /usr/libexec/openssh/sftp-server
        libutil.so.1 => /lib/libutil.so.1 (0x40018000)
        libz.so.1 => /usr/lib/libz.so.1 (0x4001c000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x4002a000)
        libcrypto.so.4 => /lib/libcrypto.so.4 (0x4003d000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0x40133000)
        libc.so.6 => /lib/libc.so.6 (0x4015f000)
        libgssapi_krb5.so.2 => /usr/kerberos/lib/libgssapi_krb5.so.2 (0x40284000)
        libkrb5.so.3 => /usr/kerberos/lib/libkrb5.so.3 (0x40298000)
        libk5crypto.so.3 => /usr/kerberos/lib/libk5crypto.so.3 (0x402f6000)
        libcom_err.so.3 => /usr/kerberos/lib/libcom_err.so.3 (0x40306000)
        libdl.so.2 => /lib/libdl.so.2 (0x40308000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
6. You should also check the ldd(1) man pages and any related man pages, such as exceve(2), for the platform to see whether additional files are needed, such as the configuration file and cache file for Linux (etc/ld.so.conf and etc/ld.so.cache). If you find additional files are needed for specific platforms, please provide a pre- or post-processing script for scponly to Joe or join the scponly mailing list and talk to the people using and contributing to the software.

The full chroot'ed directory used for the Red Hat 9 system:

  ls -lR
.:
total 16
drwxr-x---    2 root     sftptest     4096 Mar 21 18:48 etc
drwxrwx---    2 root     sftptest     4096 Mar 21 19:02 incoming
drwxr-x---    2 root     sftptest     4096 Mar 21 18:48 lib
drwxr-x---    5 root     sftptest     4096 Mar 21 18:48 usr

./etc:
total 32
- -rwxr-x---    1 root     sftptest    23802 Mar 21 18:48 ld.so.cache
- -rwxr-x---    1 root     sftptest       33 Mar 21 18:48 ld.so.conf
- -rw-r-----    1 root     sftptest       60 Mar 21 18:48 passwd

./incoming:
total 0

./lib:
total 2792
- -rwxr-x---    1 root     sftptest    88760 Mar 21 18:48 ld-linux.so.2
- -rwxr-x---    1 root     sftptest   992092 Mar 21 18:48
libcrypto.so.4
- -rwxr-x---    1 root     sftptest  1465640 Mar 21 18:48 libc.so.6
- -rwxr-x---    1 root     sftptest    14408 Mar 21 18:48 libdl.so.2
- -rwxr-x---    1 root     sftptest    86852 Mar 21 18:48 libnsl.so.1
- -rwxr-x---    1 root     sftptest    49084 Mar 21 18:48
libnss_compat-2.3.2.so
- -rwxr-x---    1 root     sftptest    49084 Mar 21 18:48
libnss_compat.so.2
- -rwxr-x---    1 root     sftptest    68280 Mar 21 18:48
libresolv.so.2
- -rwxr-x---    1 root     sftptest    12288 Mar 21 18:48 libutil.so.1

./usr:
total 12
drwxr-x---    3 root     sftptest     4096 Mar 21 18:48 kerberos
drwxr-x---    2 root     sftptest     4096 Mar 21 18:48 lib
drwxr-x---    3 root     sftptest     4096 Mar 21 18:48 libexec

./usr/kerberos:
total 4
drwxr-x---    2 root     sftptest     4096 Mar 21 18:48 lib

./usr/kerberos/lib:
total 540
- -rwxr-x---    1 root     sftptest     5572 Mar 21 18:48
libcom_err.so.3
- -rwxr-x---    1 root     sftptest    73756 Mar 21 18:48
libgssapi_krb5.so.2
- -rwxr-x---    1 root     sftptest    63880 Mar 21 18:48
libk5crypto.so.3
- -rwxr-x---    1 root     sftptest   385220 Mar 21 18:48 libkrb5.so.3

./usr/lib:
total 56
- -rwxr-x---    1 root     sftptest    52616 Mar 21 18:48 libz.so.1

./usr/libexec:
total 4
drwxr-x---    2 root     sftptest     4096 Mar 21 18:48 openssh

./usr/libexec/openssh:
total 28
- -rwxr-x---    1 root     sftptest    26520 Mar 21 18:48 sftp-server
The full chroot'ed directory used for the Solaris 9 system:

ls -lR
.:
total 3
drwxr-xr-x    2 root     root         512 Jan  6 22:48 etc
drwxrwxr-x    2 root     testsftp     512 Jan  6 22:20 incoming
drwxr-xr-x    4 root     root         512 Jan  6 22:48 usr

./etc:
total 1
-rw-r--r--    1 root     root          76 Jan  6 15:56 passwd

./incoming:
total 0

./usr:
total 2
drwxr-xr-x    3 root     root         512 Jan  6 16:03 lib
drwxr-xr-x    3 root     root         512 Jan  6 15:56 platform

./usr/lib:
total 1053
-rwxr-xr-x    1 root     root      184040 Nov 13  2002 ld.so.1
-rwxr-xr-x    1 root     root      866176 Jan  6 15:56 libc.so.1
-rwxr-xr-x    1 root     root        3984 Jan  6 15:56 libdl.so.1
drwxr-xr-x    2 root     root         512 Jan  6 15:56 ssh

./usr/lib/ssh:
total 65
-rwxr-xr-x    1 root     root       66252 Jan  6 15:56 sftp-server

./usr/platform:
total 1
drwxr-xr-x    3 root     root         512 Jan  6 15:56 SUNW,UltraAX-i2

./usr/platform/SUNW,UltraAX-i2:
total 1
drwxr-xr-x    2 root     root         512 Jan  6 15:56 lib

./usr/platform/SUNW,UltraAX-i2/lib:
total 17
-rwxr-xr-x    1 root     root       16768 Jan  6 15:56 libc_psr.so.1
Summary

I have found scponly very useful for restricted chroot sftp access for several client situations and hope that sftp will continue to replace much of the current ftp usage. I think we will continue to see an increase in the expectation for traditional ftp configuration and security controls to be available with sftp. There are, of course, other desirable security features, such as the ability to restrict the sftp commands allowed and to perform additional logging such as partially addressed in the sftp logging patch by Michael Martinez. I hope these features will be bundled with future OpenSSH releases, thereby making it easier to deploy secure sftp solutions and decreasing the need for special sftp or chroot expertise.

Resources

Bastille Linux Project -- http://www.bastille-linux.org

Center for Internet Security -- http://www.cisecurity.org

Logging/umask/chown/chgrp Patch -- http://sftplogging.sourceforge.net

OpenSSH -- http://www.openssh.com

SANS Institute -- http://www.sans.org

Scponly -- http://www.sublimation.org/scponly

Ralph Durkee has done a wide variety of consulting and training on software development, systems and networking security over his 23-year career, and is the owner and founder of Durkee Consulting, Inc. since 1996. Ralph is also the Rochester/Buffalo, NY area SANS mentor/teacher for the GIAC GSEC (Security Essentials) and GCIH (Hacker Techniques, Exploits and Incident Handling). His specialty is Internet security consulting and secure systems software development. Ralph may be reached at: sysadmin@rd1.net.