This document describes the precautions taken by the IITS Unix Group to protect its web servers, with particular emphasis on the use of chroot. Criticism is welcome; please send your comments to the author.
At our site, we have a challenging mix of requirements for web applications: we have a multiuser system where individual users must be allowed to create their own web pages and CGI scripts; we have applications whose results we have been asked to make available over the web, but which are not sufficiently robust to survive attack by hostile user-supplied data; and we have our own applications, in which we are reasonably confident, but which must run with full privileges, thus where a programming error could jeopardize the entire system. Securing web servers under these conditions presents a challenge.
Many attacks are possible on a web server. Most obviously, by deluging the server with a large number of requests, or with particularly resource-intensive requests, a denial of service can be effected on the web services, or potentially on any other service using the same host if the web requests degrade the performance of the host. Symmetrically, a denial-of-service attack on any other service on the host could potentially affect the provision of the web service. Such attacks are not specific to web services and will not be discussed any further here.
Much more dangerous are attacks whereby the web server or its associated programs (such as CGI programs) are subverted to give unauthorized access to data, or worse, to alter data or run programs in an unauthorized manner. Vulnerabilities may exist in the web server software which permit such attacks; in that case, only a patch to the web server will solve the problem. We assume that a careful webmaster will keep informed about known vulnerabilities and patch the server appropriately; however, it should be borne in mind that the server may not be watertight, and a second line of defense should be used where possible. In particular, the web server should run as an unprivileged user and should be chrooted.
Vulnerabilities may also exist in the configuration of the web server. Always use the minimum permissions that will do the job. Apache has a great page of security tips for server configuration which are required reading for any webmaster. In addition, the use of a second line of defense as mentioned above will limit the damage in case a mistake in made in the configuration.
Finally, we get to probably the most common source of web server security problems; CGI programs. Errors in CGI programs which permit user-submitted data to be processed in unexpected ways can in effect allow the remote user to execute any program with the privileges of the web user -- again, an argument in favour of the second line of defense mentioned above. The safe handling of CGI will be discussed in more detail below.
Although we try to do everything right with respect to keeping our web server up to date with all security patches, reviewing the configuration very carefully to make sure that we have complete control of who can access which files and run which programs, and vetting CGI programs to eliminate dangerous programming errors, we also assume that mistakes can easily be made in any of those areas.
Therefore, we protect the system against those mistakes by using a two-fold second line of defense. First, we minimize the web user's privileges. While the parent apache process runs as root, it is configured to drop its privileges to nul-web in all of its children. Thus, the root-owned parent process only accepts and hands off connections (and logs them), reads the SSL certificate information, and closes and reopens the log files. The nul-web user, under which run the other Apache processes (those which do the actual work), has very few privileges aside from those granted to "world" (all users of the system): it can access the SSL cache, it can write the cgiwrap log, it can execute the setuid-root set of cgiwrap programs, and in some cases it can access parts of the document tree that world cannot. Thus, if a remote attacker breaks through our protections, they can perform only actions which nul-web can perform: potentially they can read files to which we'd prefer to restrict access, and they can perform any actions which any user on the system can perform.
This latter set of actions, however, can be dangerous. First of all, system users have access to files which we might not want the rest of the planet to see; on some systems, this includes the password file with its encrypted passwords. Secondly, the operating system is much more vulnerable to local (from a session on the host) than to remote (from the network) attack, for example via vulnerabilities in privileged programs. This is the motivation for running the web server in a chroot, a Unix changed root. Under this scheme, the web server can see only that part of the filesystem under the chroot directory. Thus, we can prevent the web server from accessing most of the filesystem in any way, by running it in a "prison" where only web pages and a small number of programs are available.
When we set up the chroot environment for our web server (with the chroot directory called /public or /pubweb on various systems), we must include all of the web pages, the web server configuration and log files, the CGI programs and any support programs they might need (for example, perl), and of course cgiwrap. That much is obvious. It also helps to create /public/public (or /pubweb/pubweb) as a symbolic link to "." so that programs can be tested inside the chroot and outside it without changing the code.
We must also include copies of a few operating system files without which nothing can run; this set of files varies from one version of Unix to another. Under Digital Unix, for example, the following files are needed:
Most of our web servers have Perl CGI scripts, so on those, in addition to the perl binary and library modules in appropriate locations, we need:
In addition, on our multi-user host, a system with a wide variety of CGI programs, we added a few commonly used tools (/bin/sh, /bin/csh, /bin/date, /bin/cat, /bin/nawk), some indexing software (swish), a mail utility (minimailer), and various shared libraries and other files:
How safe is chroot? It is possible to escape a chroot environment, but it isn't easy. Two methods come to mind, and both of them require being able to run arbitrary code on the chroot side. These methods will work with some but not all flavours of Unix.
Privileged programs, in particular those that run as the Unix root user, are worthy of very close examination, since programming errors in them can permit an attacker to run arbitrary commands as root: in other words, to have complete run of your system. Most Unix exploits involve programming errors such as buffer overflows or the careless use of system or popen calls in privileged programs.
What privileged programs exist under the chroot? Two of them: first, the web server parent process, which as we have seen performs fairly minimal tasks. (The smaller and simpler a program or code section is, the easier it is to verify that it is correct; also, Apache is distributed with full source code, and this source code has been examined by hundreds of people to make sure that it is not vulnerable to security exploits.) The second privileged program is cgiwrap, which permits us to run CGI programs with the permissions of our users instead of the permissions of the web user. The author of this web page has personally read through cgiwrap, which is very small, as have hundreds of other people. It is safe.
Thus, the chances of any attacker, local or remote, being able to gain root privilege by subverting a privileged program under the chroot are negligible.
CGI programs are the most likely point of failure in the security of a web server, because they must accept and process data from untrusted sources. If this data is "hostile", and is not handled correctly, the result could be the execution of programs outside the control of the CGI program author. We have already seen how to limit the damage caused by such breaches using a second line of defense, but it is still important to vet all CGI code to make sure that it is not susceptible to such attacks.
On a system where a large number of users may create their own CGI programs, it is impossible for the webmaster to verify all of that code. Therefore, and keeping in mind the existence of the chroot prison, it is often sufficient to ensure that a user's CGI program run with her privileges only. Thus, if the program is compromised, it can damage only files writable to the user, and it can reveal only information readable to the user, under the chroot. Thus, the user's regular home directory is not attacked, but only her chrooted web directory. On our large multi-user system, we warn our users that their CGI mistakes could compromise their web files, but we otherwise allow them to create CGI programs as they please. (Of course, very sloppy programming could allow an attacker to use our system to launder connections and attack other systems, and we would of course intervene if this happened, but in view of the type of userbase on this system, the likelihood that we already have potential attackers with legitimate accounts on the system, and the difficulty of setting up such an attack, we consider the risk acceptable.)
In order to wrap the user's CGI programs, simply make sure that cgiwrap (and its cousins cgiwrapd, nph-cgiwrap, and nph-cgiwrapd) are the only CGI programs which can be run directly by the web server, by configuring the web server appropriately. Then, direct the users to use the cgiwrap URL to access their programs; full instructions come with the cgiwrap program. A user's view of cgiwrap is available on the Alcor web site.
After all these warnings, how can you make sure that you are using secure programming techniques in your CGI programs? The principle is this: treat all data as potentially hostile, or "tainted". If you are programming in Perl, you can get Perl's assistance is forcing you to explicitly check all of your data by invoking perl with the "-t" (taint-checking) option. Detailed secure CGI programming tutorials are linked from the user's view of cgiwrap page mentioned before. Please read them.
Fuller coverage of web security issues can be found in CERT's Security Improvement Module Securing Public Web Servers, a compact but complete practical guide to the configuration and operation of a web server, and in the Web Security Sourcebook by Aviel D. Rubin, Daniel Geer, and Marcus J. Ranum (1997, John Wiley & Sons), which covers browser as well as server security.
So here we are in our super-secure chroot environment. But for a production service, we need to do something that requires access to the whole system -- for example, we are running a large database and we need to generate reports from it, after authenticating the user requesting the report. How can we do this? We can do it by poking a tiny hole in the chroot.