Securing a development server

In this post I talk about setting up and securely operating development tools like Jenkins and Gitlab on a server connected to the internet. All applications run behind a firewall and a reverse HTTP proxy which allows only HTTP requests from selected users through who authenticate themselves with client certificates.

Putting web-facing software on the public internet is hard

My sysadmin persona is meek and easily intimidated, which is why I’m close to panic when I have to set up anything that needs to be reachable from the internet.  Over the holidays I set up a private build server which should be accessible by a small group of trusted developers and testers only. The server would run a few home-grown web applications and a build chain consisting of Gitlab, Jenkins, Nexus and Mediawiki. All these tools run their own embedded web servers of various flavours ranging from Nginx to Apache, they have their own web interfaces and countless binary dependencies… I honestly don’t trust myself operating this setup safely on the internet. Our servers are scanned daily by hundreds of bots for known exploits, if any of these applications were left unpatched for a few days then most likely the entire server would be compromised.

The approach we follow in securing access to the build chain is to operate all applications on a firewalled server and filter any HTTP(s) requests through a reverse web proxy which authenticates and authorizes users with client certificates.

1. lock down all TCP ports except SSH and HTTPS with a firewall

2. use a subdomain for every application like gitlab.example.com, jenkins.example.com (I’ll explain the reason later)
3. get an SSL certificate for the domain or self-sign one
4. distribute personal, client certificates to developers which they need to install in their browsers and tools like Git

5. use HAProxy for SSL termination and user authentication
6. HAProxy forwards HTTP requests to the firewalled applications

7. run dockerized images of those firewalled applications which bind to a virtual private network interface

The set-up described earlier channels, in theory, any network access to applications running on the server through a single, secure gateway. By filtering every HTTP(S) request through the secure gateway only selected users with client certificates installed in their browsers will ever get to communicate with anything that is behind the firewall, locking out casual attackers and bots that systematically scan for exploits. Even if an intruder copies a legitimate user’s certificate, there is a fail2ban filter which catches such “weird” HTTP requests in the DMZ, immediately blocks the user and sends out an alert Email. All this works under the premise, that there are no exploitable bugs in the firewall, reverse proxy and encryption modules, that there is no exploitable weakness in the encryption algorithms and that users don’t leak their client certificates.

ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp

# Haproxy Fail2Ban Regex using CLF format
[Definition]
failregex =^.*?haproxy.*?:  -.* 
ignoreregex = 

# 360 requests in 2 min :Ban for 10 min
[haproxy]

enabled = true
port = http,https
filter = haproxy
logpath = /var/log/haproxy.log
maxretry = 60 
findtime = 60
action = iptables[name=HTTP, port=http, protocol=tcp]
         mail-whois-lines[name=%(__name__)s, dest=%(destemail)s, logpath=%(logpath)s]
bantime = 600

This rule bans IP addresses that make more than 60 requests/minute to our server. We have more, much stricter filters on application logs of the internal applications. If suspicious requests show up there, then it means that either HAProxy has been compromised, a legitimate user turned rogue, has been hacked or their certificate was leaked. In these cases an alert email is sent out, HAProxy shuts down and all TCP ports are closed for a specified duration.

HAProxy [12] terminates HTTPS traffic, i.e. it accepts HTTPS requests by browsers, decrypts it and forwards it as HTTP to applications behind the firewall thus greatly simplifying tool set-up since we don’t have bother with hardening them or distributing certificates to them.

HAProxy also has a decent scripting language which can look into HTTP requests and take actions based on their contents. We use that for mapping URLs to individual applications such as Jenkins and Gitlab. The easiest way to do that is to run each tool in its own sub-domain (e.g. jenkins.example.com), mainly because most tools I saw have in one or another way hard-coded URLs in their web pages. For example, I wasn’t able to operate them in a URL hierarchy like example.com/jenkins or example.com/gitlab because they often assume that they are running at the domain root, e.g. the entire domain example.com belongs to them.

# SSL termination for  build server    

frontend www-https
        bind 78.47.203.76:8443 name https ssl crt /etc/ssl/private/server.pem ca-file /etc/ssl/private/server.pem verify required

        reqadd X-Forwarded-Proto:\ https
        use_backend gitlab  if { hdr_beg(Host) -i gitlab.example.com }
        use_backend jenkins if { hdr_beg(Host) -i jenkins.example.com }
        default_backend nobackend



backend gitlab

        http-response del-header Server
        server gitlab 127.0.0.1:10080 check

backend jenkins

        http-response del-header Server
        server jenkins 127.0.0.1:18080 check

backend nobackend
        server nobackend 127.0.0.1:65000 check
                                                 

Docker [14] needs no introduction. We’re using dockerized versions of Jenkins, Gitlab and Tomcat with locally mounted storage. We’re also using a local repository where we store our modified images. Docker runs containers on a virtual local network interface (called the “bridge”) which is convenient since it makes it easy for containers to talk to each other, but from a security perspective it’s irrelevant to our case since the entire server is fire-walled.

Apart from simplifying deployments, dockerized tools have the invaluable benefit of packaging all of their dependencies which is an elemental step towards simplifying security updates and avoiding DDL hell.

Important: keep in mind that Docker opens up all exposed ports on the host network even if ufw blocked them.

I set out to get our certificates from Let’s encrypt [15] but didn’t manage to issue our own client certificates based on the Let’s encrypt certificates. Again, I’m a horrible system administrator, so these setbacks are to be expected. We ended up using self-signed certificates (see [2,3,4,5]). Server certificates are automatically renewed with a cron job at fixed intervals and deployed to HAProxy, client certificates are also automatically and distributed over a different, secure channel to developers and testers.

We spent quite some time figuring out the ideal gateway mechanism that could distinguish legitimate from illegitimate users. We experimented a bit with HTTP basic authentication. Jenkins didn’t always like it and confused authentication present in the HTTP headers with authentication from its own database. We had HAProxy remove HTTP basic authentication headers once it had processed them which brought Jenkins back on track. However tools like Git, Maven or various IDEs can’t be easily taught to use HTTP basic authentication headers when talking to our development server, and since we wanted these tools to work on developer machines and communicate with the server, we abandoned the idea.

We briefly considered knockd [14] for port knocking [7], but in the age of mobile connections IP addresses are not stable enough to comfortably work with IP-based authorization.

Automatic security updates are necessary, but they sometimes break existing set-ups. Granted, on a headless server there is no X-server or Nouveau package to break and by running most tools in Docker images, binary dependencies are isolated for the most part. Still, I’m not too keen on finding out that e.g. Mysql has stopped working after the recent system update.Since, for better or worse, we already made up our mind about where the real danger comes from (namely the internet), but we want to minimize the number of updated packages, it’s logical that we limit updates to those public-facing packages: the network stack, iptables, network drivers, cryptographic libraries, HAProxy, the kernel and their dependencies. The OS is an Ubuntu server flavour where unattended-upgrades [1] are installed for exactly those critical parts and a daily cron job updates remaining selected packages.

Appendix

Appendix: on the need for a sub-domain per application

Most web applications we looked at require that their public URLs are mapped to the root path, e.g. https://example.com:8080/ for Jenkins. If one maps them (i.e. via URL rewriting) to a sub-path like https://example.com:8080/jenkins/ then they break in various subtle ways. Thus, hosting multiple applications that all need to be mapped to the root path on the same server is possible only when they run at either different TPC ports or at different public (sub) domains.

Since we’re trying to run a tight ship, opening more TCP ports doesn’t seem like a good idea. Instead, we have HAProxy map an individual sub-domains for every application to the respective application’s back-end.

Appendix: Docker containers talking to each other

That one popped up late. After having set up all tools we started integrating them and found out that e.g. Jenkins couldn’t talk to Gitlab over HTTP, mainly because it didn’t know how to resolve the Gitlab’s container IP. That is easily done with container linking [9]: when you link container A to container B, Docker puts container B’s name and IP address in container’s A /etc/hosts. As an added benefit, linked containers expose to each other their original TCP ports and not the ones mapped at runtime which makes for “prettier” host names and ports. For example, if the jenkins container is accessible at http://jenkins.example.com:8080 I still can map it via linking in such a way that other (linked) containers see it at http://jenkins (with port 80 implied).

Appendix: Gitlab

Gitlab doesn’t correctly work with SSL termination out of the box [8].  When it detects that its own public URL is HTTPS then the bundled Nginx will enable SSL termination and change the TCP port it is listening to, causing two problems: a) Gitlab will try to decrypt the already decrypted HTTP traffic that comes in from HAProxy and b) it is listening to a different port than agreed in the configuration. [8] tells how to fix that. Gitlab also consumes a lot of memory [9] which doesn’t exactly predestine it for containerization, but a glimpse into /etc/gitlab/gitlab.rb reveals several switches that reduce memory footprint.

Appendinx: Jenkins

The official Jenkins image (1.652.3) can’t clone git repositories because the linux user it runs under isn’t set up “correctly” (my words). When Jenkins runs Git it fails with:

docker jenkins unable to look up current user in the passwd file: no such user

The minimalistic image set-up lacks many useful tools (ifconfig, vi etc). So we’re using our own Dockerfile and fixing the Jenkins user:

FROM jenkins:1.625.3
USER root
RUN chmod 644 /etc/nsswitch.conf
RUN adduser consoleuser --uid 1002 --home /home/consoleuser --disabled-password
USER jenkins

Resources

[1] Unattended upgrades for Ubuntu server
http://askubuntu.com/questions/325998/how-to-enable-auto-security-update-in-ubuntu-12-04-server

[2] HAproxy: client side ssl certificates
https://raymii.org/s/tutorials/haproxy_client_side_ssl_certificates.html

[3] Using client certificates with haproxy
https://serversforhackers.com/using-ssl-certificates-with-haproxy

[4] OpenSSL
https://help.ubuntu.com/community/OpenSSL#Using_PKCS.2312_Certificates_in_Client_Applications

[5] Creating self-signed certificates
https://help.ubuntu.com/12.04/serverguide/certificates-and-security.html#creating-a-self-signed-certificate

[7] Port knocking
https://en.wikipedia.org/wiki/Port_knocking

[8] Gitlab behind proxy
https://github.com/gitlabhq/gitlabhq/issues/9911

[9] Docker container linking
https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/

[10] Gitlab high memory footprint
https://github.com/gitlabhq/gitlabhq/issues/903

[11] Fail2ban
http://www.fail2ban.org/wiki/index.php/Main_Page

[12] HAProxyhttp://www.haproxy.org/

[13] UFW
https://en.wikipedia.org/wiki/Uncomplicated_Firewall 

[14] Docker
https://www.docker.com/

[15] knockd
https://www.digitalocean.com/community/tutorials/how-to-use-port-knocking-to-hide-your-ssh-daemon-from-attackers-on-ubuntu

[16] Let’s encrypt
https://letsencrypt.org/