Securing Traefik with Crowdsec

Introduction #

In a previous post I wrote about how I was using Traefik through Cloudflare Tunnels in order to securely serve pages from my Homelab to the internet. As great as Cloudflare are at blocking malicious activity inbound, inevitably some bad actors will still get through, so I thought it would be wise to add another layer of security to my setup. Many options exist that will provide some sort of automatic blocking based on suspicious activity, but I chose to go with Crowdsec.

Crowdsec #

Crowdsec is a piece of open-source software that crowdsources information about IP addresses that are behaving suspiciously, allowing extremely quick reactions to emerging threats. It can also scan the request logs for your applications (including common Reverse Proxies) against known methods of attack.

Crowdsec have written an excellent post here about how to set up their service with Docker Compose, but their instructions include information on their cloud console, which I’m not using. For the sake of simplicity, I’ve decided to document my setup here.

Setup #

I added Crowdsec to my existing Traefik on Docker Swarm setup. In order to do so, we need to create an instance of the Crowdsec engine itself. I’ve done this within the existing setup I documented in my previous post which is available in full here. Within the docker-compose.yml file, I added the following service:

  crowdsec:
    image: crowdsecurity/crowdsec:v1.6.6
    restart: unless-stopped
    environment:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/http-dos"
      CUSTOM_HOSTNAME: crowdsec
      BOUNCER_KEY_PLUGIN: ReplaceMeWithASecurePassword
    volumes:
      - logs:/var/log/traefik:ro
      - crowdsec-config:/etc/crowdsec/
      - type: tmpfs
        target: /var/lib/crowdsec/data/
        tmpfs:
          size: 524288000
    labels:
      - "traefik.enable=false"
    networks:
      - traefik

I originally placed the /var/lib/crowdsec/data mount onto my GlusterFS share, however I ran into issues when Crowdsec would try to update the database, as read/write speeds aren’t fantastic. From my experiments, the only disadvantage to having this database on tmpfs is that Crowdsec will take a few seconds extra to start up each time, but the performance increase from doing this makes it worthwhile for me.

Next, we need to add a ‘Bouncer’ which will query against the Crowdsec instance for every incoming request, and return a rejection message for any bad IP addresses. There are many ways of achieving this, but I’ve found the easiest way to integrate this into Traefik is using the Crowdsec Bouncer Plugin.

I modified the existing Traefik service within my compose file with the following:

  reverse-proxy:
    image: traefik:v3.3.4
    
    command:
[...]
      - "--experimental.plugins.crowdsec-bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      - "--experimental.plugins.crowdsec-bouncer.version=v1.2.1"
[...]
    deploy:
      labels:
        - traefik.enable=true
[...]
        # Middleware configuration for bouncer
        - traefik.http.middlewares.bouncer.plugin.crowdsec-bouncer.enabled=true
        - traefik.http.middlewares.bouncer.plugin.crowdsec-bouncer.crowdseclapikey=ReplaceMeWithASecurePassword

        # https://github.com/BetterCorp/cloudflarewarp/issues/30#issuecomment-1907922601
        - traefik.http.middlewares.realip.headers.customRequestHeaders.X-Forwarded-For=>CF-Connecting-IP
[...]
      
    volumes:
      - logs:/var/log/traefik
[...]

I’ve uploaded an amended version of my full docker-compose.yml setup here.

Finally, we need to tell Crowdsec to scan the Traefik logs for any suspicious activity. This is done by editing /etc/crowdsec/acquis.yaml within our Crowdsec container and adding the following:

---
filenames:
 - /var/log/traefik/access.log
poll_without_inotify: true
labels:
  type: traefik

Utilisation #

Once Crowdsec and the Bouncer are up and running, all we need to do is tell our application to filter all incoming traffic via the new middleware:

version: '3.7'

services:
  whoami:
    image: traefik/whoami
    command:
       - --name=externalapp
    deploy:
      labels:
        - "traefik.enable=true"
        - "traefik.docker.network=traefik"  

        - "traefik.http.routers.external.rule=Host(`external.yourdomain.com`)"
        - "traefik.http.routers.external.entrypoints=websecure"
        - "traefik.http.routers.external.tls=true"
        - "traefik.http.routers.external.middlewares=bouncer@swarm,realip@swarm"
        
        - "traefik.http.services.external.loadbalancer.server.port=80"

        - "traefik.constraint=proxy-public"

networks:
  traefik:
    external: true

Note the realip@swarm middleware - without this, you’ll be querying Crowdsec against the internal IP address of your Docker service!

Testing #

At this point, all incoming requests to your external-facing service should be queried against Crowdsec. If Crowdsec determines that the IP address in question is suspicious, a 403 error will be returned. You can verify this by entering into a shell for the crowdsec Docker Service, and running the following:

$ cscli decisions add --ip <YOUR IP ADDRESS>

Wait a few minutes (there is a cache) and then try loading your webpage - you should be blocked! Don’t worry, you can remove this again by running:

$ cscli decisions delete --ip <YOUR IP ADDRESS>

Conclusion #

Another layer of security is never a bad thing - I’ve been running this for a few months, and have seen a significant number of requests blocked. As a next step, there are plenty of additional collections that you can configure Crowdsec to utilise depending on what you are hosting.