ChatGPT for cyber risk management: an opportunity, or a threat? | Read here >>

CVE-2022-3656 in Google Chrome: How to fix the new “SymStealer” vulnerability | Read here >>

New report: Get a 360° view of the cyber risk landscape in 2022 and recommendations for 2023 | See the full report >> 

ChatGPT for cyber risk management: an opportunity, or a threat? | Read here >>

CVE-2022-3656 in Google Chrome: How to fix the new “SymStealer” vulnerability | Read here >>

New report: Get a 360° view of the cyber risk landscape in 2022 and recommendations for 2023 | See the full report >> 

How-to guides

Docker Security Best Practices - Secure the Whales

Rhett | January 09, 2019

For our day to day product deployment, we use docker containers. Whenever a new piece of code is being shipped to production, our CI/CD process creates several docker images and pushes them to our private registry – standard deployment process.

In the spirit of “shifting left”, we wanted to scan our Docker images and the 3rd party dockers we use in our architecture as a part of our pipeline. We want to prevent merging to master if they contain any critical vulnerabilities. We decided to try the open source clair-scanner which is based on Clair ‘analyze-local-images’ by CoreOS for the mission.

In our environment we use Gitlab as our code repository and CI.

Gitlab Ultimate provides an integration with clair-scanner results. When merging branches to master it analyzes the vulnerabilities report and displays the results in the merge request.

Gitlab also recommends running the scanner inside a docker executer that attaches to the hosts docker socket.

Please refer to the following articles for more info about integrating container scanning in Gitlab CI: CI Container scanning, Merge request container scanning.

Both of the above presented an obstacle for us, since we use Gitlab shell executers and don’t own Ultimate license.

In addition, we wanted to be more proactive about the vulnerabilities found.

Here is how we implemented a solution that would not require upgrading our license or changing executer from shell to docker which would create an issue for each vulnerable image with the list of vulnerabilities for our team to analyze.

What needs to be done:

  • Run arminc/clair-db and arminc/clair-local-scan:v2.0.1 containers linked one to another
  • Download the latest script for Clair scanner
  • Run the script to scan the desired images
  • Report an issue in Gitlab in case of vulnerabilities found

To avoid the overhead of starting clair bundle (scanner and DB) and waiting for it to “warm up” during each CI execution we decided to run the bundle continuously on the executers.

The issues we needed to tackle:

  1. When using docker run command to start Clair scanner we’ve noticed that it created numerous ‘git-remote-http’ processes without killing them. This ended us up with a bunch of zombie processes on the CI executer, until the container was restarted.
  2. Clair DB image is frequently updated and needs to be pulled and restarted. But there is no reason to recreate it if it wasn’t updated.


Docker-compose to the rescue!

To overcome both of the above we’ve decided to run the scanner and DB using docker-compose, instead of utilizing maintenance scripts or adding logic to the docker run commands.

To fix the first issue we’ve added ‘init: true’, which forwards signals and reaps processes.

Please refer to this issue to find more info about zombie processes in CoreOS Clair.

For the second issue we run docker-compose pull and then docker-compose up, which will only recreate the containers in case they are newer than the current ones running or will just pull latest versions and run them, in case they are not yet running.

The docker-compose file is quite simple, with the addition of init to clair scanner:

version: ‘2.2’
    container_name: clair-db
    image: arminc/clair-db:latest
    restart: on-failure  clair:
    container_name: clair-scanner
    image: arminc/clair-local-scan:v2.0.1
    init: true
    restart: on-failure
      – postgres
      – “6060:6060”



Gitlab CI uses a .yml file to define the stages. Our first step was to add the “container-scanning” as a new stage in the process.

  – build  – test
  – container_scanning
  – push_docker_images
  – deploy
  – cleanup


Then we created the stage itself:

  stage: container_scanning
    – master
    – cp -r app/devops/clair_scanner/ /opt/
    – docker-compose -f /opt/clair_scanner/docker-compose.yml pull
    – docker-compose -f /opt/clair_scanner/docker-compose.yml up -d
    – curl -s -L -o clair-scanner
    – chmod +x clair-scanner
    – for image in $(grep ‘image:’ docker-compose.yml | awk ‘{print $2}’ | sort -u); do ./ $image; done

Let’s break it down a bit:

What this does is limit this stage to run only on master –

  1. Copies the folder that holds essential files for clair docker-compose (docker-compose.yml and any env files you may want to use with it) to /opt/
  2. Pulls the newest versions of images specified in docker-compose.yml
  3. Executes docker-compose up to boot/update the docker containers for Clair scanner and DB
  4. Downloads the scanner script
  5. Gives the script executable permissions
  6. Searches for images in docker-compose.yml, which defines our applications’ architecture (this is not the same as the clair docker-compose.yml) and passes each image as an argument to our custom script ( This script calls clair-scanner executable from the previous stage, parses the output and automatically opens/updates Gitlab tickets based on the results.

The whole process is quite simple and quick overall.

The real magic is in our script, (I’ll replace most of the code with pseudo code for easier reading but keep the Clair parts intact).

Feel free to contact us for the full code.

The first thing we do is run the scan and save the output. We also keep the JSON file to be later attached to the Gitlab ticket.


vuln_file_name=$(basename $IMAGE | sed ‘s/:/_/g’)

echo Scanning” $IMAGE
ticket_title_unform=$(./clair-scanner -c http://localhost:6060 –ip $(hostname -i) -r $vuln_file_name.json –threshold=”Critical”-l clair.log $IMAGE)


We then take the output, parse and strip from formatting (like shell colors) and any unneeded text.

According to the result, we decide what we want to do:

if [ “any vulnerabilities found” ]; then
        if [ “similar ticket already exists” ]; then
            echo “Ticket already exists. Checking if it needs to be updated”
            if [ “similar ticket is not up to date” ]; then
                echo “Updating existing ticket with newer data”;
                echo “Uploading json attachment”
                echo “Adding comment with pipeline details to the existing ticket”            else
                echo “Ticket is already up to date”
                exit 1
            echo “Will create a new ticket”
            echo “Uploading json attachment”
            echo “Creating GitLab ticket with pipeline details and security tag”
            echo “New ticket has been submitted. Please refer to GitLab”
            exit 1
        echo “No vulnerabilities were found for $IMAGE”;


To submit or update an issue we use Gitlab API

Voila! A totally free vulnerability scanner integrated in our pipeline, with reporting capabilities!