Dockerized Dependency Check: Building and NVD Image

Motivation

In a previous article, we discussed how we are ensuring that our software does not contain any known security vulnerabilities. For this purpose, we use an OWASP Dependency Check, which is integrated as a Gradle plugin to our CI/CD pipeline.

After releasing the original article, we heard back from many readers who wanted to know more about how we integrated/implemented this security process. In fact, there was one commonly asked question:

How long does the analysis take?

To elaborate on this, we have prepared two follow-up articles. In this article, we will answer the above question, why the default behavior of the Dependency Check is not satisfactory, and why we decided to mitigate potential issues by Dockerizing the National Vulnerability Database (NVD). Then, we will demonstrate the creation of the database image using Gradle.

Image title

In the second article, we will learn how to use our new image with the Dependency Check plugin without any additional configuration being required by the developers working on our applications.

How Long Does the Analysis Take?

So back to the point, how long does it take? This is an excellent question, because — if it was too slow — the teams would tend to save their time by postponing the analysis to the last possible moment, going directly against the preferred modus operandi: continuous integration. Also, if the length of the analysis was not predictable, it would create unnecessary friction — who likes builds which sometimes takes 7 seconds… and sometimes 7 minutes?

With our solution, we have consistently seen analysis take 15–30 seconds.

How Does the Dependency Check Work?

Before we go deeper into our implementation, let’s just quickly cover how the Dependency Check plugin (for Java) works:

  1. Gradle resolves a list of dependencies
  2. The Dependency Check stores the evidence in Lucene index. Evidence comprises of a resolved list of dependencies + contents of their manifest and pom.xml files
  3. The Dependency Check downloads the NVD Database (which contains a list of known software vulnerabilities) and materializes it as a SQL database
  4. The Dependency Check matches the contents of the Lucene index against the SQL database to find out if any of the dependencies has a known vulnerability

The Time

Steps 1, 2, and 4 are relatively fast — 5 to 15 seconds in total, depending on the size of the project. That’s not an issue and its also a relatively stable/predictable number per project. On the other hand, step 3 (download NVD database and materialize it in SQL) is a different beast entirely.

The initial load, executed with the first build on every machine, of the NVD dataset (vulnerabilities since 2002) and insertion to local H2 database takes 3–7 minutes. The resulting database is then cached in the Gradle cache, hence subsequent builds should not suffer from this delay. In reality, we found that the cache is being rebuilt from scratch on a weekly basis.

It can be clearly seen that the default properties of the Dependency Check were a bit problematic for us, as it made the build times unpredictable — and frustrated the developers. The other, less important downside was that the analysis may occasionally require an internet connection.

The Solution

The Dependency Check offers out-of-the-box the possibility of having a central corporate vulnerability database, which is centrally managed and all instances of the build plugin connect to this NVD instance. This is a great solution if you are OK with the fact that from now on your builds will require a VPN connection. This solution also requires that the centrally managed database is available at all times (HA). Lastly, you are not able to easily “freeze” the state of the database for a given project (to allow repeatable builds).

Based on these limitations, we have decided to go in a slightly different direction — to periodically build a Docker image with an H2 database containing the full NVD dataset. The image is built every week by our CI/CD engine, thereafter the individual clients/builds just need to download the latest image. In case the developer is offline, they may use the image from the previous week for local development (without any configuration change). This approach eliminates the need to have a centrally managed and highly available(HA) database.

NVD Container

Now to the code. First, we need to build a Docker image. For this task, we will use a Gradle build tool and the gradle-docker-plugin by Benjamin Muschko. Usage of Gradle may seem to be overkill, but keep in mind that this is a simplified version of the whole pipeline. In the real world, we also run a battery of smoke tests (written in JUnit) to verify that the image can be used in the wild. We also use the already mentioned gradle-docker-plugin to tag the images and push them to our Artifactory.

plugins { id 'com.bmuschko.docker-remote-api' version '4.10.0' id 'org.owasp.dependencycheck' version '5.1.0'
} group = 'com.zoomint' import com.bmuschko.gradle.docker.tasks.image.DockerBuildImage
import com.bmuschko.gradle.docker.tasks.image.Dockerfile def H2_RELEASE_DATE = '2019-03-13'
def BUILD_DIR = new File(buildDir, 'docker') dependencyCheck { cveValidForHours = 0 data { directory = BUILD_DIR }
} task copyEntryPoint(type: Copy) { from project.file('src/main/resources/docker-entrypoint.sh') into BUILD_DIR
} def H2_RELEASE_DATE = '2019-03-13'
def BUILD_DIR = new File(buildDir, 'docker') task createDockerFile(type: Dockerfile, dependsOn: [copyEntryPoint, dependencyCheckUpdate]) { from "adoptopenjdk/openjdk8:jdk8u212-b04-slim" destFile = new File(BUILD_DIR, 'Dockerfile') def H2_DATA_DIR = '/h2-data' label(["maintainer": "ZOOM International"]) runCommand "curl http://www.h2database.com/h2-${H2_RELEASE_DATE}.zip -o h2.zip && jar xvf h2.zip && rm h2.zip" runCommand "ln -s \$(ls /h2/bin/*jar) /h2/bin/h2.jar" runCommand "mkdir -p ${H2_DATA_DIR}" copyFile('odc.mv.db', "${H2_DATA_DIR}/odc.mv.db") copyFile('docker-entrypoint.sh', '/docker-entrypoint.sh') runCommand 'chmod u+x /docker-entrypoint.sh' entryPoint "/docker-entrypoint.sh" exposePort 8092, 9092 defaultCommand "java", "-Xmx512m", "-cp", "/h2/bin/h2.jar", "org.h2.tools.Server", "-web", "-webAllowOthers", "-tcp", "-tcpAllowOthers", "-baseDir", "${H2_DATA_DIR}"
} task buildImage(type: DockerBuildImage) { dependsOn createDockerFile inputDir = createDockerFile.destFile.get().asFile.parentFile tags = ["nvd-container-gradle"]
}

In the plugins section, we define the plugin we need to use. Aside from the Docker plugin, we need the Dependency Check for Gradle, which performs the database synchronization task — which is configured in the dependencyCheck section of the build.

dependencyCheck Extension

In the dependencyCheck extension, we specify that the plugin should always make an attempt to load the most recent version of the NVD database. This is technically only a concern when building on a local machine, as the database will be stored in the nvd-container-resources directory set in the directory assignment, but just to be on the safe side, it is better to make an attempt to update the dataset.

Once Gradle executes the task called dependencyCheckUpdate, it will download the NVD dataset, start up an instance of the H2 database, and load it with the NVD data. The datafile called odc.mv.db will then be written to the nvd-container-resources directory as we have configured.

Task copyEntryPoint

The next task copyEntryPoint does exactly what you would expect. Here is an example of the code from the entrypoint.sh:

#!/bin/bash
set -e exec "$@"

In our case, it is a trivial implementation of the best practice provided by Docker documentation.

Task createDockerFile

This is the part that does the bulk of the work. Firstly, we inherit the OpenJDK8 container provided by the AdoptOpenJDK project and set the location of the DockerFile generated by this task. The actual job starts with the first invocation of the runCommand, where we download and extract the current release of the H2 database. To make handling of the H2 DB version easier, we just create a simple symlink of *jar → h2.jar. Last but not least, we copy the data file from the NVD database created by dependencyCheckUpdate to the image.

Now, we just need to set the entry point with a proper default command, set the exposure of ports of the H2 database, and we are done.

Task buildImage

The buildImage task just wraps up our work by invoking the Docker binary, which compiles our image. The only thing we do here is that we assign the image a human-friendly reference nvd-container-gradle.

Summary

In this article, we have shown and discussed that the default behavior of OWASP Dependency Check is slightly problematic because the NVD database build can take up to several minutes to run. To mitigate unstable build times, we have re-laid the foundations of the solution by building a Docker image of the fully populated database. This ensures consistent build times and allows for continuous integration to be implemented by all teams.

Stay tuned for the next article, where we will learn how to integrate our Dockerized version of the database with the Dependency Check Gradle plugin.

Source Code

Get access to the full source code for the plugin and its usage in a showcase project in this GitHub repository.

This UrIoTNews article is syndicated fromDzone