Integrating static analysis tools with build servers for Continuous Assurance

by | Mar 18, 2015 | Blog, Code Dx New Features

Share This Story, Choose Your Platform

Continuous Integration (CI) has made a tremendous impact on how we develop software today since it was first coined by Grady Booch in 1993 in his book Object-Oriented Analysis and Design with Applications:

At regular intervals, the process of “continuous integration” yields executable releases that grow in functionality at every release…It is through these milestones that management can measure progress and quality, and hence anticipate, identify, and then actively attach risks on an ongoing basis.

It later took off when Kent Beck and Martin Fowler extended and evangelized the concept in 1999 as part of eXtreme Programming (XP):

… a fully automated and reproducible build, including testing, that runs many times a day. This allows each developer to integrate daily thus reducing integration problems.

Can you imagine developing software without it? I certainly cannot.

The concept is simple: fail fast and fail often. This allows the team to fix problems before it becomes a big deal; saving time and money in the end.

The benefits CI provides are numerous. For one it ensures that you always have a working build and consistent results since the build is performed on the same machine. No more “Works on my machine” issues. There’s also always a place to get the latest working build, which makes the current state of things more visible to the entire team.

The beauty of CI is that it’s automated. Once you have a build set up and configured, you can just let the CI server do its thing. No more spending time doing manual error-prone builds. Any time someone commits new code, the CI server is triggered to run a build or builds. If something goes wrong, the build will be marked broken, the developer who broke the build will get an e-mail notification, and they’ll work to resolve the issue as soon as possible; otherwise their teammates will get on their case.

Automated builds are one thing, but where it gets interesting is creating unit and integration tests that run as part of every build. Test-driven development gets to be part of your build process and you have automated regression tests running continuously.

By integrating so frequently and having automated checks, the issues will be discovered early and be easier to fix since the changes that caused the failure will be fresh in the developers’ minds, and the amount of change will be minimal, making it easier to track down the cause.

Continuous Delivery and DevOps

Today, organizations are moving at any even faster pace with Continuous Delivery and DevOps. This is the act of pushing out releases to production very frequently or even Continuous Deployment. Forget about releases twice a year, organizations like Etsy pushes out 50 updates per day, Amazon on average every 11.7 seconds, and Flickr deploys 10 times per day. This type of methodology makes it even more challenging to thoroughly test. Automation is really the only way to make this happen. And if done correctly, many find a DevOps way is actually easier since fixing things that may go wrong can happen so fast.

Continuous Assurance

The same process and benefits that people have been getting with CI can also be applied to improving code quality and security, providing for continuous assurance even in the fast pace DevOps world.

Developers should be performing these important quality and security checks during development in their IDE, the same as with unit tests. However, having a central location is still necessary to make certain this testing is actually done. Build automation provides this workflow.

Setting up Continuous Assurance with Code Dx

Setting up continuous assurance with Code Dx is a breeze. You can use our REST API to integrate Code Dx with your existing CI build server or, for those using the very popular Jenkins CI, we have a native plugin that makes it easy to get started. The rest of this section will focus on Jenkins integration and provide some tips for setting things up.

The first step is to install the Code Dx Jenkins plugin. This can be done simply by going to the Manage Plugins section of your Jenkins installation, searching for the Code Dx plugin, and following the Jenkins provided instruction to install.

Once installed you will see a new Publish to Code Dx Post-build Action for each Jenkins job. This allows you to add a continuous assurance step to your existing build process, probably one of the last steps in your job after the binaries have been built and unit tests have been run. Configuration details are described in our online help. The basic process is to tell the Code Dx Jenkins plugin where the source and binary files are located and then the plugin will automatically zip up those files and send the zip to the Code Dx server specified for analysis. You also have the option to include commercial tool results as well.

jenkins-post-build-actionsYou have the option to just fire off the Code Dx analysis and let the Jenkins build complete, or hold up the build (select Wait for Analysis Results) and wait for Code Dx to finish its analysis. The benefit to waiting is it allows you to break the build if Code Dx finds something and also to view summary results.

Breaking the build

One of the great features of CI is it allows you to set up conditions to break the build or mark a build as unstable. If a developer commits new code that causes something to fail, an e-mail can be generated to the author, the project lead, and/or the whole team.

For continuous assurance, you can specify criteria such as break the build if a new finding appears that is of high severity. Now you’ll have automated assurance that any new high severity issue will be automatically flagged and addressed.

Result summary

The Code Dx Jenkins plugin provides high level summary information such as a table breakdown of the number of findings and a breakdown of the current status of each finding, for example, New (not seen in last build), Unresolved (appeared in last build and is still there), False Positive, Escalated, or Assigned to a developer. Also, for each entry, the change since the last build will be shown.

jenkins-tableThe user can access the details of the findings via the Code Dx web interface. A link to the latest results is available directly from the Jenkins page (View latest in Code Dx). Also, Code Dx has IDE plugins that can be used to view and remediate the findings.

Monitor trends

The Code Dx Jenkins plugin also allows for monitoring trends: Are the number of high severity findings going up or down? How fast are we able to resolve findings after they are discovered? Are we managing to keep up with new findings?

jenkinds-trendsWhat to do when analysis tools take a long time to run

The analysis tools bundled within Code Dx are designed to be fast. Usually completing in a matter of minutes for most sized projects. This fits well with CI principles stating that you really want the build to finish fast; otherwise you introduce a gap between when the problem was introduced and when the developers are made aware of the problem. This leads to it being harder for the developer to fix since they’ve most likely moved onto another task in the meantime.

For those using commercial tools, such as HP Fortify or IBM AppScan, these tools can take considerably longer to run, often for good reason since they are doing in-depth analysis, but integrating this with an automated build can be a challenge. Even Code Dx’s bundled tools can take longer than acceptable for some really large projects.

Creating a build pipeline with downstream Jenkins chained jobs can be helpful since you will get your build results right away and the longer running analysis will come a bit later. However, if it’s going to take several hours to run your static analysis tools, you probably do not want to run on every code commit happening throughout the day. That would be inefficient since your static analysis job will never be able to catch up, especially if you have multiple projects contending for the same system resources.

One solution is to run these long run static analysis tools as part of a separate Jenkins job and also run them nightly or twice a day instead of with every commit.

Jenkins makes this super simple. Just create a new Jenkins job for continuous assurance:

jenkins-new-jobThen set the build trigger to have it run at midnight and noon:

jenkins-build-triggersNext you need to set up the build for this new job. One way to set up this new job is to copy your main project and run through the whole build again. But I wouldn’t recommend that since there’s really no need (unless the source code analysis tool you’re using requires it) and often times the job configuration can get complicated and then you’d be stuck keeping both jobs up-to-date as changes are made.

Instead, a nice solution is to use the Jenkins Clone Workspace SCM Plugin. This allows you to copy selected workspaces files from another job.

After installing that plugin, you can then do something like below in your main job to clone the source and bytecode. Or you can leave it blank to clone the whole workspace.

jenkins-clone-workspaceAnd something like this in your continuous assurance long running job:

jenkinds-receive-cloneYou can then have build step(s) to run your commercial tools, and finally send your source/bytecode to Code Dx along with your commercial tool results using the Code Dx Post-build Action.

Good luck taking CI to the next level by making Continuous Assurance part of your SDLC!

Share This Story, Choose Your Platform