Over the past several years, the frontend JavaScript development ecosystem has been moving increasingly towards the use of dependency managers such as NPM and application bundlers such as webpack. While these are great moves for the ecosystem - providing automated and standardized management of third-party code, along with modularity - they present some unique challenges for scanning tools such as Sonatype IQ Server.
These challenges were brought into sharp focus for us here at Sonatype when we recently updated one of our own codebases - the IQ Server itself, in fact - to fully use NPM and webpack for its dependency and bundling needs. Once we made this change, we were left to determine the best way to reconfigure our CI server's IQ Policy Evaluation tasks to accurately evaluate the application.
Previously, the Sonatype IQ frontend code had a JavaScript source code directory, and then a directory of third-party library files - some added manually and stored in source control, others downloaded at build time using Bower. This was fairly straightforward, as it more or less guaranteed that those directories only included files that we were actually using. Once we moved to NPM-managed dependencies, however, things changed.
Now we have our own sources, which are still simple enough, and a node_modules folder, which includes each third-party dependency. There are several difficulties here.
For one thing, the node_modules folder contains both build-time dependencies (such as Webpack itself) and runtime dependencies that get bundled into and distributed with the app. We are primarily concerned about policy issues only with the latter.
Secondly, many NPM packages include extra files that aren't part of the library itself, and the dependent application doesn't typically get used, which, at least not in a webpack setup. Take the angular-vs-repeat package, for example. In the current version (1.1.7), it includes not only the library code itself, but also its own build files, automated tests, and dependencies. Most notably, it includes its own copy of Angular.js - a rather old copy at version 1.2.16.
As it turns out, this happens fairly frequently - NPM packages include their own copies of certain dependencies, even though they don't use them at runtime.
Back to Sonatype IQ, we had to figure out where to point the scanner to get an accurate view of what was getting included in the application and what policy issues might be present. Scanning the bundled output file is fruitless - with all dependencies combined and minified into one file, nothing gets identified.
The other obvious option is to scan the node_modules directory. This however turns up numerous false positives for the reasons described above. We saw policy violations regarding multiple outdated instances of Angular and other libraries that dependencies include in their file tree, but which IQ isn't actually including into its bundle. We needed a way to scan only those files which Webpack actually pulled into the bundle. We needed a way to get webpack to tell us exactly which files to scan.
Luckily for us, Webpack has support for plugins. By writing a plugin, we could easily get Webpack to dump all the original files that it is bundling into a separate directory. We could then point the IQ scanner to scan exactly those dependencies with which we need to be concerned. We wrote such a plugin, and it solved the problems perfectly. Our reports are no longer littered with false positives from files that aren't truly part of the app.
We realize that these challenges, and the need for this workflow, are not unique to our own development. For that reason, we released the webpack plugin to our FOSS community repo: it can be found here. It is used much like any other webpack plugin, with a single configuration option telling it where to put the copied source files. See the README for details.
At Sonatype, we are not just the developers of our products. We are also users. Like many of you reading this article, we combine the use of those products with modern development practices. When we hit a hurdle, we address it to best enable our community to succeed.