No matter what language or package manager, dependency management in most projects suffer from many of the same problems:
Whether its npm, NuGet, PyPI, maven, Docker etc… it feels like a full-time job sometimes to be able to manage packages the way we all would like to. The problem is exacerbated depending on your company methodology for maintaining projects, the architecture of your repository granularity (i.e. do I have 100 different GIT repos to maintain packages for independently) and whether you live in a product-based or project-based world.
There must be a better way?
My first attempt towards solving the dependency management conundrums was two years ago. I started out specifically to automate some of the dependency upgrades within a .NET Core related GitHub project. This led me no further than a fantastic tool called “NuKeeper“. I’m a big NuKeeper fan. It has a ton of features that are second to none I’ve found in accomplishing some of the intricacies in dependency management that you’d want.
NuKeeper is a command line utility that will create Pull Requests based on proposed package updates with NuGet semantic versioning. As such this tool is specifically for .NET, but has fantastic support across Framework and Core (and several GIT providers).
I found its ability to configure includes and excludes based on wildcards incredibly helpful. Additionally, you can configure the granularity of Pull Requests that you want. For example, I could configure for only 10 updates, but I want it all in 1 PR or 10 PRs. One of the killer features is its ability to configure the age of a package before updating. I’ve found this to be indispensable in not updating packages before they are battle-tested by other consumers first.
The drawback is that it is a command-line tool (or a dotnet global tool). This makes it rather difficult to automate at an organizational level. Sure there are ways to do it. You can include the execution of the global tool in your project during a CI build. In an attempt to build something more agnostic, so I didn’t have to build it into every build pipeline, I put together a script, a YAML manifest and a single nightly build that would loop through and execute on each repository I had configured. This allowed the team to submit a simple PR to add new repositories without much additional work.
NuKeeper and this YAML configuration worked well for a short period of time. Its infrastructure necessary to configure internal private NuGet feeds was sloppy. Some weird exceptions ended up being problematic over time for projects with Pull Requests already submitted. It broke down after a while from the enterprise configuration and execution perspective.
During my NuKeeper analysis, I briefly looked into Dependabot as well. At the time it was a paid product (unless you were working on a public or open source project) and not something I got too far with. It looked very promising with a SaaS type of an approach, while also supporting a very wide variety of package managers. In May 2019, Dependabot announced it is joining GitHub and its public preview is now available for free or private and/or organizational usage. This changes things up quite a bit.
Over the past few months, we have been able to fully deprecate our usage of NuKeeper in favor of the Dependabot preview. This unlocked the same features we were using for .NET now across all our package managers. Thus far with its usage, my feedback is very favorable:
Getting started with Dependabot is as simple as adding the GitHub app for Dependabot and then add a config.yml to your repository.
version: 1
update_configs:
- package_manager: "dotnet:nuget"
directory: "/"
update_schedule: "live"
- package_manager: "docker"
directory: "/"
update_schedule: "daily"
It is also worth noting that GitHub has had their “Security Vulnerability” scanning feature available in their repositories for some time now. This feature is now integrated with Dependabot and you can explicitly receive Pull Requests from Dependabot for packages with identified security threats only if you wish. This is exciting to see the intersection of the GitHub features in this way. As an aside, I expect that in the future we’ll begin to see some very interesting compositions between Microsoft, GitHub Actions, Dependabot and NPM with GitHub.
Dependabot certainly scratches the itch that I’ve been trying to get at for awhile, but there are always some caveats. Consider the following thoughts and suggestions for efficiency as you get started:
I find myself still wanting some of the configuration features available in NuKeeper. The ability to configure the age of the packages was incredible. Under GitHub, I’m hoping we’ll see some of these more advanced features make their way into Dependabot, perhaps when it emerges from “preview”?
Dependabot is still in preview. This is always a concern when analyzing and rolling out tooling to your organization and/or enterprise. That being said, I can only see a bright future for the integration of Dependabot with GitHub, and we have chosen to accept the risk and dive in based on all the problems it can solve (it also feels low risk in the surface area if it is a flop). The Dependabot GitHub app does allow you to “enable for your organization” as a whole. This allows your developers in the organization to simply add that config.yml and begin receiving Pull Requests (no other configuration needed).
There are of course many advantages to incrementally staying up to date with the latest package dependencies that we have mentioned. But another internal advantage that is a large driver in pushing Dependabot is the integration with internal or private package feeds as well. This gives your organization the ability to push updates when internal package dependencies are updated as well. We share a number of libraries across teams, products, and projects. It is essential that when we make a bug fix, security patch or even just add a new feature that the library is pushed out to all consumers automatically and notifies them of exactly what has changed (Dependabot includes a list of GIT Commits in that release from the repository automatically in each Pull Request). This allows our teams to move fast with less friction on shared libraries.
Thanks Travis, good article. Can you expand on “Easily configure private feed credentials”? We’re investigating this – without baking credentials into the nuget.config file – and I haven’t found dependabot documentation covering this aspect. I have found a GitHub page suggesting Q4 2020 support; however, the source code suggests it is already functional.
Hey Brian – this post was written before some of the information about how Dependabot would be integrated natively into Github was released. As a result, “Dependabot” and that suggestion on Q4 support for 2020 is the native version of Dependabot that will be integrated with every repository for some of the features like vulnerability scanning, etc.
Since the native version does not support Private Nuget credentials at the org level yet, I am still using the Dependabot-Preview app and service which does. Dependabot has indicated to continue using the preview until support is available for private feeds, in which case I’ll need to migrate over to a slightly different config YAML format.
More Info: https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/
Dependabot Preview:
https://github.com/marketplace/dependabot-preview
https://dependabot.com/
[…] part of our standard development processes near the beginning of 2020. It is the final solution to a bit of my own journey in search of a tool to keep dependencies up to date. But it was an awkward position to begin onboarding to a set of preview functionality that was […]
[…] Discovering Dependabot many years ago was one of those moments of incredible excitement. It was not the same kind of excitement as when you finished creating and publishing that new feature to production, or even the same as when you got some amazing feedback about how your software helped save someone a ton of time… or maybe even changed their lives. This type of excitement is what you get when you realize someone else’s code is going to either change the way you develop and/or increase your quality of life. It is the epiphany that you get when you realize that this is the way it should have been all along… but for some reason wasn’t, and you could never place your finger on it. Dependabot was a breath of fresh air, in a world of DevSecOps that felt like my responsibility and requirements were ever-increasing. Dependabot can make at least a sliver of that pain go away. […]