Dependency Manager

Modular software is composed of modules that could depend on other modules. Source: Elena 2017.

A good practice in software design is to build software from smaller, single-purpose modules that expose well-defined interfaces. This is also in the spirit of software reuse. With the widespread adoption of open source software, often our own applications or modules depend on those written by others. Thus, to build our software we need to bring in all parts on which it depends, including language libraries and remote third-party modules.

But it's not trivial to ensure that we have all necessary dependencies, particularly when dependencies themselves depend on others. This is why we need a Dependency Manager, often invoked during the software build process. A useful definition states that,

Dependency management is a technique for declaring, resolving and using dependencies required by the project in an automated fashion.

Discussion

Isn't dependency manager same as a package manager?
No. A package manager operates at the system level and most often deals with binaries. It's role is to install libraries, tools, and other applications. It affects the entire system and not just one project. Because of this, it requires root access. Examples of package managers are apt-get (Linux), Brew (MacOS), and Chocolatey (Windows).
A dependency manager operates at source code level. It's role is to setup the right dependencies for a particular application. It therefore implies that a dependency manager is something used by developers, not users or system admin.
Thus, a dependency manager is project specific. It helps in easy management of project dependencies. It has to ensure that same source code is used across environments, for example, when moving from Windows developer machine to Linux production system. There should be end-to-end visibility of dependencies. A dependency manager is essential for Continuous Integration. A good dependency manager should also make it easy to publish new packages.
What are the main components of a dependency management system?
A dependency management system typically has the following components:
- Module: A software artefact, that we wish to use in our project. Depending on the language, modules are also called packages or libraries. Modules come with metadata to indicate dependencies.
- Manifest: Usually a file that specifies immediate/direct dependencies for our project. It specifies intent, such as, saying that we desire "version 2.3 or above of module A".
- Lock: Usually a file that captures a particular realization of the developer's intent. To extend the above example, version 2.3.2 of module A might be used.
- Repository: A place where modules are stored. Often repos are remote and accessed over the internet. Dependency manager usually knows the location of the repo but location can also be specified in the manifest file.
- Dependency Constraint: Usually in the manifest file, this is a constraint about versions of a module that are allowed for this project.
- Resolution Rule: Rules are applied by the dependency manager to arrive at a suitable module version. Selecting the right version is called Dependency Resolution.
What's the usual Project Dependency Management (PDM) pipeline?
A PDM pipeline involves four states and their forward transitions. Source: Boyer 2016.
Dependency managers bring together all necessary project dependencies before passing them to the compiler or interpreter. In this sense, their work is somewhat a preprocessing phase before the actual build happens.
Dependency managers start by reading the manifest file, in which direct dependencies are noted. They then read the metadata of these dependencies from their repositories to figure out the next level of dependencies. In other cases, they may download the dependencies right away and then process their dependencies. Either way, all dependencies must be downloaded and installed.
Exact installed versions of dependencies are recorded in a lock file. While the manifest file is important for a developer, the lock file has greater importance for operations folks. The lock file makes the project's dependencies reproducible in any environment.
What is transitive dependency?
Transitivity is a concept in logic and mathematics. In the context of software dependencies, it can be explained as follows. If module A depends on module B, which in turn depends on module C, we can say that module A also depends on module C. Many levels of dependencies produce what's called a Dependency Graph.
When a module appears multiple times in the graph, with different version numbers, the dependency manager must take a decision about which version to use. A similar decision must also be made when dynamic version numbers are specified. This selection process is called Dependency Resolution.
A good dependency manager will not only resolve direct dependencies as mentioned in your project's top-level manifest file but also resolve all transitive dependencies. It should be able to handle any level of nesting. Typically, cyclic dependencies will cause problems.
Could you mention some rules used in Dependency Resolution?
We mention a few sample rules below for specific dependency managers. However, similar rules are likely to exist across many dependency managers.
In Gradle, after taking into account all transitive dependencies and constraints, the highest suitable version is selected. If there's a conflict, an error is thrown. You can also use dependency configurations and make manual adjustments. Maven uses the nearest version but if conflicts are at the same depth, the first one wins based on the order of declaration.
NuGet uses four main rules: lowest applicable version, floating versions, nearest-wins, and cousin dependencies. This rule is applied with a warning since the package at a deeper level might get downgraded to an older version.
NPM actually builds a dependency tree, where different versions of dependent packages can be installed in different branches of the tree. However, using what's called "peer dependencies", NPM allows us to enforce same versions where desired.
What's the purpose of lock files used by dependency managers?
The auto-generated Gopkg.lock is used in Go. Source: Yuen 2018.
If dependencies are specified in terms of exact versions, anyone else trying to build your project in any other environment will use the same versions.
When dynamic version numbers are specified in manifest files, such as a range of versions, the version used may depend on the environment or the latest version available in the repository at build time. Dynamic version numbers are good because they allow us to bring in latest features/patches of dependent modules into our project. However, this causes uncertainty for continuous integration and production release. To solve this, versions used are recorded in a lock file. This lock file must be committed to version control so that anyone else using the project is guaranteed to use the same versions.
What if you use only exact versions? Is lock file required in such a case? Yes. Even if direct dependencies are exact, transitive dependencies might be specified in a non-exact manner. Therefore, a lock file is essential in all cases.
Lock file is sometimes called "a manifestation of the manifest".
What should I look for when choosing a dependency manager?
The purpose of a dependency manager is to allow developers create a consistent and reproducible set of dependencies for their projects. It should be easy to use, have a well-defined behaviour and hooks for customizations. Some essential features to compare are the following:
- Organization: For example, Yarn uses flat mode while NPM builds a dependency tree.
- Semantic Versioning: Has provision to specify a range of versions.
- Dependency Locking: Uses lock files to reproduce exact versions.
- Dependency Resolution: Handles transitive dependencies. Resolves conflict with a well-defined set of rules.
- Multiple Configurations: Allows different versions of the same package. This is enabled by defining different configurations or groups, such as build vs testing.
- Performance: Caches downloaded packages for faster updates. Parallel downloads.
- Security: Does integrity checks via package checksums.
- Customizations: Dependencies can be made optional or versions can be manually specified. Can be configured to fetch from any repository. Supports offline installation.
- Support: Has good community support and adoption, with frequent updates.
What are some best practices that developers can adopt to better manage project dependencies?
Developers need to version control not only the source code but also the manifest file and the lock file. In particular, always version control your lock file if you're building an application. If you're building a library, it's a good idea to do this as well. However, if you wish to build your library with the latest versions of its dependencies, it's recommended to use a second continuous integration build.
If you're making a Java library, let it not depend on frameworks such as Spring. Changes to frameworks can cause "dependency hell" for users of your library. Don't ship fat JAR files, files that include all dependencies. Only depend on what you need.
Finally, it's best to avoid dependencies if you can implement them yourself. The argument is that this will keep your application lean and secure. Check if there's a native library to do the equivalent. Look for those with less transitive dependencies. Check memory usage and load time. Use fewer high-level interfaces. Prefer low-level function calls. Use abstractions. Test regularly on target environments.
Could you give examples of dependency managers in various languages?
A few examples of dependency managers are listed below in the format name: (language, default repo, default manifest, default lock):
- Composer: (PHP, Packagist, composer.json, composer.lock)
- Gradle: (Java, Maven Central, build.gradle, gradle/dependency-locks/*.lockfile)
- Node Package Manager (NPM): (NodeJS, NPM, package.json, package-lock.json)
- Yarn: (NodeJS, NPM, package.json, yarn.lock)
- Bundler: (Ruby, RubyGems, Gemfile, Gemfile.lock)
- dep: (Go, -, Gopkg.toml, Gopkg.lock)
- Cargo: (Rust, Crates.io, Cargo.toml, Cargo.lock)
- Pipenv: (Python, PyPI, Pipfile, Pipfile.lock)
In many languages, newer, better dependency managers are being preferred over older ones: Gradle over Maven and Any-Ivy (Java); Yarn over NPM and Bower (JavaScript); Bundler over RubyGems (Ruby); Composer over PEAR (PHP); dep over godep, glide and govendor (Go). In some cases, such as dep in Go, there's no default repository. The manifest file includes information on package sources.
Gradle did not initially support locking but locking could be implemented using Netflix's Nebula Plugins project. This plugin inspired Gradle to support locking. While build.gradle specifies dependencies at the top level, module metadata files (.module, .pom or ivy.xml) must also be read.
In which languages are dependencies more often used?
A look of dependency counts in some popular languages. Source: Szulik 2018.
One study of 6.9 million versions of open source packages across 14 repository showed that NPM and CPAN have higher dependency counts; all others have 5 or less dependencies, on average. NPM and CPAN also have a larger spread. In general, higher the count, more complex is the dependency graph. While using third-party packages saves development time, there's also risks in terms of security, licensing and performance. Application health depends on all dependencies that we bring in.
Each repository contains packages of a particular language: NPM (JavaScript), CPAN (Perl), Rubygems (Ruby), Maven (Java), Packagist (PHP), Cargo (Rust), Atom (JavaScript), Pub (Dart), CRAN (R), PyPI (Python), NuGet (.NET/C#/F#/Visual Basic), Elm (Elm), Hex (Erlang), Dub (D). Libraries.io tracks open source packages across different repositories. Since Go language doesn't have a default repository, Go Search provides a handy search functionality.

Milestones

Mar
2002

Based on Project Object Model (POM), Maven 1.0-beta-2 is released for managing Java projects. Maven 1.x reaches end of life in 2014. As a rewrite of Maven 1.x, with extra functionality, Maven 2.0 is released in October 2005. Popular Maven plugins are also converted to the new version. Maven 2.0 can manage transitive dependencies.

Jul
2009

For managing dependencies for Java, including Android projects, Gradle v0.7 is released. Version v1.0 follows in June 2012. Version v4.8 is released in June 2018. Version v4.8 includes support for dependency locking.

Sep
2010

As a dependency manager for Ruby, Bundler V1.0.0 is released. Earliest release can be traced to 2009. Ruby community is credited with introducing the concept of dependency locking.

Oct
2010

Maven Dependency Diagrams from JetBrains' IntelliJ IDEA can flag version conflicts. Source: Bulenkov 2010.

While being compatible with Maven 2.x projects and plugins, Maven 3.0 is released. Among other things, it isolates project and plugin dependencies, and does better dependency resolution.

Mar
2011

Version v1.0.0rc9 of Node Package Manager (NPM) for NodeJS/JavaScript is released. Release v0.0.1 can be traced to January 2010.

Sep
2011

As a dependency manager for Xcode projects (Objective-C or Swift), CocoaPods 0.0.1 is released. Version 1.0.0 is released in May 2016. An alternative to CocoaPods is Carthage that trades configuration for convention.

Jul
2013

Version 1.0.0-alpha1 of Composer is released as a dependency manager for PHP. Version 1.0.0 is released in April 2016. Composer has been inspired by Node's NPM and Ruby's Bundler. Composer uses Packagist as its default repository but for internal/Intranet use, Satis is an alternative.

Sep
2014

To overcome the limitations of NuGet, Paket is released as a dependency manager for .NET. NuGet can be seen as a package manager. It doesn't handle transitive dependencies and selects the latest version when there are conflicts. Paket solves these problems of NuGet.

Aug
2015

With initial code commits from March 2014, Cargo version 0.4.0 is released as a dependency manager for Rust. As of June 2018, version 0.28.0 is the latest available.

Jun
2016

Steps taken by Yarn for managing dependencies. Source: Hanif 2017.

Version 0.2.0 of Yarn, is released for dependency management for NodeJS/Javascript. Version 1.0.0 is released in September 2017. For performance, Yarn caches downloaded packages and does downloads concurrently. It uses checksums to verify integrity of downloaded packages before installing them. It's inspired by Bundler, Cargo and NPM.

May
2017

Version 5.0.0 of NPM is released for NodeJS/JavaScript. This version introduces dependency locking. In July, version 5.1.0 is released to fix a problem that ignored changes to packages.json when lock file is present.

Jul
2017

For managing dependencies for Go, dep v0.2.0 is released.

Sample Code

javascript

// Source: https://packagist.org/
// File: composer.json (for PHP Composer)
// Example showing package information along with dependencies
// :dependencies are specified within "require"
// :version numbers use semantic versioning
{
    "name": "your-vendor-name/package-name",
    "description": "A short description of what your package does",
    "require": {
        "php": "^5.3.3 || ^7.0",
        "vendor/package": "1.3.2",
        "vendor/package2": "1.*",
        "vendor/package3": "^2.0.3"
    }
}

References

Article Stats

2274

Words

Authors

Edits

Chats

Likes

28K

Hits

Cite As

Devopedia. 2022. "Dependency Manager." Version 4, February 15. Accessed 2023-11-12. https://devopedia.org/dependency-manager

Contributed by
3 authors

Last updated on
2022-02-15 11:48:21

languages tools build

Dependency Manager

Discussion

Milestones

Sample Code

References

Further Reading

Article Stats

Cite As

See Also

Dependency Manager

Discussion

Milestones

Sample Code

References

Further Reading

Article Stats

Author-wise Stats for Article Edits

Cite As

See Also

Login