Python Dependency Management: requirements.txt with pipdeptree

Managing Python project dependencies is crucial for reproducibility and collaboration. The conventional way to list project dependencies is using a requirements.txt file. While pip freeze is a common tool for this, it lists all installed packages (including transitive dependencies) in the current environment, which can lead to bloated requirements.txt files and make it difficult to discern direct project requirements.

pipdeptree is a tool that displays installed Python packages in a dependency tree format. It’s particularly useful for generating a requirements.txt that lists only your top-level project dependencies, making dependency management cleaner and more explicit.

1. Installing pipdeptree

Install pipdeptree into your Python environment (preferably within a virtual environment):

pip install pipdeptree

2. Generating requirements.txt with Top-Level Dependencies

To create a requirements.txt file containing only the packages you explicitly installed (your top-level dependencies), you can combine pipdeptree with grep.

pipdeptree --warn silence | grep -E '^\w+' > requirements.txt

Command Breakdown:

• pipdeptree: Generates the dependency tree of installed packages.
• --warn silence: Suppresses warnings from pipdeptree, leading to cleaner output.
• |: Pipes the output of pipdeptree to the grep command.
• grep -E '^\w+': Filters the output to include only lines that start with a word character (\w, which includes letters, numbers, and underscore). This effectively captures only the top-level packages (which are not indented in pipdeptree’s output) and their versions.
• > requirements.txt: Redirects the filtered output to a file named requirements.txt.

This command produces a requirements.txt file that is often much smaller and more relevant to your project’s direct needs, improving clarity and reducing unnecessary dependency declarations.

3. Best Practices and Alternatives

• Virtual Environments: Always generate your requirements.txt from within an activated virtual environment to ensure you’re only capturing the dependencies relevant to that specific project.
• Version Pinning: It’s a good practice to pin exact versions of your dependencies (e.g., package==1.2.3) for reproducible builds. pipdeptree and pip freeze naturally provide this.
• Advanced Tools: For more complex dependency management scenarios (e.g., managing development vs. production dependencies, handling lock files), consider tools like:
  • pip-tools: Automates the compilation of requirements.txt from requirements.in files, handling pinning and transitive dependencies.
  • Poetry / Rye / PDM: Modern Python packaging and dependency management tools that abstract away much of the manual requirements.txt management and provide robust dependency resolution and lock file generation.