Proper way to parse requirements file after pip upgrade to pip 10.x.x?

First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of “install dependencies” are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.

But everyone has different needs, that lead to different workflows. So with that said… There are 3 possibilities to handle this, depending on where you want your project’s package metadata to be written: pyproject.toml, setup.cfg, or setup.py.


Words of caution!

If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the “source distribution” (sdist) otherwise installation will fail, for obvious reasons.

These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.


pyproject.toml

[project]
# ...
dynamic = ["dependencies"]

[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt

setup.cfg

Since setuptools version 62.6 it is possible to write something like this in setup.cfg:

[options]
install_requires = file: requirements.txt

setup.py

It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.

It could more or less look like this:

#!/usr/bin/env python

import pathlib

import pkg_resources
import setuptools

with pathlib.Path('requirements.txt').open() as requirements_txt:
    install_requires = [
        str(requirement)
        for requirement
        in pkg_resources.parse_requirements(requirements_txt)
    ]

setuptools.setup(
    install_requires=install_requires,
)

Notes:

Leave a Comment