A few days ago, setuptools 50.0.0 release caused breakage to many projects. SecureDrop package builds was also broken. We use dh-virtualenv tool to build the packages. Initially, we tried to use the experimental build system from dh-virtualenv. We could specify the version of the setuptools to be installed in the virtualenv while creating it.

This approach worked for Xenial builds. As we are working to have proper builds on Focal (still work in progress), that was broken due to the above-mentioned change.

So, we again tried to use Python's venv module itself to create the virtual environment and use the wheels from the /usr/share/python-wheels directory to build the virtual environment. Which works very nicely on Xenial, but on Focal the default setuptools version is 44.0.0, which also failed to install the dependencies.

Now, we are actually getting the setuptools 46.0.0 wheel and replacing the build container's default setuptools wheel. The team spent a lot of time in debugging and finding a proper fix for the package builds. Hopefully, we will not get a similar breakage on the same kind of dependency error soon (the actual package dependencies are pinned via hashes).

A couple of weeks back, I gave a talk at PyCon US 2019, "Building reproducible Python applications for secured environments".

The main idea behind the talk is about the different kind of threats in an application which has dependencies (with regular updates) coming from various upstream projects, and also the final deployable artifact (Debian package in this case) needs to audit-able, and reproducible.

Before my talk on Saturday, I went through the whole idea and different steps we are following, with many of the PyPA (Python Packaging Authority) and other security leads in various organizations.

You can view the talk on Youtube. Feel free to give any feedback over email or Twitter.

Let me introduce rpm-macros-virtualenv 0.0.1 to you all.

This is a small set of RPM macros, which can be used by the spec files to build and package any Python application along with a virtualenv. Thus, removing the need of installing all dependencies via dnf/rpm repository. One of the biggest usecase will be to help to install latest application code and all the latest dependencies into a virtualenv and also package the whole virtualenv into the RPM package.

This will be useful for any third part vendor/ISV, who would want to package their Python application for Fedora/RHEL/CentOS along with the dependencies. But, remember not to use this for any package inside of Fedora land as this does not follow the Fedora packaging guidelines.

This is the very initial release, and it will get a lot of updates in the coming months. The project idea is also not new, Debian already has dh-virtualenv doing this for a long time.

How to install?

I will be building an rpm package, for now download the source code and the detached signature to verify it against my GPG key.

wget https://kushaldas.in/packages/rpm-macros-virtualenv-0.0.1.tar.gz
wget https://kushaldas.in/packages/rpm-macros-virtualenv-0.0.1.tar.gz.asc
gpg2 --verify rpm-macros-virtualenv-0.0.1.tar.gz.asc rpm-macros-virtualenv-0.0.1.tar.gz

Untar the directory, and then copy the macros.python-virtualenv file to the RPM macros directory in your system.

tar -xvf rpm-macros-virtualenv-0.0.1.tar.gz
cd rpm-macros-virtualenv-0.0.1/
sudo ***** macros.python-virtualenv /usr/lib/rpm/macros.d/

How to use?

Here is a minimal example.

# Fedora 27 and newer, no need to build the debug package
%if 0%{?fedora} >= 27 || 0%{?rhel} >= 8
%global debug_package %{nil}
%endif
# Use our interpreter for brp-python-bytecompile script
%global __python /opt/venvs/%{name}/bin/python3


%prep
%setup -q

%build
%pyvenv_create
%{__pyvenvpip3} install --upgrade pip
%pyvenv_build

%install
%pyvenv_create
%{__pyvenvpip3} install --upgrade pip
%pyvenv_install
ln -s /opt/venvs/%{name}/bin/examplecommand $RPM_BUILD_ROOT%{_bindir}/examplecommand

%files
%doc README.md LICENSE
/opt/venvs/%{name}/*

As you can see, in both %build and in %install, first we have to call %pyvenv_install, that will create our virtualenv. Then we are installing the latest pip in that environment.

Then in the %build, we are calling %pyvenv_build to create the wheel.

In the %install section, we are calling %pyvenv_install macro to install the project, this command will also install all the required dependencies (from the requirements.txt of the project) by downloading them from https://pypi.org.

If you have any command/executable which gets installed in the virtualenv, you should create a symlink to that from $RPM_BUILD_ROOT/usr/bin/ directory in the %install section.

Now, I have an example in the git repository, where I have taken the Ansible 2.7.1 spec file from Fedora, and converted it to these macros. I have build the package for Fedora 25 to verify that this works.