Python Executable (PEX) to the rescue

Background

In a recent project I had to deploy a python application in almost 100k hosts. Almost 50% of the hosts run Ubuntu-14 and rest of them run Ubuntu-18. We were also testing some Ubuntu-20 hosts to run the same application. Ubuntu-14 supports python2.7, Ubuntu-18 hosts support python3.6 and Ubuntu 20 hosts supports 3.8. Ubuntu 14 had an older python version (python 3.4) where running our application using python3 was challenging as some of the libraries were not getting installed. Upgrading the python3 version in Ubuntu-14 was not an option as there were many python applications running and upgrade may impact applications that were already using older python version. We generated debian package of our application and deployed them using puppet. The challenge was how to run the application using python-2 in Ubuntu-14, python-3 in ubuntu-18 and ubuntu-20.

Solution

We explored how to use PEX files to solve this problem. PEX are self contained virtual environment where we can install required libraries and dependencies. PEX does not depend on system python libraries. For example if application needs “requests” library, we can install “requests” library in PEX file itself. So PEX will work even if the library is not installed in system python. More details can be found here.

PEX generation process

1. We used jenkins job to generate debian packages. The job also uploaded the packages to respective artifact storages for Ubuntu-14 and Ubuntu-18 hosts. Initially the job uploaded the same debian package to both the storages.
2. We need to change the job in such a way that the current debian package gets uploaded only to Ubuntu-14 artifact storage. We will create a new job which will be spawned by the original job and the new job will create Ubuntu-18 python3.6 compatible artifact which supports PEX.
3. To generate PEX for a particular python version, we need to install that python version as the main python version in the environment. For example to generate PEX for python 3.6.9 we need to make sure that the python command points to python 3.6.9.
4. We can either create a python3 virtual environment or use python3 docker container to generate PEX.
5. We created a Dockerfilethat contains Ubuntu-18 as base image and installs python 3.6 in the container.

6. We copied python scripts and other related files from the host to the container using volume mapping.

7. Once the files were copied we executed the following command to generate the PEX file.

pex -f $PWD my_application -e my_package.my_application -r requirements.txt -o collectors/my_application.pex

8. The PEX file was packaged into a debian package and uploaded to Ubuntu-18 artifact repository.

Downside of PEX

PEX is extremely powerful tool to run python application in different environment without needing to install dependent libraries in the environment. But we can not see the content of PEX file as it is binary format. One of the advantage of using python application is that we can read the script that is installed in production and sometime we make changes in the installed script in the development environment and test. With PEX we loose this flexibility. In case we encounter any error, we need to generate PEX with the fix and test it again. PEX files sized are also bigger than python script. So we need to keep storage limitations in mind.

Conclusion

PEX is recommended only when we need to run a python application in heterogeneous environment and deploying dependent python libraries are not possible in all the environment.