Python Distributions and Packaging

The Python Software Foundation does regular releases of Python, which we call the standard distribution. This is a good starting point for beginners. However, there are alternatives distributions. Developers may opt for these alternatives to suit their specific requirements or simplifying the installation process.

Third-party packages are usually distributed via a central repository called the Python Package Index (PyPI). For developers who wish to release their software to the Python community, Python provides many tools to package their software in a standard way and release the same on PyPI. As of February 2018, PyPI has over 130,000 packages.

Discussion

  • What's the standard Python distribution and what does it include?

    The Python Software Foundation (PSF) releases the standard distribution of Python periodically at Python.org. This includes Python 3.x and Python 2.x until the latter is retired. Installables are available for Windows and Mac OS X. It's also possible to install Python on Windows using a package manager such as Chocolatey.

    For Linux/UNIX, source code is available although most of them come pre-installed with Python that can also be updated with their respective package managers. Python is also available for other OS such as IBM i 7.1+, iOS, VMS, HP-UX, and more; but these lag behind the official release.

    The standard implementation of Python is in C and is named CPython. This includes implementation of the entire standard library.

  • Are there implementations of Python other than CPython?

    Yes. Implementation refers to that of the interpreter itself that takes Python code and converts into bytecode. With CPython, the interpreter is written in C. This enables easy interfacing of C code with Python code.

    Likewise, we have IronPython for .NET and Jython for JVM. PyPy uses a JIT compiler for faster execution. Stackless Python augments CPython to support microthreads. MicroPython is a slim version of CPython that can run on microcontrollers.

    These alternative implementations lag CPython releases. For example, as of February 2018, IronPython is not available for Python 3 and its Python 2 release hasn't been updated since December 2016. Jython is available only for Python 2. PyPy is available for both 2 and 3 though lagging the latest version of CPython.

  • What are some alternatives to the standard Python distribution?
    What Python(x,y) includes in its distribution. Source: Davar 2015.
    What Python(x,y) includes in its distribution. Source: Davar 2015.

    There are a number of alternative Python distributions. ActivePython is a commercial distribution that bundles many useful third-party packages along with the standard library. There's also a free Community Edition. Anaconda and Enthought are two distributions customized for scientific computing and analysis. While Anaconda is free, Enthought is commercial. Intel offers an enhancement of Anaconda.

    Python(x,y) is based on Qt and Spyder. This may be good place to start for MATLAB/IDL/C/C++/Fortran programmers who wish to switch to Python. Other alternatives include WinPython, Conceptive Python, PyIMSL, and more. Stackless Python is limited by the heap and not the stack for recursion, and it consumes less memory.

    The purpose of all of these distributions is to simplify the process of installation and package updates. Commercial distributions also offer support such porting Python to an exotic platform or maintaining Python 2.x for legacy codebases.

    What if you wish to use Python without bothering with any installation at all? PythonAnywhere allows you to code and execute on the cloud from a web browser. It's ideal for teachers and students.
  • What's the purpose of PyPI?
    The PyPI logo. Source: Pypi.org 2018.
    The PyPI logo. Source: Pypi.org 2018.

    PyPI serves as a central repository for sharing third-party packages. Developers can go to PyPI and search for packages that they might find useful in their applications. If source code is open (in sites outside PyPI), they can also improve the package. This is often a better approach than developing something from scratch. Note two variants of PyPI:

    • Production site: Main download repository, powered by a codebase called Warehouse.
    • Test site: Site where developers can test their packaging before releasing to production site.

    While it's possible to download a package manually (as source or binary), build and install the same, having packages on PyPI makes the process easier. Packages are easier to find. We can use a standard tool such as pip to manage packages and pip automatically knows where to look for packages.
  • As a Python user, how can I install a package?

    The tool pip can be used to install a package from PyPI. It replaces an earlier tool named easy_install. If pip itself is missing on your system, use system tools to install the same; or download get-pip.py and execute python get-pip.py. While pip is sufficient to install pre-built binaries, to build from source you will also need setuptools and wheel, which can installed with the command pip install setuptools wheel.

    If you're working on multiple Python projects, each on a different version of Python, then it's recommended to set up virtual environments and install required packages within each environment. Run pip install virtualenv and then create/activate it using virtualenv myproj; source myproj/bin/activate. It's common to list all project requirements in a file such as requirements.txt and then install all of them by running pip -r requirements.txt. This file typically mentions each package name followed by a suitable version in a format specified in PEP 440.

  • I don't find pip suitable for my workflow. What are my options?
    pyenv could be used to manage different environments. Source: Google Cloud Platform 2017.

    An easier approach to using pip and virtualenv separately, or updating requirements.txt independently, is to use the package pipenv, which manages and tracks dependencies using Pipfile and Pipfile.lock files.

    If you wish to create custom workflows, pip-tools is recommended. Other alternatives include hatch and poetry.

  • What's the definition of a package, packaging and distribution?

    Package is a bundle of one or more Python modules that can be shared, downloaded and installed. A package is really nothing more than a folder that contains its module files (and optionally sub-packages) and a __init__.py. The process and tools used in making and uploading packages is called packaging. The term distribution is usually reserved for a Python distribution and must not be confused for package distribution.

  • What are Eggs and Wheels?

    Both are built distribution formats for packages. Wheels have replaced Eggs as the preferred format. Installer pip supports only Wheels while Eggs can be installed with the older easy_install tool. Wheels don't include compiled Python files. In a built distribution, files and metadata are included. The files are simply moved to correct locations while installing. There's no separate build step before installation.

    Python packaging specifies how to express dependencies and integration in the database of Python distributions. Wheels implements these but not Eggs. Wheels have a richer naming convention. Their binary format is standardized. For these reasons, Wheels are preferred.

    There are three types of Wheels:

    • Universal Wheel: Project is pure Python and natively supports both Python 2 and 3.
    • Pure Python Wheel: Project is pure Python and does not natively supports both Python 2 and 3.
    • Platform Wheel: Project contains compiled extensions.

    Since pip will prefer a wheel over source distribution, avoid making universal wheels if your project contains extensions. If you're not providing the source and your project is not pure Python, then multiple wheels will need to be provided for every platform that you wish to support.
  • Can you guide me in packaging and distributing my own package?
    An example file structure of a project to be packaged. Source: Feng 2017.
    An example file structure of a project to be packaged. Source: Feng 2017.

    The following are needed and can be installed using pip:

    • setuptools: This extends the older distutils. The project specification file is setup.py, which can further be controlled with setup.cfg.
    • wheel: This enables creation of wheels.
    • twine: It's possible to upload your package to PyPI by executing setup.py (register and upload). However, Twine is preferred. This enables you to upload your package to PyPI securely. Control of where to upload can be specified in file .pypirc in your home folder.
    • tox: This enables local installation and testing of your package in different environments. Configuration is usually given in a file named tox.ini. A similar testing can be done on the cloud using a service such as Travis CI.

    The command python setup.py clean --all sdist bdist_wheel will perform the build: sdist for source distribution and bdist_wheel for creating wheels. This will create two folders build/ and dist/. The former contains temporary files used during the build process. The latter contains the final source archive (*.tar.gz) and/or binary wheels (*.whl).

Milestones

2000

Python adds disutils to the standard library as a cross-platform build and distribution system. A replacement to this called disutils2 was worked on during 2009-2012 but this didn't make it into Python 3.3.

2003

PyPI goes online.

2004

Package installer easy_install is released as part of setuptools. Problem with easy_install is that it isn't easy to uninstall packages.

2008
Comparing package installers pip and easy_install. Source: Python Packaging 2018.
Comparing package installers pip and easy_install. Source: Python Packaging 2018.

Package installer pip is released as an alternative to easy_installer. This uses Wheel rather than Egg as the built distribution format. It's possible to use this to save package requirements to a file (pip freeze > reqs.txt) and recreate the same environment on another system (pip install -r reqs.txt). Wheel itself is published later in 2012-13.

Feb
2011

The Python Packaging Authority (PyPA) is created. It's job is to drive standardization, maintain tools and documentation related to Python packaging. It recognizes that the Python packaging and distribution infrastructure needs improvements.

Jul
2012

Anaconda 0.8.0. In February 2018, Anaconda 5.1.0 is released.

Sep
2012

WinPython v0.6 is released. In November 2017, WinPython 3.6.3 is released.

Mar
2014

Python 3.4 is released and it includes pip installer by default.

Apr
2018

Kenneth Reitz releases version 11.10.1 of pipenv at PyPI. Meanwhile, Warehouse replaces legacy code to power PyPI. This means that requests to pypi.python.org are redirected to pypi.org and they both share the same data store.

References

  1. ActiveState. 2018. "Python 2.7." Accessed 2018-02-20.
  2. Anaconda Docs. 2018. "Release Notes." Accessed 2018-02-25.
  3. Coghlan, Nick. 2013. "PEP 440 -- Version Identification and Dependency Specification." Python.org. March 18. Updated 2014-08-22. Accessed 2018-02-20.
  4. Das, Anwesha. 2017. "How to upload a package in PyPI using twine?" July 11. Accessed 2018-02-25.
  5. Davar, Gabi. 2015. "Python(x,y) - the scientific Python distribution." November. Accessed 2018-02-20.
  6. Feng, Jie. 2017. "A Simple Guide for Python Packaging." Little Big Programming. February 19. Accessed 2018-02-25.
  7. Google Cloud Platform. 2017. "Which Python Package Manager Should You Use?" YouTube, December 21. Accessed 2018-05-29.
  8. Henschel, Lacey Williams. 2018. "Why Python devs should use Pipenv." Opensource.com, February 28. Accessed 2018-05-29.
  9. IronPython. 2018. "IronPython." Accessed 2018-02-20.
  10. Jython.org. 2018. "Jython: Python for the Java Platform." Accessed 2018-02-20.
  11. Murray, R. David. 2018. "What’s New in Python 3.4." Python Docs, V3.6.4. Accessed 2018-02-20.
  12. Pieters, Martijn. 2013. "Python vs Cpython." StackOverflow. June 16. Updated 2017-12-19. Accessed 2018-02-20.
  13. PyPA. 2018a. "Packaging History." February 1. Accessed 2018-02-25.
  14. PyPA. 2018b. "PyPA Goals." February 1. Accessed 2018-02-25.
  15. PyPI. 2017. "wheel 0.30.0." September 10. Accessed 2018-02-25.
  16. PyPI. 2018a. "PyPI - the Python Package Index." Accessed 2018-02-23.
  17. PyPI. 2018b. "pipenv." Accessed 2018-05-29.
  18. PyPy.org. 2018. "PyPy." Accessed 2018-02-20.
  19. Pypi.org. 2018. "Pre-production PyPI." Accessed 2018-02-20.
  20. Python Docs. 2018a. "The Python Standard Library." V3.6.4. Accessed 2018-02-20.
  21. Python Packaging. 2018a. "Installing Packages." Accessed 2018-02-20.
  22. Python Packaging. 2018b. "pip vs easy_install." Accessed 2018-02-25.
  23. Python Packaging. 2018c. "Glossary." Accessed 2018-02-25.
  24. Python Packaging. 2018d. "Packaging and Distributing Projects." Accessed 2018-02-25.
  25. Python Packaging. 2018e. "Managing Application Dependencies." May 29. Accessed 2018-05-29.
  26. Python Packaging. 2020. "Using TestPyPI." July 22. Accessed 2020-07-22.
  27. Python Wiki. 2013. "Stackless Python." January 29. Accessed 2018-02-20.
  28. Python Wiki. 2018. "Warehouse roadmap." February 20. Updated 2020-04-28. Accessed 2020-07-22.
  29. Python.org. 2018a. "Python Downloads." Accessed 2018-02-20.
  30. Python.org. 2018b. "Download Python for Other Platforms." Accessed 2018-02-20.
  31. Python.org. 2018c. "Alternative Python Implementations." Accessed 2018-02-20.
  32. Rahman, Tasdik. 2015. "Submitting python package to pypi." November 10. Accessed 2018-02-20.
  33. WinPython GitHub. 2017. "WinPython Releases." November 10. Accessed 2018-02-25.
  34. Yegulalp, Serdar. 2016. "Intel's Python distribution turbocharges data science." InfoWorld. September 8. Accessed 2018-02-20.
  35. Yegulalp, Serdar. 2017. "How to get started with Python." InfoWorld. July 19. Accessed 2018-02-20.

Further Reading

  1. Yegulalp, Serdar. 2017. "How to get started with Python." InfoWorld. July 19. Accessed 2018-02-20.
  2. Tiwari, Shantnu. 2014. "Stop struggling with Python on Windows." Python for Engineers. July 19. Updated 2014-07-24. Accessed 2018-02-20.
  3. Rahman, Tasdik. 2015. "Submitting python package to pypi." November 10. Accessed 2018-02-20.
  4. Ziadé, Tarek. 2012. "Python Packaging." July 7. Accessed 2018-02-20.
  5. PEAK. 2008. "The Quick Guide to Python Eggs." November 15. Accessed 2018-02-25.
  6. Python Packaging. 2018d. "Python Packaging User Guide." Accessed 2018-02-20.

Article Stats

Author-wise Stats for Article Edits

Author
No. of Edits
No. of Chats
DevCoins
Avatar of user mohanjo mohanjo
4
0
2129
Avatar of user arvindpdmn arvindpdmn
3
0
74
DevCoins due to articles, chats, their likes and article hits are included.
1629
Words
2
Authors
7
Edits
1
Chats
5
Likes
16K
Hits

Cite As

Devopedia. 2020. "Python Distributions and Packaging." Version 7, July 22. Accessed 2023-11-12. https://devopedia.org/python-distributions-and-packaging
Contributed by
2 authors

Last updated on
2020-07-22 12:42:25