Making a Python Package for PyPI - Easy Steps
This blog will give you step by step instructions on how to setup a python project so that you can upload it to PyPI and make it distributable and installable via pip.
First, here are the things you need to know to follow these instructions:
- You are able to code in python
- You are familiar with git
- You know about unit tests
It also helps if you have an actual need to share a python solution with others such as a library you wrote, or a CLI tool, or a game etc.
We will use the free for open source github and travis services during this process. So first step is to make sure you have a github account and you use that account to login travis as well. You will also need to be a registered user with PyPI, go ahead and open an account in each of these sites if you do not already have them.
Go to your github site and create a new project repository. As eventually we want to upload this project to PyPI it is important that the repository name is unique. Something like sample_youruserid_date. For a real project you should search PyPI beforehand to see if your project name is taken already.
Do not initialize project repository with README or license documentation as we will do those later with the cookiecutter tool.
My repository for this article will be called sampleotuk20180511
It is a good python practice to have a dedicated virtual environment for each project. There are different tools to achieve the same end result for this step. I use conda which is part of anaconda. If you have conda, execute the following in command line to create a new virtual environment called sampleVE:
conda create -n sampleVE --python=3.6
conda activate sampleVE
If you instead use virtualenv the command line is as follows:
virtualenv -p /usr/bin/python3 -v ~/sampleVE
The above can be done by any other tool, the point is to have a virtual environment for our sample project and activate it by the end of this step.
Now that we are in activated virtual environment, let’s use pip to install any library/tool we need for the project.
pip install cookiecutter flake8 tox twine
Any other packages you need to use just do the same.
We will use cookiecutter to create a project scaffold with as much boilerplate as possible. Go to the parent directory of your projects and run
Here the argument to cookiecutter is a reference to a python project template available on the web. Later you can replace that reference with your own template once you have one. This command will start asking you a series of questions regards to your project and yourself. Make sure the project slug matches the Github repository name.
cookiecutter gh:audreyr/cookiecutter-pypackage full_name [Audrey Roy Greenfeld]: OtukK. email [email@example.com]: firstname.lastname@example.org github_username [audreyr]: otuk project_name [Python Boilerplate]: Sample Project project_slug [sample_project]: sampleotuk20180511 project_short_description [Python Boilerplate contains all the boilerplate you need to create a Python package.]: pypi_username [otuk]: otuk version [0.1.0]: 0.0.1 use_pytest [n]: y use_pypi_deployment_with_travis [y]: y add_pyup_badge [n]: n\nSelect command_line_interface: 1 - Click\n2 - No command-line interface Choose from 1, 2 : create_author_file [y]: Select open_source_license: 1 - MIT license 2 - BSD license 3 - ISC license 4 - Apache Software License 2.0 5 - GNU General Public License v3 6 - Not open source\nChoose from 1, 2, 3, 4, 5, 6 : 1
This will generate the directory structure we need for the project development and packaging.
. ├── AUTHORS.rst ├── CONTRIBUTING.rst ├── docs │ ├── authors.rst │ ├── conf.py │ ├── contributing.rst │ ├── history.rst │ ├── index.rst │ ├── installation.rst │ ├── make.bat │ ├── Makefile │ ├── readme.rst │ └── usage.rst ├── HISTORY.rst ├── LICENSE ├── Makefile ├── MANIFEST.in ├── README.rst ├── requirements_dev.txt ├── sampleotuk20180511 │ ├── cli.py │ ├── __init__.py │ └── sampleotuk20180511.py ├── setup.cfg ├── setup.py ├── tests │ └── test_sampleotuk20180511.py └── tox.ini
It is time for making this directory a git repository and syncing it with your github repository.
git init git remote add origin https://github.com/otuk/sampleotuk20180511.git git add . git commit -m"initial commit" git push --set-upstream origin master
After the above steps go to your github site and verify your directory is visible there.
Now you are ready to your code, and implement the great idea. You will do that inside the “sampleotuk20180511” directory above. You could use the sampleotuk20180511.py or add new .py files with your code. For this exercise we will implement a simple say_hi function inside the sampleotuk20180511/sampleotuk20180511.py file.
def say_hi(name): """ simple method to try out pypi packaging """ return "hi " + name
Step 6: Write some units tests for your code.
We will only have a single test that we will add to the end of the tests/test_sampleotuk20180511.py file.
def test_say_hi(): """ test the only method """ out = sampleotuk20180511.say_hi("otuk") assert("hi otuk" == out)
At this point, you can run py.test within the sampleotuk20180511 directory from the command line and the tests should pass, if there is any problem fix them. We will do more though. We will verify the package runs also with other python versions. Take a look at the tox.ini file and add/delete any python versions you would like to support. I removed all python 2 from mine.
[tox] envlist = py34, py35, py36, flake8 [travis] python = 3.6: py36 3.5: py35 3.4: py34
Now you can run
This will execute pytests in different environments and also execute flake8 or any other commands specified in the tox.ini file.
summary ERROR: py34: InterpreterNotFound: python3.4 py35: commands succeeded py36: commands succeeded flake8: commands succeeded
Here my test failed for python 3.4 because I do not have it on my local dev. But next step will take care of that when we check in and run this in travis.
(if you are getting module not found errors while trying to py.test, run thus from your tests directory)
If Locally your code logic passed the unit tests, now you are ready to do tests in travis’s environment.
Go to travis site first and “sync account” for your profile. This will bring all your github projects to travis. By default the repositories are not turned ON for building them in travis, turn it on for your sample project. For future reference you can have more control on travis build using the settings, for now defaults are OK.
Now you can add and commit changes. Push them changes to start the travis build like shown below.
git add . git commit -m"initial tests passed" git push
When you push to github, the travis integration will automatically pick the changes and start the build. You can watch the build online on travis, it will show you the duration as it installs requirements, builds project and executes tests.
If all is OK travis will report the build is good, tests passed, and will list which environments tests passed.
We can now build a distribution package, go back to your project directory and from command line run:
python setup.py sdist bdist_wheel
This will create 2 directories called build and dist. These directories are not part of your repository but just distribution related files.
Step 10: Now it is time to use twine and upload these files to PyPI. But it may be a good idea at least for the first time to upload first to PyPI’s test environment. You will need to get an userid for this as well. Do it, because we are almost there.
Execute this within the sample project:
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
This will ask for your userid/name, if all the information is correct you can now go and see your package under your name online, in the test repository.
OK, now the drum roll, execute the following to upload to PyPI:
twine upload dist/*
And you are done!
You can now go and install your package on any machine, on any virtual environment and globally, simply using pip:
pip install YOUR_PACKAGE_NAME
Later if you want to make a change and push a new version use install and use bumpversion package.
pip install bumpversion
Do your changes, pass unit tests, travis builds as above and when you are ready to upload and share the new version of your project do:
bumpversion minor setup.py && python setup.py sdist bdist_wheel
And upload using twine again:
twine upload dist/*
You should be able to see your new version and version history on PyPI.