This tutorial is hosted at http://www.scotttorborg.com/python-packaging/
Feedback is welcome, please submit a pull request or email [email protected].
Tutorial on how to structure Python packages
Home Page: http://www.scotttorborg.com/python-packaging/
This tutorial is hosted at http://www.scotttorborg.com/python-packaging/
Feedback is welcome, please submit a pull request or email [email protected].
Is this guide outdated?
Could you please say something about wheel
? Even one sentence can be helpful.
Hi,
This issue (quite extensive) revolves around the lack of a .pypirc
file and lack of clarity around the fact that it is needed.
I found that the python-packaging docs don't mention this file (specifically, this page), so I got stuck at the upload step.
I used this python.org page, which does mention the file, to get un-stuck.
How can I solve this issue ? Thanks!
Collecting test
Downloading http://pypi.example.com/repository/2/test-0.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 20, in
File "/tmp/pip-build-EAllOQ/test/setup.py", line 11, in
long_description=readme(),
File "/tmp/pip-build-EAllOQ/test/setup.py", line 4, in readme
with open('README.md') as f:
IOError: [Errno 2] No such file or directory: 'README.md'
Thanks for these docs. I was searching for how to handle package data and it took me to http://python-packaging.readthedocs.io/en/latest/non-code-files.html. I followed that technique in one project before I realized there's a more automated way to include such files.
Including setuptools_scm as a setup_requires
declaration will invoke the setuptools file_finder
to automatically include any files found in the SCM (e.g. Git) repository. Thus, the developer doesn't have to maintain a separate manifest (one in a MANIFEST.in and another in the SCM repo) and run the risk of the two being inconsistent.
I believe non-code-files deserves at least a nod to file finders as a more automated solution to the problem of non-code-files.
Aloha,
"you’ll need to supply include_package_data=True to the setup() function."
/usr/local/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'include_package_data'
warnings.warn(msg)
It seems that (at least with python3) this option is no longer available.
hi storborg,
perhaps you should include a section on how to include submodules and ensure that they work
ie either specify the submodules specifically by name on the packages section or include
packages=setuptools.find_packages(),
in your setup.py
thanks
Hi, thanks for this great tutorial, I wanted to ask a question -
My python script which I have packed writes to a file, this file is not shipped with the package so in essence it is created on the fly for the user (that is when he runs the package.).
Now I want to know, how do I handle such a file?
Currently I have not specified it in my setup.py because it has not been shipped with the package, do I need to define it in setup.py and maybe ship it is an empty file?
Setup.py refers to "README.rst.example" but everything.rst refers simply to README.rst - if you go strictly by http://python-packaging.readthedocs.org/en/latest/everything.html it won't work because it expects the README.rst.example, which isn't called out as a needed file on the page.
I'd suggest just renaming README.rst.example to README.rst and updating the reference in setup.py accordingly.
(I assume it was renamed to keep it from rendering the readme on the github page for https://github.com/storborg/python-packaging/tree/master/funniest - an empty README.md if present should take priority in rendering for the purpose of how github's UI displays things - see github/markup#360 )
The last commit is 6 years old and so is the information's.
Your nice looking readthedocs version of that repo expose a lot of informations about python and packaging that are not only not up to date but very problematic.
Today python itself has a lot of very good resources where you can learn how to package something. There is no need for such a 3rd party documentation anymore. That is why I would say don't upgrade your repo because there is no need for this anymore.
It will save a lot of time if this outdated information's wouldn't exist anymore. I often have people pointing me to your readthedocs and I have to explain them why the advices from there are not up to date anymore.
Under the project description, there is a dedicated field for a URL. I suggest placing http://www.scotttorborg.com/python-packaging there so the tutorial is easy to find (and visit) from this repo.
Hi,
Thanks for your great tutorial!
At the end, though, I was thinking: is there somewhere I can go and download the source to funniest
so that I can use it as a template for my own package? Turns out, yes, there's this repository - but its not quite set up to do that. Would you mind splitting this into two repositories, so that funniest
could be forked, downloaded, and tested independently of the tutorial?
Would you please change basestring
to str
. The basestring
is no longer available in Python 3, and it was removed. Use str instead.
At the time of writing, Python 2.7 is EOL in under 10 months, but https://python-packaging.readthedocs.io/en/latest/ says:
At this time, this documentation focuses on Python 2.x only, and may not be as applicable to packages targeted to Python 3.x.
So as increasing numbers move to Python 3.x, it needs a sanity check to make sure it's not accidentally misleading people who miss that caveat.
Could you add an explicit statement about a requirements.txt file in the repo if your package relies on something besides the python core library? I can make the actual documentation update if I knew what the answer was.
In setup.py:
https://github.com/storborg/python-packaging/blob/master/funniest/setup.py
the url (line 20) is listed as follows:
url='http://github.com/storborg/funniest',
This repo does not exist, I believe it ought to be:
url='http://github.com/storborg/python-packaging',
The section Packages Not On PyPI recommends using the 'dependency_links' argument to setup().
According to the User Guide
Warning: Dependency links support has been dropped by pip starting with version 19.0 (released 2019-01-22).
Please update the Guide.
python3 setup.py register
is deprecated and as per the official guide is not recommended due to security issues. Similar issues with
python2 setup.py upload
The recommended way is to use twine to register and upload the package.
This was by far the best tutorial I've read on creating pip-compatible setups. However, the use of "cmd.py" in the examples could possibly shadow the standard library's cmd.py so I'd recommend changing it to something else in the locations it exists.
I am a maintainer of Python packages in openSUSE, and I am on my crusade of eliminating nose1 from our distribution. When I look at its repository on https://github.com/nose-devs/nose, the last release 1.3.7 was on 2 Jun 2015, and even the last commit on the master branch was on 4 Mar 2016. Moreover, nose is not compatible with Python >= 3.9.
So, it was with horror I was looking at testing.rst and see this:
The best way to get these tests going (particularly if you're not sure what to use) is Nose.
Please, do not recommend this wreck. python3 -munittest discover -v
works usually just fine, and if you have to use alternative test runner, then please recommend pytest
.
You might mention this option:
pip install git+https://github.com/myaccount/mypackage
or even
pip install git+https://github.com/myaccount/mypackage@version
where version
can be either a branch, a tag or a commit
I'm working on a homework assignment for our budding Python packagers, based on this repo.
We're using Python 2.7.13
built via spack.
The script sets up working directory and a "dev virtual environment" (perhaps not strictly necessary) the clones this repo and tries to run python setup.py test
.
The first time it runs it tries to install nose-cover3
twice and then complains about not being able to install nose
and exits. If I run python setup.py test
again it installs nose
and then things work as they should.
I discovered that if I reverse the order of the test dependencies in setup.py
then it succeeds, e.g.:
--- a/funniest/setup.py
+++ b/funniest/setup.py
@@ -26,7 +26,7 @@ setup(name='funniest',
'markdown',
],
test_suite='nose.collector',
- tests_require=['nose', 'nose-cover3'],
+ tests_require=['nose-cover3', 'nose'],
entry_points={
'console_scripts': ['funniest-joke=funniest.command_line:main'],
},
I'm not enough of a Python guru to understand what's going on. Perhaps nose-cover3
has messed up metadata or .... In any case it doesn't seem to be a bug here but I'm hoping you might be able to shed some light.
Below is a cut-and-paste of a sample session.
Thanks!
g.
[me@blah ~]$ # make a directory for the demo
[me@blah ~]$ DEMO_DIR=$(mktemp -t -d ${USER}-PYTHON_DEMO.XXXXX)
[me@blah ~]$
[me@blah ~]$ # set up a virtualenv for our dev work
[me@blah ~]$ # this might not be *required*, but it's safer when you're me...
[me@blah ~]$ V_DEV=${DEMO_DIR}/dev_virtenv
[me@blah ~]$ virtualenv ${V_DEV}
test deps.... SIGH...
python setup.py test
python setup.py test
New python executable in /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin/python2.7
Also creating executable in /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin/python
Installing setuptools, pip, wheel...done.
[me@blah ~]$ source ${V_DEV}/bin/activate
(dev_virtenv) [me@blah ~]$
(dev_virtenv) [me@blah ~]$ # clone the python packaging repo (w/ funniest project)
(dev_virtenv) [me@blah ~]$ PP=${DEMO_DIR}/python-packaging
(dev_virtenv) [me@blah ~]$ git clone https://github.com/storborg/python-packaging ${PP}
Cloning into '/tmp/me-PYTHON_DEMO.tNpOB/python-packaging'...
remote: Counting objects: 167, done.
remote: Total 167 (delta 0), reused 0 (delta 0), pack-reused 167
Receiving objects: 100% (167/167), 34.13 KiB | 0 bytes/s, done.
Resolving deltas: 100% (79/79), done.
(dev_virtenv) [me@blah ~]$ cd ${PP}/funniest
(dev_virtenv) [me@blah funniest]$
(dev_virtenv) [me@blah funniest]$ # install prerequisites
(dev_virtenv) [me@blah funniest]$ python setup.py develop
running develop
running egg_info
creating funniest.egg-info
writing requirements to funniest.egg-info/requires.txt
writing funniest.egg-info/PKG-INFO
writing top-level names to funniest.egg-info/top_level.txt
writing dependency_links to funniest.egg-info/dependency_links.txt
writing entry points to funniest.egg-info/entry_points.txt
writing manifest file 'funniest.egg-info/SOURCES.txt'
reading manifest file 'funniest.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'funniest.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/lib/python2.7/site-packages/funniest.egg-link (link to .)
Adding funniest 0.1 to easy-install.pth file
Installing funniest-joke script to /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin
Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest
Processing dependencies for funniest==0.1
Searching for markdown
Reading https://pypi.python.org/simple/markdown/
Downloading https://pypi.python.org/packages/29/82/dfe242bcfd9eec0e7bf93a80a8f8d8515a95b980c44f5c0b45606397a423/Markdown-2.6.9.tar.gz#md5=56547d362a9abcf30955b8950b08b5e3
Best match: Markdown 2.6.9
Processing Markdown-2.6.9.tar.gz
Writing /tmp/easy_install-f7q0Sx/Markdown-2.6.9/setup.cfg
Running Markdown-2.6.9/setup.py -q bdist_egg --dist-dir /tmp/easy_install-f7q0Sx/Markdown-2.6.9/egg-dist-tmp-80aaNi
zip_safe flag not set; analyzing archive contents...
Moving Markdown-2.6.9-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/lib/python2.7/site-packages
Adding Markdown 2.6.9 to easy-install.pth file
Installing markdown_py script to /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/bin
Installed /tmp/me-PYTHON_DEMO.tNpOB/dev_virtenv/lib/python2.7/site-packages/Markdown-2.6.9-py2.7.egg
Finished processing dependencies for funniest==0.1
(dev_virtenv) [me@blah funniest]$
(dev_virtenv) [me@blah funniest]$ # edit, test, edit, test, yay
(dev_virtenv) [me@blah funniest]$ # edit...
(dev_virtenv) [me@blah funniest]$ # install test deps fails to install nose the first time around...
(dev_virtenv) [me@blah funniest]$ # do it again and it works. Sigh...
(dev_virtenv) [me@blah funniest]$ # or swap order of 'nose' and 'nose-cover3' in test deps.... SIGH...
(dev_virtenv) [me@blah funniest]$ python setup.py test
running test
Searching for nose-cover3
Reading https://pypi.python.org/simple/nose-cover3/
Downloading https://pypi.python.org/packages/f0/17/8c55242e86830a006bbaa0463f4a1da44f332ef7cd5a402f459c8dbaaf84/nose-cover3-0.1.0.tar.gz#md5=82f981eaa007b430679899256050fa0c
Best match: nose-cover3 0.1.0
Processing nose-cover3-0.1.0.tar.gz
Writing /tmp/easy_install-81NDHC/nose-cover3-0.1.0/setup.cfg
Running nose-cover3-0.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-81NDHC/nose-cover3-0.1.0/egg-dist-tmp-V4JbFB
warning: no previously-included files matching '*.pyc' found anywhere in distribution
creating /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Extracting nose_cover3-0.1.0-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs
Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Searching for nose
Downloading https://pypi.python.org/packages/f0/17/8c55242e86830a006bbaa0463f4a1da44f332ef7cd5a402f459c8dbaaf84/nose-cover3-0.1.0.tar.gz#md5=82f981eaa007b430679899256050fa0c
Best match: nose cover3-0.1.0
Processing nose-cover3-0.1.0.tar.gz
Writing /tmp/easy_install-OZM0ym/nose-cover3-0.1.0/setup.cfg
Running nose-cover3-0.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-OZM0ym/nose-cover3-0.1.0/egg-dist-tmp-OmgVZ0
warning: no previously-included files matching '*.pyc' found anywhere in distribution
removing '/tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg' (and everything under it)
creating /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Extracting nose_cover3-0.1.0-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs
Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Traceback (most recent call last):
File "setup.py", line 34, in <module>
zip_safe=False)
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-fiaifkmalam23l42s4lyqg2kithvppfi/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-fiaifkmalam23l42s4lyqg2kithvppfi/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/python-2.7.13-fiaifkmalam23l42s4lyqg2kithvppfi/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/setuptools/command/test.py", line 199, in run
installed_dists = self.install_dists(self.distribution)
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/setuptools/command/test.py", line 195, in install_dists
tr_d = dist.fetch_build_eggs(dist.tests_require or [])
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/setuptools/dist.py", line 375, in fetch_build_eggs
replace_conflicting=True,
File "/sc1/apps/spack/v0.3.0/opt/spack/linux-centos7-x86_64/gcc-5.4.0/py-setuptools-35.0.2-xpv4en5bcykcoeorgv5ym3sklzpbow6u/lib/python2.7/site-packages/pkg_resources/__init__.py", line 854, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'nose' distribution was not found and is required by the application
(dev_virtenv) [me@blah funniest]$ python setup.py test
running test
Searching for nose-cover3
Best match: nose-cover3 0.1.0
Processing nose_cover3-0.1.0-py2.7.egg
Using /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose_cover3-0.1.0-py2.7.egg
Searching for nose
Reading https://pypi.python.org/simple/nose/
Downloading https://pypi.python.org/packages/58/a5/0dc93c3ec33f4e281849523a5a913fa1eea9a3068acfa754d44d88107a44/nose-1.3.7.tar.gz#md5=4d3ad0ff07b61373d2cefc89c5d0b20b
Best match: nose 1.3.7
Processing nose-1.3.7.tar.gz
Writing /tmp/easy_install-LokDOS/nose-1.3.7/setup.cfg
Running nose-1.3.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-LokDOS/nose-1.3.7/egg-dist-tmp-YF_UWz
no previously-included directories found matching 'doc/.build'
creating /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose-1.3.7-py2.7.egg
Extracting nose-1.3.7-py2.7.egg to /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs
Installed /tmp/me-PYTHON_DEMO.tNpOB/python-packaging/funniest/.eggs/nose-1.3.7-py2.7.egg
running egg_info
writing requirements to funniest.egg-info/requires.txt
writing funniest.egg-info/PKG-INFO
writing top-level names to funniest.egg-info/top_level.txt
writing dependency_links to funniest.egg-info/dependency_links.txt
writing entry points to funniest.egg-info/entry_points.txt
reading manifest file 'funniest.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'funniest.egg-info/SOURCES.txt'
running build_ext
test_basic (funniest.tests.test_command_line.TestCmd) ... ok
test_is_string (funniest.tests.test_joke.TestJoke) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.030s
OK
setup(
...
test_suite='nose.collector',
tests_require=['nose'],
)
...
$ python setup.py test
If you actually try this, you'll get
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
(from pypa/setuptools#1878). There's a long thread (pypa/setuptools#931) where the PyPA devs talk about their plan to drop setup.py test
entirely, apparently in favour of expecting you to tox
instead. Except the tox maintainer told me that tox
isn't really meant to be used that way.
Anyway, in the middle of all this confusion pypa/setuptools#1684 (comment) claims
I don't think we've ever tried to disguise this. For several years now we have been actively warning people not to invoke setup.py directly at every opportunity, and we've been quite explicit that the end goal is for setuptools to lose as many of its extraneous features as possible and become a standard build tool.
but I'd never seen any of those warnings. You, as one of the top hits for "python packaging" was also unaware. So I don't really know whose advice to trust. What do you think should be recommended?
The source code and tutorial are both available online, but the default copyright in most countries forbid people from using it legally
Add a license to the repository and all it's files and other people can use it legally.
Because it is not really source code it shouldn't be a program license like GPL.
My advice:
If you want help in implementing one or another, tell me. I'm happy to help!
I think this project brings more confusion than benefit.
If the upstream docs are missing something, then please talk to the upstream.
According to this page, the guide is eight years old: https://python-packaging.readthedocs.io/en/latest/about.html
Unfortunately python-packaging is the first match if you google for "setup.py dependencies".
Instead of more documentation, we need a clear central place for the python packaging docs.
tests [..] should be placed in a submodule of funniest. so that they can be imported, but won't pollute the global namespace.
There's a good case to be made that tests should not live inside your package, but instead inside a separate top-level tests/
. This is how one of the PyPA core devs does it. This is also what https://packaging.python.org/tutorials/packaging-projects/ (which makes sense, it's written by the same group). The reason for this is it lets you cull tests from your binary distribution, since people installing from binary almost never care to run your tests or have them take up space -- though you need to go out of your way to make sure to re-include them in your sdist
, but that's a one-line addition.
https://stackoverflow.com/questions/61151/where-do-the-python-unit-tests-go covers the three main options:
The current recommendation in this doc is for 3. PyPA thinks you should use 2. What are your thoughts?
It would be really cool to have a little section about hosting your sphinx generated site hosted through gh_pages!
Hi!
Im following the tutorial and copied the code, but the
python setup.py test
seems like not detecting the test.
I found the solution here:
https://stackoverflow.com/questions/17001010/how-to-run-unittest-discover-from-python-setup-py-test
In which the test_suite in the setup.py file is changed to
test_suite="tests",
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.