Roll tests into the urllib3 package about urllib3 HOT 8 CLOSED

Is it unreasonable to expect someone who is going to be debugging the library to fetch the source of the library?

I have mixed feelings about this. I generally would like to avoid distributing tests and other dev-only content with the user-targeted package. Especially since it's not uncommon for test data to get rather large and weird dependencies. For instance, we wouldn't to force people to install eventlet just to install urllib3, but I'd say it's not unreasonable to have this requirement for developers of urllib3.

from urllib3.

I'm really 👎 on this. There's little value to including the tests within the package, and a considerable amount of new hassle that actual developers and new contributors would have to go though.

from urllib3.

Sorry if my comment seems blunt :). I know that distribute recommends that you ship tests with your packages, but I've personally never sipped that coolaid. I always feel like it's a path hack.

from urllib3.

@shazow — Someone debugging the library must grab the source, I agree. But I was imagining someone who might not even have the expertise to do any debugging, but wants to answer the question “does this work on my platform at all?” and my own libraries are easier to debug if I can answer a new bug report on an unfamiliar platform with “could you run python -m unittest discover ephem” and let me know if it even passes all tests on your platform?”

But I completely agree that including the tests would not be a good step if it increased the number of dependencies that the library requires for install. Only include the tests if either (a) the dependencies can be removed through simpler tests or (b) if you are willing to throw a little code into test/__init__.py that tries a few imports and can exits with an error “the urllib3 tests require A and B and C, please install those and try again.” If neither is possible, then leave the tests out and keep the dependencies lean.

@kennethreitz — I am not sure what you mean by “path hack” but, so far as I understand the mind of Tarek, I see the official recommendation about shipping tests as having two foundations: first, as I mentioned above, that anyone with the package installed can, as part of a bug report, note whether the tests pass in their environment, and that they can do so successfully even if they are new to Python and only know about package installation (but maybe have not yet figured out what on earth version control is); and, second, that having ad-hoc top-level packages sitting in the repository as they are today (like “test” and “dummyserver”) is not only a bit odd, but is fragile because you have to be sitting in that directory — or at least loading the package from that directory — for them to be visible. For many Python developers I know, the procedure these days is to spin up a virtualenv for each project, and if they “pip install -e urllib3/” to activate the project in their virtualenv, then they are left with no way to run the tests without cd'ing over to its particular directory.

I will leave aside the crazy theory of Christian Theune that successful tests only prove that the package works in the presence of its tests, and that removing them at install time means that you are deploying an untested state that might not work, because I still think his point of view is a bit extreme. :)

Please note that my long-windedness here is not because I think I should be dictating a new approach to someone else's important and successful project! The paragraphs are long just because these emerging best-practices are ones that are still emerging, and your very specific questions were a great chance for me to re-think, in my own mind, the rationales behind the practices to see if they hold up.

from urllib3.

@brandon-rhodes Are these best-practices emerging or were they written down in an age when the Python and the Open Source community were a very different beast?

I cannot recall a single time when I ran (or desired to run) python -m unittest discover <somepackage>, to be honest I didn't even know that existed, and I've been developing in Python for a fairly long time now. On the other hand, there have been many dozens of occasions when I Googled <somepackage> source, copied a git or hg url, pasted it in my terminal and ran the tests.

I've never considered that some package I'm using may not work for my platform while using Python. SSL-related things won't work if you compiled your Python without SSL, but you'll get a helpful error when that happens. I don't think there's anything else that may fail on foreign Pythons where the rest of Python and its standard library do succeed. I can't speak the same for the tests (eventlet/gevent won't work on all platforms).

Perhaps I'm living in an odd developer bubble but this "best practice" has never been applicable to me so I'm having a hard time relating to it. :)

from urllib3.

One more note to bring this back:

I don't believe that it's okay for a mainstream package to fail on any mainstream platform. If there are tricky C-modules included optimized for specific platforms and you need to test that it'll work on yours and it's not something you can do during import, then I completely agree with you. But I don't think this is the case for urllib3.

from urllib3.

First: I completely agree that mainstream packages should not fail on mainstream platforms. No argument there. :)

Second: they sometimes, nevertheless, do fail. Core committer Armin Ronacher, for example, has been having problems lately (see his tweets and posts) with well-tested cross-platform code breaking because of novel encodings that his users have configured as their system default. But, of course, that tells us nothing about whether urllib3 in particular will ever have problems that are triggered by local conditions that no one thought to test for; it might not.

Third: -m unittest discover is very recent, and happened in Python 2.7 and 3.2 when Michael Foord got his unittest2 accepted into Standard Library as the new official unittest. It gives all Python developers, out-of-the-box, what once would have required an extra test-discovery dependency like py.test or nose. Michael's great success has, in fact, lead the nose team to decide to deprecate their whole code base and start over again with a nose2 project that will build atop Michael's unittest discover:

http://groups.google.com/group/nose-dev/browse_thread/thread/6e3c546f7cd2692a#

Fourth: including tests with a package is probably a much smaller phenomenon than I imagined. A quick glance I just made at some packages suggests that tests tend to be included by people with Zope or, more generally, European backgrounds — like core committer Armin Ronacher (see Werkzeug and Flask) and like Chris McDonough (see his Pyramid framework). But enough high-profile packages like mock and coverage and Django omit any tests from their distributions that it's probably just a habit I picked up years ago, back when I was involved with Grok and met lots of Zope people in the process, and then confirmation bias set in and prevented me from stopping to notice how many projects out in the rest of the Python world fail to ship tests. So it's probably a quirk of my own that needn't affect urllib3 unless you happen to like the idea on its own merits. I will bring the issue up at the testing BOF at PyCon this year and see which side of the room roars loudest. :)

from urllib3.

Let's discuss this further at PyCon. :) Closing for now.

from urllib3.