Make freeze functional tests depend on less external services

[msabramo]

This change:

1. prevents a failure if we can't reach:
   • bazaar.launchpad.net
   • bitbucket.org
   • github.com
   • pypi.python.org
   • svn.colorstudy.com
2. for me this reduced the time to run:

.tox/py27/bin/py.test tests/functional/test_freeze.py --tb=short

from 90 seconds to 76 seconds. I think that a lot of the remaining time is spent on creating virtualenvs rather than downloading things.

[pfmoore]

I've not looked at this in context, but should this not use the test data directory fixture data?

[msabramo]

Oh probably. I'm having trouble grokking how the tests fit together.

[msabramo]

OK, updated.

[msabramo]

By the way, I notice that even without hitting the network, these tests are pretty slow and it seems like a good chunk of time is spent on creating the virtualenvs -- any ideas for how to speed that up?

[pfmoore]

Nope, on Windows creating virtualenvs seems to fail with no explanation (at least sometimes, it hit me earlier today). Someone with way more time than I have needs to get the tests working on Windows (or lots of people need to fix them one little bit at a time, as per your posting on distutils-sig - do you have access to an army of open-source enthusiasts who are Windows experts? ;-))

[msabramo]

test_install could use the same treatment.

But tests are passing so this could be merged as-is. Either way.

[xavfernandez]

Do we really need to include the whole django-wikiapp ? We could certainly make a simplebzr module and put it in a bzr repo. Maybe even via a fixture ?

Edit: something similar to https://github.com/pypa/pip/blob/develop/tests/lib/__init__.py#L461-L485 maybe ?

[dstufft]

We don't need too. The pip tests have a bad habit of using whatever package the person writing the test happened to be using at the time. We should absolutely try to get rid of these things and use dummy packages. Ideally local test fixtures where possible.

On Mar 10, 2015, at 8:00 AM, Xavier Fernandez [email protected] wrote:

Do we really need to include the whole django-wikiapp ? We could certainly make a simplebzr module and put it in a bzr repo. Maybe even via a fixture ?

—
Reply to this email directly or view it on GitHub.

[msabramo]

Yeah, I just pulled that in because that's what it had, but I could certainly change it to use something a little smaller.

The fixtures -- hmmm -- I guess I could see using fixtures for dynamically creating git repos (since you had to have git to clone the pip tree) and maybe mercurial because most folks probably have that, but I don't know if I would use that for bzr and then force people to have bzr on their system. I feel like for bzr I would rather commit a small package -- you guys okay with that?

[pfmoore]

Surely if you don't have bzr the bzr tests won't work anyway, so needing bzr as a prerequisite to setting up a bzr test isn't any extra issue? Just don't try to create the bzr repo at the start of the test run and fail everything if you can't ;-)

[msabramo]

@pfmoore: Good point. I hadn't had my coffee yet. I could probably just skip the test if bzr is not available.

Crap - now I need to look up how to use bzr again :)

[msabramo]

@dstufft, @pfmoore : You okay with the overall idea of this though?

[pfmoore]

Sure. Anything to make the tests less fragile :-)

[msabramo]

Cool, I'm on it

[msabramo]

I just made a bunch of changes to test_freeze_git_clone to make it use a dynamically created git repo (created with tests.lib._create_test_package) instead of cloning from GitHub. This seems like it will be more robust and it doesn't require network access.

I still need to do something similar for the bzr test.

[msabramo]

BTW, I wouldn't tackle it here, but has there ever been discussion of moving all the VCS code out into some kind of plugins? That seems like it would remove a good bit of code from pip, though of course we'd have to come up with an API.

[msabramo]

I changed test_freeze_bazaar_clone in 86fcd3c so that it dynamically creates a small toy bzr repo for testing purposes and I removed the copy of django-wikiapp.

None of the tests in test_freeze.py depend on having network access now and to reflect that, I removed the @pytest.mark.network decorator from all the functions that used it.

Travis CI tests are passing. I hope that this can be reviewed and merged soon so that the tests are less fragile.

[pfmoore]

Spurious "tidy up" change, presumably?

[msabramo]

Oops. Fixed.

[pfmoore]

(Rats, got distracted by adding line comments and lost my larger comment-in-progress :-()

Wow, that's a big change! A lot of it looks like a more or less opaque change to tests/data/repos/hg/pip-test-package, but it'll take me some time to work out what's going on here.

Any chance you could summarise what the various bits of the PR are about? You've got 8 separate commits here. Maybe you could summarise what each one does? (Maybe it would have been easier to review if the PR was split into separate PRs for each commit). I'll try to get a chance to review properly, but it could be some time before I'm able to.

[msabramo]

I could split it into separate PRs, if that makes it easier to review. Little atomic chunks should be easier to understand and review quickly I think.

[pfmoore]

Thanks, @msabramo - I'll go through the individual PRs.

[msabramo]

This can be closed as it was replaced by multiple small PRs:

• test_freeze_mercurial_clone: no dep on bitbucket test_freeze_mercurial_clone: no dep on bitbucket #2525
• test_freeze_svn: No dep on bitbucket test_freeze_svn: No dep on bitbucket #2517
• test_freeze_git_clone: No dep on github.com test_freeze_git_clone: No dep on github.com #2516
• Make test_freeze_bazaar_clone not depend on bazaar.launchpad.net Make test_freeze_bazaar_clone not depend on bazaar.launchpad.net #2515
• test_freeze: Don't depend on PyPI test_freeze: Don't depend on PyPI #2514