setUpClass is not called with either pytest>4.1.1 or pytest-django>3.4.6

[belugame]

This is similar to #710 and #624

These 2 combinations work for me on python2:

pytest==4.1.1
pytest-django==3.5.1

pytest==4.6.4
pytest-django==3.4.6

But I cannot run the 2 libraries together on their latest release as in one test method out of a test suite with more than 5000 unit tests it occurs that the setUpClass method for the one test is then not called. The setUpClass is defined in a widely used mixin class (not inheriting from a TestClass base) and works everywhere else.

It is really weird how two testcases seem to disturb one another but it does not make any sense to me:

Here a quick description of my scenario:

mixin.py
class MyMixin() with a setUpClass

firstmodule.py:
class FirstTestCase(MyMixin) with test_foo

secondmodule.py

from firstmodule import FirstTestCase <---- when commenting this out my setUpClass gets run again for FirstTestCase::test_foo
class OtherModulesTestCase(FirstTestCase)

[blueyed]

Thanks for the info.
This might help to debug this.
/cc @debnet
IIRC there is still no test to reproduce this (i.e. a failing one), which might help anybody looking into this. So you could try turning your example into a testdir style test (see the existing tests, simulating a pytest run).

[belugame]

Would you know of an example unit test where one test_module imports the other, @blueyed ?

[blueyed]

Not from the top of my head, sorry - but there are not that many, and IIRC some unittest specific ones already.

[blueyed]

Maybe related: pytest-dev/pytest#5201

[belugame]

I'm trying to isolate it, sadly my setup is really complex. Another factor is that I use pytest-test-groups, possibly running the one test while the 2nd who imports it is not run, has an effect.
Simply having a test_foo.py in the 2nd module with only the import-line is enough for it to no longer run setUpClass.

[belugame]

pytest-test-groups seems to have an effect. That is the diff of running the bad test with group-slicing and without, both with --setup-plan. Where I force an exception in the setUpClass. One can see that it is not called.

[blueyed]

Another factor is that I use pytest-test-groups

That cannot be factored out / disabled?
Ok, it might even trigger it according to your last comment.

Then try to isolate it based on what it is doing.

Yes, the problem appears to be non-obvious, otherwise it would likely have been fixed already - I was not able to reproduce it myself.

If you cannot isolate it, I suggest debugging it with pdb / dumping import traceback; traceback.print_stack(file=sys.stdout) etc.

[belugame]

I have spent many hours on debugging this and have finally at least some new evidence:
The failing test gets collected twice, when I look at the line:

fixturenames_closure = self._getautousenames(parentid)
For the first collection it has '_UnitTestCase__pytest_class_setup' in it, for the 2nd it does not. I will try to work out why it gets collected twice, I guess that is the root cause.

[belugame]

This helped me:
https://stackoverflow.com/questions/32960334/import-runs-tests-twice-in-pytest

Changing it to:

import firstmodule

class OtherModulesTestCase(firstmodule.FirstTestCase):

It makes absolutely no sense to me though oO. Sure hope this does not cost anyone else thaaat much time as it did me. I have this import concept all over the codebase, I don't understand why it would only happen there.

[blueyed]

Interesting!
Can you turn this into a failing test then, please?
See testing/test_unittest.py for a starting point, e.g. test_runTest_method.
The description in the initial comment is not really clear, e.g. how / where is FirstTestCase defined when commented out?

from firstmodule import FirstTestCase <---- when commenting this out my setUpClass gets run again for FirstTestCase::test_foo

[blueyed]

Looks related to how pytest-django disables class methods btw (which is not done with newer pytest anymore):

pytest-django/pytest_django/plugin.py

Lines 380 to 394 in c1bdb8d

	def _disable_class_methods(cls):
	if cls in _disabled_classmethods:
	return
	_disabled_classmethods[cls] = (
	# Get the classmethod object (not the resulting bound method),
	# otherwise inheritance will be broken when restoring.
	cls.__dict__.get("setUpClass"),
	_classmethod_is_defined_at_leaf(cls, "setUpClass"),
	cls.__dict__.get("tearDownClass"),
	_classmethod_is_defined_at_leaf(cls, "tearDownClass"),
	)
	cls.setUpClass = types.MethodType(lambda cls: None, cls)
	cls.tearDownClass = types.MethodType(lambda cls: None, cls)

[blueyed]

(Oh, thought this issue would be in pytest's repo.)

[belugame]

(Oh, thought this issue would be in pytest's repo.)

It would make more sense there, though at the time of reporting I did not know better.

[blueyed]

Sure. I've looked quickly into reproducing this using https://github.com/blueyed/debug-pytest-unittest - it shows:

test_test.py::Test::test_1st <- t-unittest/mod1/__init__.py test_1st <test_test.Test object at 0x7fa7954d13d0>
PASSED
test_test.py::Test::test_2nd <- t-unittest/mod2/__init__.py test_2nd <test_test.Test object at 0x7fa7954d1390>
PASSED
test_test.py::Test::test_test test_test <test_test.Test object at 0x7fa79546b1d0>
PASSED

Not sure if that is what you are seeing etc - please either create a pytest test to reproduce your issue, or a patch for the above initial layout.

[belugame]

Your repo is reproducing it nicely, the setUpClass is not called. I can put a raise Exception() in it and tests still pass.

pytest -s -vv 
============= test session starts =============
platform linux2 -- Python 2.7.12, pytest-4.6.4, py-1.8.0, pluggy-0.12.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/foo/repos/debug-pytest-unittest
plugins: pylama-7.0.7
collected 3 items                                                                                                                      

test_test.py::Test::test_1st <- mod1/__init__.py ('test_1st', <test_test.Test instance at 0x7fcbef711c68>)
PASSED
test_test.py::Test::test_2nd <- mod2/__init__.py ('test_2nd', <test_test.Test instance at 0x7fcbef726248>)
PASSED
test_test.py::Test::test_test ('test_test', <test_test.Test instance at 0x7fcbef7266c8>)
PASSED

============= 3 passed in 0.02 seconds =============

[blueyed]

Your repo is reproducing it nicely, the setUpClass is not called. I can put a raise Exception() in it and tests still pass.

This changes when inheriting from TestCase (to make them unittests actually): blueyed/debug-pytest-unittest@ae1f1d1

This then also runs 5 tests in total:

collected 5 items

test_test.py::SecondTestCase::test_1st <- t-unittest/mod1/__init__.py MyMixin.setUpClass <class 'mod2.SecondTestCase'>
test_1st test_1st (mod2.SecondTestCase)
PASSED
test_test.py::SecondTestCase::test_2nd <- t-unittest/mod2/__init__.py test_2nd test_2nd (mod2.SecondTestCase)
PASSED
test_test.py::Test::test_1st <- t-unittest/mod1/__init__.py MyMixin.setUpClass <class 'test_test.Test'>
test_1st test_1st (test_test.Test)
PASSED
test_test.py::Test::test_2nd <- t-unittest/mod2/__init__.py test_2nd test_2nd (test_test.Test)
PASSED
test_test.py::Test::test_test test_test test_test (test_test.Test)
PASSED

With --collect-only:

collected 5 items
<Module t-unittest/test_test.py>
  <UnitTestCase SecondTestCase>
    <TestCaseFunction test_1st>
    <TestCaseFunction test_2nd>
  <UnitTestCase Test>
    <TestCaseFunction test_1st>
    <TestCaseFunction test_2nd>
    <TestCaseFunction test_test>

[blueyed]

What happens here is that only "test_test.py" is collected, which has SecondTestCase and Test in it.
This is kind of what I would expect also - is that your issue?

[blueyed]

As for the "setUpClass is not called" issue: are you deriving from unittest.TestCase?

[belugame]

The setUpClass is on a non-TestCase class. I expected it to be called non the less. Though I was so unfar unable to reproduce my exact scenario. :-/