Is it possible to run Robot Framework's unittests with Pytest?

[Tset-Noitamotua]

Hi guys! Could you please help with this Stackoverflow Question.

I would like to run all unit tests from Robot Framework repo with Pytest. But when I call pytest utest/ I get nothing but lot of errors. When I point pytest to single test files e.g. pytest utest/api/test_logging_api.py it works in many cases but not in all. E.g. the next one doen not work

pytest utest/api/test_run_and_rebot.py
====================================== test session starts ======================================
platform linux2 -- Python 2.7.13, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/wlad/_GITHUB/robotframework, inifile:
collected 0 items / 1 errors 

============================================ ERRORS =============================================
_______________________ ERROR collecting utest/api/test_run_and_rebot.py ________________________
ImportError while importing test module '/home/wlad/_GITHUB/robotframework/utest/api/test_run_and_rebot.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
utest/api/test_run_and_rebot.py:18: in <module>
    from resources.runningtestcase import RunningTestCase
E   ImportError: No module named resources.runningtestcase
!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================== 1 error in 0.34 seconds ====================================

I think it's because run.py gets some stuff from the atest folder which does not happen when I just call pytest utest/.

addition information

• I am a python beginner
• OS: Ubuntu 17.04 - VirtualBox VM on Windows 7 64bit host.

(PYTEST_RF) wlad@wlad-VirtualBox:~/_GITHUB/robotframework/utest$ pip list
Package        Version           Location                             
-------------- ----------------- -------------------------------------
pip            9.0.1             
pkg-resources  0.0.0             
py             1.4.34            
pytest         3.1.2             
robotframework 3.0.3.dev20170213 /home/wlad/_GITHUB/robotframework/src
setuptools     36.0.1            
wheel          0.29.0       

[RonnyPfannschmidt]

if it uses creepy hacks to manage sys.path, do the same ^^

[RonnyPfannschmidt]

closing as pytest itself cant really do anything about robot framework

[Tset-Noitamotua]

@RonnyPfannschmidt What do you mean by "using creepy hacks to manage sys.path"?

What do I need to change in below part of [run.py] (https://github.com/robotframework/robotframework/blob/master/utest/run.py#L24) to make test execution happen with pytest?

if __name__ == '__main__':
    docs, vrbst = parse_args(sys.argv[1:])
    tests = get_tests()
    suite = unittest.TestSuite(tests)
    runner = unittest.TextTestRunner(descriptions=docs, verbosity=vrbst)
    result = runner.run(suite)
    rc = len(result.failures) + len(result.errors)
    if rc > 250:
        rc = 250
    sys.exit(rc)

I have not found anything about how to transform unittest.TestSuite into pytest world :(

[nicoddemus]

@Tset-Noitamotua looking at the code you posted:

for path in ['../src', '../atest/testresources/testlibs']:

It seems run.py adds some paths to sys.path. You can do the same by setting the PYTHONPATH environment variable (that's what @RonnyPfannschmidt mentioned by "sys path trickery" 😁).

[Tset-Noitamotua]

@nicoddemus thanks for the hint

I did two things and I am almost happy with the result

1. in run.py add import pytest
2. in run.py edit the if __name__ == '__main__': section so that it looks like that:

if __name__ == '__main__':
    docs, vrbst = parse_args(sys.argv[1:])
    tests = get_tests()
    pytest.main()

Then in utest folder just call python run.py and the tests run :))) (There is only one error and some warnings)

(PYTEST_RF) wlad@wlad-VirtualBox:~/_GITHUB/robotframework_myfork/utest$ python run.py
================================== test session starts ===================================
platform linux2 -- Python 2.7.13, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/wlad/_GITHUB/robotframework_myfork, inifile:
collected 1533 items 

api/test_deco.py ....
api/test_exposed_api.py ............
api/test_logging_api.py .....
api/test_run_and_rebot.py ..................................
api/test_using_libraries.py ....
conf/test_settings.py ............
htmldata/test_htmltemplate.py ...

...

========================================= ERRORS =========================================
_____________________________ ERROR at setup of test_convert _____________________________
file /home/wlad/_GITHUB/robotframework_myfork/utest/testdoc/test_jsonconverter.py, line 10
  def test_convert(item, **expected):
E       fixture 'item' not found
>       available fixtures: cache, capfd, capsys, doctest_namespace, monkeypatch, pytestconfig, record_xml_property, recwarn, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/wlad/_GITHUB/robotframework_myfork/utest/testdoc/test_jsonconverter.py:10
==================================== warnings summary ====================================
utest/model/test_filter.py::TestSuite
  cannot collect test class 'TestSuite' because it has a __init__ constructor

...

-- Docs: http://doc.pytest.org/en/latest/warnings.html
=================== 1532 passed, 45 warnings, 1 error in 8.81 seconds ====================

[nicoddemus]

I see thanks for the follow up. 👍

run.py configures sys.path, that's why it works.

test_convert fails because pytest assumes it is a test function (it begins with test_) so pytest tries to execute it. IIRC you can add an attribute to the function so pytest skips it:

def test_convert(item, **expected):
    ...
test_convert.__test__ = False

[Tset-Noitamotua]

@nicoddemus What about the warnings? Can they be suppressed somehow?

[nicoddemus]

You can set the same __test__ = False attribute in the TestSuite class so pytest doesn't try to consider it a test.

[Tset-Noitamotua]

@nicoddemus Sorry, but I'm stuck, may I ask for one short example?

e.g. this warning utest/model/test_filter.py::TestSuite cannot collect test class 'TestSuite' because it has a __init__ constructor

I don't see any TestSuite class in utest/model/test_filter.py

So where should I set __test__ = False in this case?