Python UnitTest Discover Quirks

Pelican contains a test suite that uses Python’s UnitTest 2 [1], and working on Pelican’s source I had run that suite from time to time.

Yesterday, however, I was delighted with this:

$ unit2 discover -v

tests.test_settings (unittest2.loader.ModuleImportFailure) ... ERROR
tests.test_importer (unittest2.loader.ModuleImportFailure) ... ERROR
tests.test_utils (unittest2.loader.ModuleImportFailure) ... ERROR
tests.test_generators (unittest2.loader.ModuleImportFailure) ... ERROR
tests.test_contents (unittest2.loader.ModuleImportFailure) ... ERROR
tests.test_readers (unittest2.loader.ModuleImportFailure) ... ERROR
tests.test_pelican (unittest2.loader.ModuleImportFailure) ... ERROR

======================================================================
ERROR: tests.test_settings (unittest2.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_settings
Traceback (most recent call last):
  File "/home/dm/myprojects/python27-env/local/lib/python2.7/site-packages/unittest2/loader.py", line 267, in _find_tests
        module = self._get_module_from_name(name)
  File "/home/dm/myprojects/python27-env/local/lib/python2.7/site-packages/unittest2/loader.py", line 243, in _get_module_from_name
        __import__(name)
ImportError: No module named test_settings

[... several more tracebacks ...]

----------------------------------------------------------------------
Ran 7 tests in 0.013s

FAILED (errors=7)

Hmm, obviously unit2 was able to locate the tests, but it failed importing them as modules. How could that be? In ipython shell I checked the contents of sys.path, but could not see anything suspicious: a path to the Pelican sources was correctly present.

Remembering the IT Crowd, I deinstalled Pelican and installed it again. Alas, the same errors occurred. Out of a hunch I deinstalled Pelican again and run discover then. — To my big surprise, the suite ran fine!

This riddle I could not let go uninvestigated. So I patched the loader of unittest to print sys.path everytime it tries to import a module. The paths appeared identical, whether Pelican was installed or not. Yes, even with uninstalled Pelican a path to its sources were present. It seems UnitTest inserts a path to the current working directory if such was not already there. But, when UnitTest inserted the path, it was at the topmost position, when Pelican was installed, it was located somewhere in-between the other paths. So, is the position of the Pelican path relevant?

To verify my theory, I installed Pelican again, and patched UnitTest’s loader so that Pelican’s path always was topmost. Yay, the suite runs fine.

But why is the position of the path relevant?

Then it came to me: I had also installed feedgenerator, a different package, from its sources, and those contain a directory tests with test modules. As is the case with Pelican! So, when the Pelican path is located after feedgenerator’s, discover finds all of Pelican’s test files, because it walks the directory tree directly, but when it attempts to load them as modules, like module tests.test_settings, Python now walks through its sys.path and the first occurence it finds for tests is the directory of feedgenerator, which of course does not contain a module test_settings [2].

Mystery solved, but the quirks still remain. If I order the paths so that Pelican appears first, I can test Pelican, but the tests of feedgenerator will fail. If I switch the order, I can test feedgenerator, but not Pelican. Now, that’s uncomfortable, to say the least.