Add tests for cell/test correspondence, and test skipping

[ceball]

This PR is not for merging. It demonstrates failing tests on master. These same tests are then fixed in #112.

See first comment below for a summary of the test results.

[ceball]

Here are the test failures (from the CI logs)...

bugs

Things that are not really up for discussion...

check for confusion by non-code cells

Cells+tests out of correspondence after markdown cell:

________________ TestCellCounting.test_still_ok_after_markdown _________________

self = <nbcelltests.tests.test_test.TestCellCounting testMethod=test_still_ok_after_markdown>

    def test_still_ok_after_markdown(self):
        """Correspondence still ok after markdown cell?"""
>       self.t.test_cell_1()

[...]

E               Exception: Cell execution caused an exception\n\n---------------------------------------------------------------------------\nNameError                                 Traceback (most recent call last)\n<ipython-input-1-1a44e218ec40> in <module>
E               ----> 1 assert x == 1
E                     2 
E               \nNameError: name 'x' is not defined

nbcelltests/tests_vendored.py:137: Exception

Cells+tests out of correspondence after raw cell:

___________________ TestCellCounting.test_still_ok_after_raw ___________________

self = <nbcelltests.tests.test_test.TestCellCounting testMethod=test_still_ok_after_raw>

    def test_still_ok_after_raw(self):
        """Correspondence still ok after raw cell?"""
>       self.t.test_cell_2()

[...]

E               Exception: Cell execution caused an exception\n\n---------------------------------------------------------------------------\nNameError                                 Traceback (most recent call last)\n<ipython-input-1-f2b7bef70563> in <module>
E               ----> 1 assert x == 1
E                     2 # Cell {2} content
E                     3 
E                     4 x = 3
E                     5 
E               \nNameError: name 'x' is not defined

Make it clear which cells are tested

Things that are up for discussion, but which have been addressed in #112 - an attempt to make it clear which cells are tested and which are not:

• test methods only generated for code cells
• test method generated for every code cell
• empty code cell results in skipped test
• no or empty test results in skipped test
• code cell not injected into test results in skipped test

Can't test non-code cells

Testing a non-code cell results in failure to generate test script in the first place

_____ TestMethodGenerationError.test_non_code_cell_with_test_causes_error ______

self = <nbcelltests.tests.test_test.TestMethodGenerationError testMethod=test_non_code_cell_with_test_causes_error>

    def test_non_code_cell_with_test_causes_error(self):
        try:
            _generate_test_module(self.NBNAME, "module.name.irrelevant")
        except ValueError as e:
            assert e.args[0] == 'Cell 1 is not a code cell, but metadata contains test code!'
        else:
>           raise Exception("Test script should fail to generate")
E           Exception: Test script should fail to generate

nbcelltests/tests/test_test.py:120: Exception

Empty code cell results in skipped test

________________ TestSkips.test_skip_completely_empty_code_cell ________________

self = <nbcelltests.tests.test_test.TestSkips testMethod=test_skip_completely_empty_code_cell>

    def test_skip_completely_empty_code_cell(self):
        """Test+cell where cell has nothing at all in should be skipped"""
>       self._assert_skipped(self.t.test_cell_0, "empty code cell")

nbcelltests/tests/test_test.py:143: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nbcelltests/tests/test_test.py:99: in _assert_skipped
    self.assertTrue(hasattr(mthd, '__unittest_skip__'), msg=msg)
E   AssertionError: False is not true : Should have generated a skipped test method

____________________ TestSkips.test_skip_no_code_code_cell _____________________

self = <nbcelltests.tests.test_test.TestSkips testMethod=test_skip_no_code_code_cell>

    def test_skip_no_code_code_cell(self):
        """Test+cell where cell had e.g. just a comment in should be skipped"""
>       self._assert_skipped(self.t.test_cell_1, "empty code cell")

nbcelltests/tests/test_test.py:147: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nbcelltests/tests/test_test.py:99: in _assert_skipped
    self.assertTrue(hasattr(mthd, '__unittest_skip__'), msg=msg)
E   AssertionError: False is not true : Should have generated a skipped test method

skip test if cell has no test or empty test

__________________ TestSkips.test_skip_completely_empty_test ___________________

self = <nbcelltests.tests.test_test.TestSkips testMethod=test_skip_completely_empty_test>

    def test_skip_completely_empty_test(self):
        """Cell where test is completely empty should be skipped"""
>       self._assert_skipped(self.t.test_cell_4, "no test supplied")

nbcelltests/tests/test_test.py:159: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nbcelltests/tests/test_test.py:99: in _assert_skipped
    self.assertTrue(hasattr(mthd, '__unittest_skip__'), msg=msg)
E   AssertionError: False is not true : Should have generated a skipped test method

_____________________ TestSkips.test_skip_no_code_in_test ______________________

self = <nbcelltests.tests.test_test.TestSkips testMethod=test_skip_no_code_in_test>

    def test_skip_no_code_in_test(self):
        """Test+cell where test is just e.g. a comment should be skipped"""
>       self._assert_skipped(self.t.test_cell_3, "no test supplied")

nbcelltests/tests/test_test.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nbcelltests/tests/test_test.py:99: in _assert_skipped
    self.assertTrue(hasattr(mthd, '__unittest_skip__'), msg=msg)
E   AssertionError: False is not true : Should have generated a skipped test method

________________ TestSkips.test_skip_no_test_field_in_metadata _________________

self = <nbcelltests.tests.test_test.TestSkips testMethod=test_skip_no_test_field_in_metadata>

    def test_skip_no_test_field_in_metadata(self):
        """Cell with no test metadata should be skipped"""
>       self._assert_skipped(self.t.test_cell_2, "no test supplied")

nbcelltests/tests/test_test.py:151: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nbcelltests/tests/test_test.py:99: in _assert_skipped
    self.assertTrue(hasattr(mthd, '__unittest_skip__'), msg=msg)
E   AssertionError: False is not true : Should have generated a skipped test method

skip test if cell not injected into test

___________________ TestSkips.test_skip_if_no_cell_injection ___________________

self = <nbcelltests.tests.test_test.TestSkips testMethod=test_skip_if_no_cell_injection>

    def test_skip_if_no_cell_injection(self):
        """Test+cell where test does not inject cell should be skipped"""
>       self._assert_skipped(self.t.test_cell_5, "cell code not injected into test")

nbcelltests/tests/test_test.py:163: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
nbcelltests/tests/test_test.py:99: in _assert_skipped
    self.assertTrue(hasattr(mthd, '__unittest_skip__'), msg=msg)
E   AssertionError: False is not true : Should have generated a skipped test method