Clarify what is linting and what is testing

[ceball]

I'd like to remove the duplication of concepts and code between linting and testing, making nbcelltests simpler and easier to understand for users, and simplifying the code.

nbcelltests.lint allows specification of rules:

max_cells_per_notebook
max_class_definitions
max_function_definitions
max_lines_per_cell
magics_whitelist/magics_blacklist
kernelspec_requirements

lint.run() checks these rules and returns the results (details, plus pass/fail). At the command line, nbcelltests.lint prints the details and exits with status according to pass/fail.

Meanwhile, nbcelltests.test works by generating a unittest class with test_cell methods corresponding to code cells. However, after those methods, the following methods are also added to the class, based on rules supplied to test.run():

test_cell_coverage
test_cells_per_notebook
test_class_definition_count
test_function_definition_count
test_lines_per_cell

Such methods each contain just something like this: assert 5 <= 4 (e.g. where you have said max cells = 4, but there are actually 5 cells).

I think we should remove these rules from nbcelltests.test, and make a clear distinction between what is linting, what is testing, and what is coverage.

Testing refers to cell-by-cell testing (based on execution of) the code. Coverage refers to how many cells have tests*. Linting refers to static checks. I.e. the distinction is not based on how important each type of thing is. Users can easily weight the importance of each as they wish (e.g. if some lint rule is critical to you, you can make sure it fails your process by using the details returned from lint()).

(*) at least for now - we might also have an alternative measure in the future, measuring how much of the notebook code is actually covered

[vidartf]

I'm +1 on removing the "quality tests", and keeping them only as lints. As long as we make the linting part easy to understand that should cover it.

Re the coverage test, I say lets leave that in until we have the improved coverage solution in place. Hopefully this code can help us there: https://github.com/computationalmodelling/nbval/blob/master/nbval/_cover5.py

[ceball]

Re the coverage test, I say lets leave that in until we have the improved coverage solution in place.

Maybe we should rename the existing "coverage" while we still have chance? E.g. maybe something more like "cell coverage" than "coverage".

[vidartf]

I would say "cell coverage" is a stop-gap until we have line coverage, and should be removed in the future.

[ceball]

I just realized it's already called cell_coverage anyway :)

[ceball]

Was closed by #124, github