Not noticing zip import failed causes later workflow problems

[jeremydouglass]

From @scottkleinman:

Common Errors in Notebooks 1 and 2

You have supplied an incorrect path to the zip files containing your data. It is possible not to notice the error message until you run Notebook 2. A good practice is to check whether there are files in your caches/json folder. If not, the import has failed, and you should go back and check the path(s) you supplied in Notebook 1.

If Notebook 2 still doesn't run, check whether there are any files in your caches/text_files and caches/text_files folders. This will help you determine what stage in the workflow is encountering an error.

[jeremydouglass]

I have an idea for patching the zip import cell -- something like this:

from IPython.core.display import display, HTML

!mkdir -p caches/json
unzipped = []
errors = []
for datafile in datafile_list:
    datapath = jsondatadir + datafile
    result = !unzip -j -o -u {datapath} -d caches/json
    result_str = '\n'.join(result)
    if 'unzip:' in result_str:
        errors.append(result_str)
    else:
        unzipped.append(result_str)

if unzipped:
    for event in unzipped:
        print(event)

if errors:
    display(HTML('<h1>ERRORS</h1>'))
    for error in errors:
        display(HTML('<strong>Error:</strong>'))
        print(error)