Data loaders with jupyter?

[nmondon]

I am already using the Observable framework with data loaders in Python, and I would like to use it with Jupyter Notebook for more flexibility in the development process. However, I admit that I am a bit lost in the configuration options for extending the managed interpreters.

I have tried several things with papermill and jupyter nbconvert but haven’t been able to find the right configuration.

Let me know if you need further adjustments!

For my .ipynb file, I'm using the same code than the loader-python-to-csv example provided in the repo. (I've just modified the path to load the penguins file)

import pandas as pd
from sklearn.linear_model import LogisticRegression
import sys

# Read the CSV
df = pd.read_csv("./penguins.csv")

# Select columns to train the model
X = df.iloc[:, [2, 3, 4, 5]]
Y = df.iloc[:, 0]

# Create an instance of Logistic Regression Classifier and fit the data.
logreg = LogisticRegression()
logreg.fit(X, Y)

results = df.copy()
# Add predicted values
results["species_predicted"] = logreg.predict(X)

# Write to CSV
results.to_csv(sys.stdout)

For the configuration, I've tried papermill and jupyter nbconvert but have to admit I'm really struggling to find the proper settings.

{
  ".ipynb":["papermill", "--stdout-file /dev/stdout"]
}

Thanks If you can explain how to deal with this!

Nicolas.

[mbostock]

My understanding is that jupyter nbconvert --to notebook --inplace --execute foo.ipynb will execute the notebook, but the output of the cell won’t go to standard output (and thus won’t be accessible via a data loader) — it’ll simply be written back into the foo.ipynb file. So you’ll need a different command that extracts the output of the desired cell(s) from the .ipynb file format.

The .ipynb file format is JSON, so you could use jq to extract cell outputs. For example, this will output the contents of the first cell. (It appears that the JSON format represents each line of output as an array of strings, so -j reconstructs it.)

export default {
  interpreters: {
    ".ipynb": ["jq", "-j", ".cells.[0].outputs.[0].text.[]"]
  }
};

Then if you have a foo.csv.ipynb with the Python cell above, you can access the output in Framework. And you can use Jupyter to re-run the cell manually.

If you want Framework to also execute the notebook for you, you probably need to write a helper script which runs jupyter nbconvert --execute to execute the notebook and then the jq command to extract the output.

[nmondon]

Thanks @mbostock!
I'm gonna try the second solution encapsulated in a script and will post the given result

An alternative solution I didn't think of is to use a .py file with Jupyter code cells in VSCode for instance. This way, you can develop in a python file with the ease of notebook and don't have to customise the Observable framework configuration...

# %%
msg = "Hello"
print(msg)