workingnotes.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db431c29",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i underthecovers/python/common.py\n",
    "%run -i underthecovers/python/ln_preamble.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ad98800",
   "metadata": {},
   "source": [
    "# Status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c54e53",
   "metadata": {},
   "source": [
    "## Problems\n",
    "\n",
    "1. Very poor and inconsistent performance on operate-first\n",
    "    - discussed this with RedHat performance seems to be getting better\n",
    "2. nootbook output cells data is painful in git consider other options:\n",
    "    - In some sense this is what Juptext is supposed to help with .... but I could not figure out the right way to get jupyter-book to recongnize Juptext markdown documents to be executable and hence they did not have the launch icon generated for them.\n",
    "    - in the end I have settled on using [nbstripout](https://github.com/kynan/nbstripout) ... container has a default install of it configured for all users ... see `/etc/gitconfig` \n",
    "3. Docker Image is very large\n",
    "    - have not been worried about size\n",
    "        - broken down into 3 docker stages:\n",
    "            1. base  \n",
    "                - From jupyter/minimal-notebook\n",
    "                    - Installs a bunch of ubuntu packages to make generic book development and teaching easier \n",
    "                    - includes a build of a recent version of gdb as distros don't seem to include one yet\n",
    "            2. base-unmin\n",
    "                - From base (above)\n",
    "                    - invokes unminize command so that user experience is more like a full blown system.  Eg manpages and other doc are present\n",
    "            3. bu-cs-book-dev\n",
    "                - From base-unmin\n",
    "                    - adds all the extra python packages for the jupyter-book and rise stuff I depend on for UC:SLS\n",
    "                    - adds custom startup scripts so that starting on MOC/operate-first behaves correctly\n",
    "                    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a89d217",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "1. Work on improving Dockerfiles to avoid unnecessary layers\n",
    "2. Exploit startup hooks to try and customize the machine a bit more:\n",
    "    - add correct user name to /etc/passwd\n",
    "    - see if we can figure out a way to rename home dir\n",
    "    - fix env then USER and HOME, etc\n",
    "    - fix hostname\n",
    "3. document that we are using the default jupyter user jovyan (https://jupyter.readthedocs.io/en/latest/community/content-community.html#what-is-a-jovyan) to be our container user\n",
    "4. Add ability to communicate between shell/terminal and python.  \n",
    "    - useful things to keep in mind : https://askubuntu.com/questions/853102/how-do-i-run-a-command-before-or-after-every-command-written-in-the-console\n",
    "    - socat as a man in the middle\n",
    "    - named pipes\n",
    "    - background python threads\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b57cf1ab",
   "metadata": {},
   "source": [
    "## Things to think about using\n",
    "\n",
    "- https://towardsdatascience.com/7-essential-tips-for-writing-with-jupyter-notebook-60972a1a8901#231d\n",
    "-"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ea7410",
   "metadata": {},
   "source": [
    "## Working  \n",
    "0.  This url works locally if you have your jupyter server running locally:<br> (eg make nb on container repo)\n",
    "    http://127.0.0.1:8888/git-pull?bu-cs-jupyter-book%3Alatest=&repo=https%3A%2F%2Fgithub.com%2Fjappavoo%2FUndertheCovers&urlpath=tree%2FUndertheCovers%2Funderthecovers%2F&branch=main\n",
    "1.  Operate First direct spawn url for bu-cs-jupyter-book:latest:<br> \n",
    "https://jupyterhub-opf-jupyterhub.apps.smaug.na.operate-first.cloud/hub/spawn?bu-cs-jupyter-book:latest\n",
    "2. To get nbgitpuller to clone or sync the UndertheCovers repo:<br>\n",
    "https://jupyterhub-opf-jupyterhub.apps.smaug.na.operate-first.cloud/hub/user-redirect/git-pull?bu-cs-jupyter-book%3Alatest=&repo=https%3A%2F%2Fgithub.com%2Fjappavoo%2FUndertheCovers&urlpath=tree%2FUndertheCovers%2Funderthecovers%2FL00_210_JA.ipynb&branch=main\n",
    "  - Other very usefull urls:\n",
    "      - authenticated users can see the Jupyterhub namespace on openshift here:<br>\n",
    "https://jupyterhub-opf-jupyterhub.apps.smaug.na.operate-first.cloud/k8s/ns/opf-jupyterhub/pods <br>\n",
    "      - Your pod can be found here: <br>\n",
    "https://jupyterhub-opf-jupyterhub.apps.smaug.na.operate-first.cloud/k8s/ns/opf-jupyterhub/pods/jupyterhub-nb-jappavoo-40bu-2eedu <br>\n",
    "This can give you a bit more info on the state of your pod\n",
    "3. 2 required futzing with the image so that the group id of the joyvan home directory is root (see [dockerfile](https://github.com/jappavoo/bu-cs-book-dev/blob/c88bc65d65b334150724981dad145f395e177505/Dockerfile#L87)): <br>\n",
    "``` shell\n",
    "# as a hack we are going to try changing group id of /home/joyvan to be root to see if I can trick things into\n",
    "# working on the moc\n",
    "RUN chgrp -R root /home/jovyan\n",
    "```\n",
    "The startup logic of the container which we inherited from the jupyter/minimal-notebook:latest (https://github.com/jupyter/docker-stacks/blob/master/base-notebook/start.sh) does a bunch magic:\n",
    "  - when spawned by the main jupyterhub server on oprate-first the container seems to launch with a userid of the operate-first uesr (eg in my case 1000950000) and a gid of root (0)\n",
    "  - which does not match the uid of the joyvan user in the container (1000) so the startup script replaces the joyvan user with a new joyvan user who's uid is the operate-first user id.  \n",
    "  - however it does nothing to the permissions of the homedir and it seems to let the gid of root persist\n",
    "  - so by default the home directory is readable but can not be written\n",
    "  - to fix this I modify the container's base permission of the joyvan homedir to be group 0\n",
    "  - oddly enough this all seems to work\n",
    "4. Oddly enough this also seems to solve my problem of persistent state:  The default logic is starting the jupyter server in the joyvan home dir which is not on the volume mounted by operate-first and its contents is server instance specific.  Destroying the server (stopping it) and restarting seeds the joyvan home dir back to its initial state from the container\n",
    "5. To access the api we must have the api token this however is not present in the same way as it was when running the jupyter notebook server locally -- see ln_preamble.py:<br>\n",
    "   on operate-first jupyerhub the api token is in an environment variable\n",
    "6. To make get requests we must use a full url to the localhost introduced localhost_url in ln_preamble\n",
    "7. To access the users server urls we need to prefix with `/hub/user-redirect` or any equivalent eg. to get access terminal 3 it would be `/hub/user-redirect`.  So now to embed a terminal we would can succesfully do the following: <br>\n",
    "``` python\n",
    "IFrame('/hub/user-redirect/terminals/2', 1400, 600)\n",
    "```\n",
    "or\n",
    "``` python\n",
    "IFrame('/user/jappavoo@bu.edu/terminals/2', 1400, 600)\n",
    "```\n",
    "or\n",
    "``` python\n",
    "IFrame('/hub/user/jappavoo@bu.edu/terminals/2', 1400, 600)\n",
    "```\n",
    "which means that we can use base_url as reported by server info in both environments to embedded a terminal\n",
    "``` python\n",
    "IFrame(base_url + 'terminals/2', 1400, 600)\n",
    "```\n",
    "The above now lets us embed terminals in the notebooks similar to the functionality of running a locally hosted version of the container and starting the jupyter notebook server by hand (see make devnb).  Have rewriten python/ln_preamble.py appropriately.\n",
    "8. Had a very hard time with getting python package installed and configure in a way that worked on opf with all constraints meet.  The current head of the bu-cs-book-dev repo has the version that I found to work:  \n",
    "    - boots on opf (from some reason prior commit in which I very carefully tested python package commands fails to boot).  \n",
    "    - all jupyter notebook extensions that I want installed by default work and are on when user opens a notebook\n",
    "    - the right version of jupyter books is installed -- use of conda got the wrong version switch to using pip install and this got the right version\n",
    "9. cleanup home dir and added stuff to bash rc to have better default behaviour\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e9bd05d",
   "metadata": {},
   "source": [
    "## TIPS and TRICKS\n",
    "\n",
    "- to remove and hide cells from jupyter book -  add the following tags to the meta data of the cell: `remove-input, remove-output, remove-cell, hide-input, hide-output, hide-cell` see [jupyter-book doc](https://jupyterbook.org/interactive/hiding.html#removing-code-cell-content)\n",
    "- to hide from rise use skip slide tag\n",
    "- see [example](#splitColumn) of split column extension for two column layout rather than using html\n",
    "- see my [displayBytes](#displayBytes) below for better binary data support through panda tables\n",
    "- see my [htmlFig](#htmlFig) below for a figure examples\n",
    "- see my [TermShellCmd](#TermShellCmd) below for running shell commands and showing their output in a nice way... **This is becoming out of date.  See the bash and tty session stuff below**\n",
    "- see my [TtySession](#TtySession) below for a a more interesting way for running terminal based examples\n",
    "- see my [MDBox](#MDBox) and friends for creating scrolled output content that is really usefull when mixing with terminals "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e360fd9",
   "metadata": {
    "cell_style": "split"
   },
   "source": [
    "<a id='splitColumn'></a>\n",
    "# Better two split column html\n",
    "- from https://en.wikipedia.org/wiki/VT220"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f169c13",
   "metadata": {
    "cell_style": "split"
   },
   "source": [
    "## Manual html Figures\n",
    "Some html for better figures.  Here is a single figure that spans the full column\n",
    "<table id=\"fig:1Term\" align=\"center\" width=\"100%\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" style=\"border-collapse: collapse; margin: auto 2em auto auto\" >\n",
    "    <tr style=\"padding: 0; margin: 0;\">\n",
    "        <td style=\"padding: 0; margin: 0; background-color: white;\">\n",
    "            <div style=\"margin-right: auto; margin-left: auto; padding: 0; margin: 0;\">\n",
    "              <figure style=\"width:100%; padding: 0; margin: 0;border: 1px solid red;\" >\n",
    "                  <img src=\"underthecovers/images/shell.jpg\" width=\"100%\" style=\"padding: 0; margin: 0;\">\n",
    "                   <div align=\"right\" style=\"color: #978282; line-height: 0; font-size: 2vmin\">\n",
    "                    <em>\n",
    "                      See <a href=\"https://en.wikipedia.org/wiki/VT220\">vt220</a>\n",
    "                    </em>\n",
    "                  </div>\n",
    "                  <figcaption>\n",
    "                    <div style=\"background-color: red; margin-right:auto; margin-left:auto; text-align: center;\">\n",
    "                       <i style=\"color: white\">B: Terminal 2</i>\n",
    "                    </div>\n",
    "                  </figcaption>\n",
    "                </figure>\n",
    "            </div>\n",
    "        </td>\n",
    "    </tr>\n",
    "    <caption align=\"bottom\" style=\"text-align: left; padding: 0; margin: 0;\" >\n",
    "          <i>Figure 1: An <a href=\"https://en.wikipedia.org/wiki/VT220\">vt220</a> ASCII Temrinal</i> \n",
    "    </caption>\n",
    "</table>\n",
    "\n",
    "In the <a href=\"#fig:1Term\">figure</a> we see a classic ASCII terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dddabed",
   "metadata": {},
   "source": [
    "<table id=\"fig:1Term-2\" align=\"right\" width=\"50%\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" style=\"border-collapse: collapse; margin: auto auto auto .5 em\" >\n",
    "    <tr style=\"padding: 0; margin: 0;\">\n",
    "        <td style=\"padding: 0; margin: 0; background-color: white;\">\n",
    "            <div style=\"margin-right: auto; margin-left: auto; padding: 0; margin: 0;\">\n",
    "              <figure style=\"width:100%; padding: 0; margin: 0;border: 1px solid red;\" >\n",
    "                  <img src=\"underthecovers/images/shell.jpg\" width=\"100%\" style=\"padding: 0; margin: 0;\">\n",
    "                   <div align=\"right\" style=\"color: #978282; line-height: 0; font-size: 2vmin\">\n",
    "                    <em>\n",
    "                      See <a href=\"https://en.wikipedia.org/wiki/VT220\">vt220</a>\n",
    "                    </em>\n",
    "                  </div>\n",
    "                  <figcaption>\n",
    "                    <div style=\"background-color: red; margin-right:auto; margin-left:auto; text-align: center;\">\n",
    "                       <i style=\"color: white\">B: Terminal 2</i>\n",
    "                    </div>\n",
    "                  </figcaption>\n",
    "                </figure>\n",
    "            </div>\n",
    "        </td>\n",
    "    </tr>\n",
    "    <caption align=\"bottom\" style=\"text-align: left; padding: 0; margin: 0;\" >\n",
    "          <i>Figure 1: An <a href=\"https://en.wikipedia.org/wiki/VT220\">vt220</a> ASCII Temrinal</i> \n",
    "    </caption>\n",
    "</table>\n",
    "Here is a single figure that spans that text wraps to left around.\n",
    "In the <a href=\"#fig:1Term-2\">figure</a> we see a classic ASCII terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b9cac4",
   "metadata": {
    "cell_style": "center",
    "hide_input": true
   },
   "source": [
    "<table id=\"fig:3Terms\" align=\"left\" width=\"50%\" cellpadding=\"0\" cellspacing=\"0\" border=\"0\" style=\"border-collapse: collapse; margin: auto 2em auto auto\" >\n",
    "    <tr style=\"padding: 0; margin: 0;\">\n",
    "        <td style=\"padding: 0; margin: 0; background-color: white;\">\n",
    "            <div style=\"margin-right: auto; margin-left: auto; padding: 0; margin: 0;\">\n",
    "            <figure style=\"width:100%; padding: 0; margin: 0;border: 1px solid red;\" >\n",
    "                <img src=\"underthecovers/images/shell.jpg\" width=\"100%\" style=\"padding: 0; margin: 0;\">\n",
    "                <figcaption>\n",
    "                    <div style=\"background-color: red; margin-right:auto; margin-left:auto; text-align: center;\">\n",
    "                       <i style=\"color: white\">B: Terminal 2</i>\n",
    "                    </div>\n",
    "                </figcaption>\n",
    "            </figure>\n",
    "            </div>\n",
    "        </td>\n",
    "        <td style=\"padding: 0; margin: 0; background-color: white;\">\n",
    "            <div style=\"margin-right: auto; margin-left: auto; padding: 0; margin: 0;\">\n",
    "            <figure style=\"width:100%;  padding: 0; margin: 0; border: 1px solid green;\">\n",
    "                <img src=\"underthecovers/images/shell.jpg\" width=\"100%\"style=\"padding: 0; margin: 0;\">\n",
    "                <figcaption> \n",
    "                    <div style=\"background-color: green; margin-right:auto; margin-left:auto; text-align: center;\">\n",
    "                        <i style=\"color: white\">B: Terminal 2</i>\n",
    "                    </div>\n",
    "                </figcaption>\n",
    "            </figure>\n",
    "            </div>\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr style=\"padding: 0; margin: 0;\">\n",
    "         <td colspan=\"2\" width=\"50%\" style=\"background-color: white; padding: 0; margin: 0;\" >\n",
    "           <div style=\"margin-right: auto; margin-left: auto;\">\n",
    "              <figure style=\"margin-right: auto; margin-left: auto;  width:50%; border: 1px solid blue;\">\n",
    "                  <img src=\"underthecovers/images/shell.jpg\" align=\"center\" width=\"100%\">\n",
    "                  <figcaption> \n",
    "                      <div style=\"background-color: blue; margin-right:auto; margin-left:auto; text-align: center;\">\n",
    "                          <i style=\"color: white\">A: Terminal 3</i>\n",
    "                      </div>\n",
    "                  </figcaption>\n",
    "              </figure>\n",
    "            </div>\n",
    "         </td>\n",
    "    </tr>\n",
    "    <caption align=\"bottom\" style=\"text-align: left; padding: 0; margin: 0;\" >\n",
    "        <i>Figure 4: 3 Terminals</i>\n",
    "    </caption>\n",
    "</table>\n",
    "\n",
    "This is a more complex example where we layout three images with a custom layout.  This <a href=\"#fig:3Terms\">figure</a> has sperate captions for each sub figure.  Alignment allows left, right and center.  Left and right permit text wrapping around the figure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa3470a8",
   "metadata": {},
   "source": [
    "<a id='htmtFig'></a>\n",
    "\n",
    "# htmlFig\n",
    "\n",
    "Like above but using htmlFig code to generate the html automatically.  In this case 3 figure in the top row and one in the bottom. \n",
    "\n",
    "htmFig provides an interface for generating html figures that use tables to permit complex layouts. \n",
    "However, the interface is designed to permit simple use cases and progressively more complex ones with little effort.\n",
    "The following demonstrates a progressive series of uses to get to the equivalent of the above example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b06971d",
   "metadata": {},
   "source": [
    "## Version 1: One image no styling or captions.\n",
    "\n",
    "A string file path is assumed to be an image file that will be used to generate the figure.\n",
    "We passing in a width to set the global size of the complete figure to be 25% of the page width. \n",
    "If we omitted this the default is 100%.  Note the function returns plain text html.  \n",
    "To display it from a code cell you will need to wrap the result in calls to `HTML()` or `Markdown()` and then\n",
    "`display()`.  I find `Markdown()` to be more flexible so that extra output can be markdown and styled appropriately.   Eg. the below output was generate by\n",
    "```\n",
    "display(Markdown(htmlFig(\"underthecovers/images/shell.jpg\",width=\"25%\")))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8641d77c",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(Markdown(htmlFig(\"underthecovers/images/shell.jpg\",width=\"25%\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0fe5ccf",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "## Version 2:  images on two rows no styling or captions\n",
    "\n",
    "By passing in a list of file paths we can get rows of images in our figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f16dc5d0",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(Markdown(htmlFig([\"underthecovers/images/shell.jpg\",\n",
    "                      \"underthecovers/images/shell.jpg\"],\n",
    "                     width=\"25%\"\n",
    "                     )))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fbaaf0c",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "## Version 3:  Adding column of images on two rows no styling or captions\n",
    "\n",
    "By passing in a list for a row specifies that the row should have multiple columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab46283a",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(Markdown(htmlFig([\n",
    "    # first row\n",
    "    [\"underthecovers/images/shell.jpg\",\n",
    "     \"underthecovers/images/shell.jpg\",\n",
    "     \"underthecovers/images/shell.jpg\"],\n",
    "    # second row\n",
    "    \"underthecovers/images/shell.jpg\"\n",
    "], width=\"50%\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76a36af3",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "## Version 4:  Provide styling for an img \n",
    "\n",
    "Image specific parameters can be passed by using an dictionary rather than just a file path \n",
    "for an image.  In this case you specify key and values.  For the file path the key is 'src'.\n",
    "The following documents many of the keys that you might want to override.  There are more but\n",
    "I don't have time to document them now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61e9f6f2",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(HTML(htmlFig([\n",
    "    # first row\n",
    "    [\"underthecovers/images/shell.jpg\",\n",
    "     \"underthecovers/images/shell.jpg\",\n",
    "     \"underthecovers/images/shell.jpg\"],\n",
    "    # second row\n",
    "    {'src':\"underthecovers/images/shell.jpg\",\n",
    "     # only one figure in this row colspan = 3\n",
    "     'colspan': '3',\n",
    "     # since colspan > 1 we overide the figure width \n",
    "     'figwidth': '33.3%',\n",
    "     # add colored border\n",
    "     'border': '1px solid red',\n",
    "     # add subfigure specific caption\n",
    "     'caption': 'C: Terminal 3',\n",
    "     # override color of font and background for caption\n",
    "     'capcolor': 'white',\n",
    "     'capbgcolor': 'red'\n",
    "    }   \n",
    "], width=\"50%\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaca616c",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "## Version 5:  More per image customization and a global settings\n",
    "\n",
    "Ok lets get things looking nice for all the images now by adding dictionaries for each image.  We also  add a caption for the entire figure, align it to the left (default is center) so that \n",
    "text below it will wrap to the right. So we also override the margins to give us 1 font width of\n",
    "spacing on the right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43bd6fd7",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(Markdown(htmlFig([\n",
    "    # first row\n",
    "    [\n",
    "      {'src':\"underthecovers/images/shell.jpg\",\n",
    "       'border': '1px solid green',\n",
    "       'caption': 'A: Terminal 1',\n",
    "       'capcolor': 'white',\n",
    "       'capbgcolor': 'green'\n",
    "      },\n",
    "      {'src':\"underthecovers/images/shell.jpg\",\n",
    "       'border': '1px solid blue',\n",
    "       'caption': 'B: Terminal 2',\n",
    "       'capcolor': 'white',\n",
    "       'capbgcolor': 'blue'\n",
    "      },\n",
    "      {'src':\"underthecovers/images/shell.jpg\",\n",
    "       'border': '1px solid orange',\n",
    "       'caption': 'C: Terminal 3',\n",
    "       'capcolor': 'white',\n",
    "       'capbgcolor': 'orange'\n",
    "      }\n",
    "    ],\n",
    "    # second row\n",
    "    {'src':\"underthecovers/images/shell.jpg\",\n",
    "     'colspan': '3',\n",
    "     'figwidth': '33.3%',\n",
    "     'cellmargin': '0 auto 0 auto',\n",
    "     'figmargin': '0 auto 0 auto',\n",
    "     'divmargin': '0 auto 0 auto',\n",
    "     'border': '1px solid red',\n",
    "     'caption': 'C: Terminal 4',\n",
    "     'capcolor': 'white',\n",
    "     'capbgcolor': 'red'\n",
    "    }   \n",
    "], \n",
    "    caption=\"Figure: A bunch of VT220 Terminals\",\n",
    "    align=\"left\",\n",
    "    margin=\"auto 1em auto auto\",\n",
    "    width=\"50%\") + '''\n",
    "Now this looks nice!  Last thing to do is add an id so that \n",
    "we can have references to the figure.  See the last version below.\n",
    "''' \n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33162e40",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "## Version 6:  Finally we can add an id so that we can reference the figure from other locations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa94c2df",
   "metadata": {
    "hide_input": false
   },
   "outputs": [],
   "source": [
    "display(Markdown(htmlFig(id=\"fig:4Terms\",\n",
    "    imgs=[\n",
    "    # first row\n",
    "    [\n",
    "      {'src':\"underthecovers/images/shell.jpg\",\n",
    "       'border': '1px solid green',\n",
    "       'caption': 'A: Terminal 1',\n",
    "       'capcolor': 'white',\n",
    "       'capbgcolor': 'green'\n",
    "      },\n",
    "      {'src':\"underthecovers/images/shell.jpg\",\n",
    "       'border': '1px solid blue',\n",
    "       'caption': 'B: Terminal 2',\n",
    "       'capcolor': 'white',\n",
    "       'capbgcolor': 'blue'\n",
    "      },\n",
    "      {'src':\"underthecovers/images/shell.jpg\",\n",
    "       'border': '1px solid orange',\n",
    "       'caption': 'C: Terminal 3',\n",
    "       'capcolor': 'white',\n",
    "       'capbgcolor': 'orange'\n",
    "      }\n",
    "    ],\n",
    "    # second row\n",
    "    {'src':\"underthecovers/images/shell.jpg\",\n",
    "     'colspan': '3',\n",
    "     'figwidth': '33.3%',\n",
    "     'border': '1px solid red',\n",
    "     'cellmargin': '0 auto 0 auto',\n",
    "     'figmargin': '0 auto 0 auto',\n",
    "     'divmargin': '0 auto 0 auto',\n",
    "     'caption': 'C: Terminal 4',\n",
    "     'capcolor': 'white',\n",
    "     'capbgcolor': 'red'\n",
    "    }   \n",
    "], \n",
    "    caption=\"Figure: A bunch of VT220 Terminals\",\n",
    "    align=\"left\",\n",
    "    margin=\"auto 1em auto auto\",\n",
    "    width=\"50%\") + '''\n",
    "By adding the id field we can now reference the figure using the id in the normal way\n",
    "<a href=\"#fig:4Terms\">figure of 4 Terminals</a>.  Go <a href=\"#farawayref\">here</a> to test coming back via \n",
    "the id.  Remember the reference is done by adding an html anchor tag with an href to be the id prefixed with the '#' \n",
    "Eg. `<a href=\"#fig:4Terms\">figure of 4 Terminals</a>`\n",
    "''' \n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12543507",
   "metadata": {},
   "source": [
    "## htmlFig different sized imgs on the same row\n",
    "\n",
    "When you need to have images of scale images on the same row you will want to override the cellwidth not the figwidth like the example below.   Use figwidth when you have a cell that is spanning multip columns as the above example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01a351cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(htmlFig(\n",
    "    [[\n",
    "        {'src':\"underthecovers/images/UnixL01_SHCHT/041SHLLChat.png\",\n",
    "         'cellwidth': '47.5%'},\n",
    "        {'src':\"underthecovers/images/terminalwins.png\",\n",
    "         'cellwidth': '52%'}\n",
    "     ]],\n",
    "    caption=\"Figure: Today terminal emulators running on our desktop act act as ASCII terminals. Every new window is a new terminal connection.\"\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf6a0fc",
   "metadata": {},
   "source": [
    "This example uses div instead of the table tags.   As far as I can tell this is becoming the preferred way of doing it but it seems to have more css dependencies and for the moment the python code I have written htmFig uses tables.  Maybe in the future will switch to using divs to simplify the html produced.  But not today.\n",
    "\n",
    "<div class=\"row\">\n",
    "     <div class=\"column\" style=\"float: left; width: 33.33%; padding: 2px;\">\n",
    "       <img src=\"underthecovers/images/UnixL01_SHCHT/06SHLLChat.png\" style=\"width:100%\">\n",
    "     </div>\n",
    "     <div class=\"column\" style=\"float: left; width: 33.33%; padding: 2px;\">\n",
    "        <img src=\"underthecovers/images/UnixL01_SHCHT/07SHLLChat.png\" style=\"width:100%\">\n",
    "     </div>\n",
    "     <div class=\"column\" style=\" float: left; width: 33.33%; padding: 2px;\">\n",
    "       <img src=\"underthecovers/images/UnixL01_SHCHT/08SHLLChat.png\"  style=\"width:100%\">\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7eaddd7",
   "metadata": {},
   "source": [
    "Can't see this in the note book but there is some html here that causes two images to overlap\n",
    "<div style=\"float:right\">\n",
    "<svg width=\"338\" height=\"104\">\n",
    "  <clipPath id=\"myContainer\">\n",
    "    <rect width=\"338\" height=\"104\"></rect>\n",
    "  </clipPath>\n",
    "  <image width=\"338\" height=\"104\" xlink:href=\"https://www.edoardovignati.it/wp-content/uploads/2021/06/overlap-sunset.png\" clip-path=\"url(#myContainer)\"></image>\n",
    "  <image width=\"338\" height=\"104\" xlink:href=\"https://www.edoardovignati.it/wp-content/uploads/2021/06/overlap-mountain.png\" clip-path=\"url(#myContainer)\"></image>\n",
    "</svg>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d99a93d",
   "metadata": {},
   "source": [
    "<a id='displayBytes'></a>\n",
    "\n",
    "# displayBytes\n",
    "\n",
    "displayBytes is some python code to make displaying binary bitwise data easier and flexible.\n",
    "\n",
    "The following are some examples of its use.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b88fa01-3911-42da-9932-dd23a7121f22",
   "metadata": {},
   "outputs": [],
   "source": [
    "X=np.uint16(0x8fff)\n",
    "#count=16\n",
    "#np.unpackbits(np.array([X], dtype=\">i4\").view(np.uint8))\n",
    "#np.unpackbits(np.array([X], dtype=\">i\"+str(np.int8(count/8))).view(np.uint8))\n",
    "displayBytes([X],numbits=16,columns=bitLabels(16), labels='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6896ac7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# as simple as it gets\n",
    "displayBytes([0xff],labels='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee42fdb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# no headers\n",
    "displayBytes([0xff],columns=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9226979",
   "metadata": {},
   "outputs": [],
   "source": [
    "# row labels\n",
    "displayBytes([0xff],labels=[\"ALL ON\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88a1a239",
   "metadata": {},
   "outputs": [],
   "source": [
    "# no column labes but row labels\n",
    "# displayBytes([0xff],columns=[],labels=[\"ALL ON\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71c16b90",
   "metadata": {},
   "outputs": [],
   "source": [
    "# how one might use display bytes to illustrate a problem \n",
    "# and solution\n",
    "# create some binary varaible to use in our examples\n",
    "# We use numpy types so that we get familar C/assembly byte types\n",
    "u=np.uint8(0x55)\n",
    "v=np.uint8(0xaa)\n",
    "w=np.bitwise_and(u,v)\n",
    "\n",
    "# Problem uses a blank \"answer\" row\n",
    "displayBytes([[u],[v],[\"\"]],labels=[\"u\",\"v\", \"u & v\"], td_height=\"85px\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8353fbee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# updated to include answer\n",
    "displayBytes([[u],[v],[w]],labels=[\"u\",\"v\", \"u & v\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2c6d1a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# something fancier that uses the ability to display other bit widths\n",
    "X=np.uint8(0xae)\n",
    "XL=np.bitwise_and(X,0xf)\n",
    "XH=np.bitwise_and(np.right_shift(X,4),0xf)\n",
    "displayBytes(bytes=[[X]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "024d3654",
   "metadata": {
    "cell_style": "split"
   },
   "outputs": [],
   "source": [
    "displayBytes(bytes=[[XH]], numbits=4,columns=[\"[$b_7$\", \"$b_6$\", \"$b_5$\", \"$b_4$]\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4283997f",
   "metadata": {
    "cell_style": "split"
   },
   "outputs": [],
   "source": [
    "displayBytes(bytes=[[XL]], numbits=4,columns=[\"[$b_3$\", \"$b_2$\", \"$b_1$\", \"$b_0$]\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19364924",
   "metadata": {
    "cell_style": "split"
   },
   "outputs": [],
   "source": [
    "displayStr(format(XH,\"1X\"), size=\"2em\", align=\"center\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc5bcb6f",
   "metadata": {
    "cell_style": "split"
   },
   "outputs": [],
   "source": [
    "displayStr(format(XL,\"1X\"), size=\"2em\", align=\"center\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bbffa59",
   "metadata": {},
   "outputs": [],
   "source": [
    "displayStr(\"0x\"+format(X,\"02X\"), size=\"2em\", align=\"center\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b793d12",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of using displayBytes to create an hex table\n",
    "displayBytes(bytes=[[i] for i in range(16)],td_font_size=\"1.5vw\", th_font_size=\"1vw\", numbits=4, columns=[\"[$b_3$\", \"$b_2$\", \"$b_1$\", \"$b_0$]\"], labels=[format(i,\"1X\")for i in range(16)], labelstitle=\"HEX\", center=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb1a752d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# an quick and dirty table of all byte values\n",
    "displayBytes(bytes=[[i] for i in range(256)], labelstitle=\"HEX (DEC)\", labels=[\"0x\"+format(i,\"02x\")+ \" (\" + format(i,\"03d\") +\")\" for i in range(256)], center=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea86e84d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# quick and dirty ascii table\n",
    "displayBytes(bytes=[[i] for i in range(128)], labels=[\"0x\"+format(i,\"02x\")+ \" (\" + format(i,\"03d\") +\") ASCII: \" + repr(chr(i)) for i in range(128)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f18859c",
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "display(Markdown('''# Experiment to see how to get another header above a set of colomns\n",
    "\n",
    "No support in displayBytes for this yet.\n",
    "'''))\n",
    "cols=[\"C\", \"B\", \"A\"]\n",
    "cols=list(zip(['BINARY']*3,cols))\n",
    "cols = pd.MultiIndex.from_tuples(cols, names=[' ', 'HEX']) \n",
    "df = pd.DataFrame([[1, 2, 3],[4, 5, 6],[7,8,9]],index=[\"r1\", \"r2\", \"r3\"], columns=cols)\n",
    "#df.rename_axis(\"hex\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ae2816",
   "metadata": {},
   "source": [
    "<a id=\"TermShellCmd\"></a>\n",
    "\n",
    "# TermShellCmd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "460faa45",
   "metadata": {},
   "outputs": [],
   "source": [
    "TermShellCmd(\"ls\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec6d089d",
   "metadata": {},
   "outputs": [],
   "source": [
    "TermShellCmd(\"stty -a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21f1bd2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "TermShellCmd(\"foobarblah one two three\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6feff2e1-cb32-42d5-8b8e-c863ac165acd",
   "metadata": {},
   "source": [
    "<a id='TtySession'></a>\n",
    "\n",
    "## TtySession, bashSession and the BashSession Class "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fe9a828-7cf5-4260-bf2e-ef33f53206fb",
   "metadata": {},
   "source": [
    "Simple examples: Starts a new shell for the commands and then kills the Shell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d7489cf-bec3-459e-ba13-213e04ae9179",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"ls\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2d47375-0904-48af-9579-efd9f0fae0a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"pwd\\nls\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b252f10-a0dd-4b57-adbf-5805b628717b",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"help\", height=\"10em\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a30c4e4-4179-48c4-bed8-cdfe5a1999ab",
   "metadata": {},
   "source": [
    "We can also create a persistent session and then can reuse it to run commands.  Important point is that the same shell process will be kept alive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6eb54fd8-70c4-4c5a-83b0-542ac3d815e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "session = bashSessionOpen()\n",
    "#print(session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41b9c9ba-c9b0-44ae-ae7d-ba1574b15787",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"pwd\\nls\\necho $x\", session=session, close=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "971d0a83-beaf-450e-80be-68a318ebae12",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"x=foo\", session=session, close=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f96780d9-c727-4401-b804-71100a592c2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"echo $x\", session=session, close=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e72a251c-82e4-41b5-bcc1-441098f13df4",
   "metadata": {},
   "source": [
    "You can render all the accumulated output from a session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8baa0e14-b641-4923-bc6b-834c85e01a20",
   "metadata": {},
   "outputs": [],
   "source": [
    "renderTtySessionOutput(session)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6de521f0-4feb-47b6-b1d3-9b0d81d12b2c",
   "metadata": {},
   "source": [
    "You can also use the raw write interface to send a raw bytes in stead of command lines to the bash process.  By default bytes will be sent one at a time with a 140 ms delay between characters.  After the data is sent and EOF will be generated.  All these defaults can be over ridden as needed.  See the code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d0352da-f803-4d03-93dc-f1a7cf51b7ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "output, session = bashSessionRawWrite('''cat\n",
    "hello\n",
    "''', session, 1)\n",
    "renderTtySessionOutput(session) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f423204-f352-461a-b86a-0238b93fbfd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "bashCmds(\"date\", session=session, close=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59bfcfcf-96f3-4351-abf6-3b243483c6da",
   "metadata": {},
   "source": [
    "To clean up a session -- kill the associated process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ad3196e-282e-4bc9-838e-4bdd5d5c44f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(TtyOpenSessions)\n",
    "closeTtySession(session)\n",
    "print(TtyOpenSessions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f95cec2-3c07-493f-afe1-78339fc601fd",
   "metadata": {},
   "source": [
    "Cleanup all sessions that have been created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd936ad1-a2d9-4f55-a2b7-ae8ac2c2a863",
   "metadata": {},
   "outputs": [],
   "source": [
    "closeAllOpenTtySessions()\n",
    "print(TtyOpenSessions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ada14475-a8de-4f6e-aa38-f2ba427325ae",
   "metadata": {
    "tags": []
   },
   "source": [
    "### The BashSession Class\n",
    "\n",
    "The BashSession make it easier to create and use sessions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fe8dd66-e585-4487-a6e3-aeda7af11466",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = BashSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15e75e3e-6782-4c44-a824-482241546748",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.run(\"ls\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d2534de-bb6f-4344-b68d-351ee23d54c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.run(\"y=2\\n((y++))\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1254424c-82bd-4229-97d9-14f2a8b0856b",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.run(\"echo $y\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65592687-6f24-406d-be00-2bbcabb56d81",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5592ded-6ae8-412b-bf99-1ee079cbf937",
   "metadata": {},
   "source": [
    "Or we can get running output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "721bb48a-52e1-4984-8100-46fa48e68d88",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.runAllOutput(\"echo 'foo'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a6cb64e-4239-4315-8d32-b3801b17df8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.runNoOutput(\"date\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b00d3965-8665-42fb-be13-2a175335ae96",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20e0f14f-3547-4bcb-a545-a6fb4763cf89",
   "metadata": {},
   "outputs": [],
   "source": [
    "bs.rawWrite('''cat\n",
    "hello\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9d853f9-0b37-4534-b491-ee6944853adf",
   "metadata": {},
   "source": [
    "Close the session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a753b9ee-3dbe-4ef5-ae6d-54faa2e5861d",
   "metadata": {},
   "outputs": [],
   "source": [
    "del bs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbea8b3f-e3e3-400a-88a3-15f453a7be12",
   "metadata": {},
   "source": [
    "We can use raw writes incrementally to interact with a program such as `cat`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0208199c-dc65-41db-a37d-4919d57a7619",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat = BashSession()\n",
    "cat.rawWriteAllOutput('''cat\n",
    "''', sendEOF=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a23eb2f-3843-4b20-a3a5-dc728330fa80",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.rawWrite('''hello\n",
    "''', sendEOF=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2747e723-a1db-4126-aff4-fba375854133",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.rawWrite('''goodbye\n",
    "''', sendEOF=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d82363f0-5c8d-4ac3-af9e-cdfc2e28e2c5",
   "metadata": {},
   "source": [
    "To send EOF and gather remaining output write a empty string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cc7ac7f-9a2d-4f54-a92e-60c7f760f2f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.rawWrite('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4b7ce34-a4f0-4104-9ed6-1b6b1f5ee27b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat.output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbccff59-d646-4dfb-966f-b570a7ac42ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "del cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4f5f350-d783-4d62-8e8b-85fdf8c2527a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(TtyOpenSessions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09e8ccc1-f868-4abf-9648-d635b4bd646a",
   "metadata": {},
   "outputs": [],
   "source": [
    "closeAllOpenTtySessions()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff6d7de4",
   "metadata": {
    "hide_input": true
   },
   "source": [
    "<a id=\"MDBox\"></a>\n",
    "# MDBox, FileMDBox, FileCodeBox"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2088d6ea",
   "metadata": {},
   "source": [
    "# Raw Markdown and html\n",
    "\n",
    "<div style=\"width:100%; height:200px; overflow: auto;\" >\n",
    "\n",
    "#### lots of little programs that naturally feel like built-ins\n",
    "- `echo $PATH`\n",
    "   - `ls /bin`\n",
    "      - see `ls` there?\n",
    "      - notice `bash` ;-)\n",
    "   - how can we figure out where `ls` is coming from\n",
    "      - bash has handy built in called `type`\n",
    "        - `help type`\n",
    "        - `type -a ls`\n",
    "   - by tradition all preinstalled programs should have a manual page\n",
    "        - `man ls`\n",
    "        - `man bash`\n",
    "        - `man <something in /bin>`\n",
    "\n",
    "## extendent the shell is easy\n",
    "- set `PATH=\"$HOME/bin:$PATH`\n",
    "  - putting a program or script called `foobar` in `$HOME/bin`\n",
    "    - now `$ foobar` will feel like a built-in\n",
    "\n",
    "## overide what's there\n",
    "- putting `ls` in `$HOME/bin` will now overide the others\n",
    "- Even override a built in\n",
    "  - `function echo() { builtin echo -n \"myecho: \"; builtin echo $@ }`\n",
    "  - will talk about this one later\n",
    "\n",
    "## natural ways of composing and extending via programming\n",
    "Two important ones:\n",
    "\n",
    "1. shell scripts\n",
    "2. command pipelines\n",
    "\n",
    "### shell provide natural model for composition\n",
    "\n",
    "- put shell commands in a file : eg put this in `foobar`\n",
    "``` bash\n",
    "#!/bin/bash\n",
    "echo \"My first shell script\"\n",
    "echo \"foobar\"\n",
    "```\n",
    "- mark as executable\n",
    "- now shell will be able to run `foobar`\n",
    "\n",
    "### pipeline: allow programs to be easily composed\n",
    "\n",
    "- `ls -1` list files - one per line\n",
    "- `wc -l` counts lines\n",
    "- `ls -l | wc -l` - tells us how man files in this direcotyr\n",
    "- `ls -l /bin | wc -l`\n",
    "-  or to get really fancy\n",
    "  - exploit more knowledge about shell expansion abilities\n",
    "  - `echo $PATH`\n",
    "  - `echo ${PATH//:/ }`\n",
    "  - `ls -1 ${PATH//:/ } | wc -l`\n",
    "     - what do you think this did?  \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8474be97",
   "metadata": {},
   "source": [
    "# using a markdown file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f043158d",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(Markdown(FileMDBox(title=\"# Intro to the Book\", file=\"underthecovers/intro_tb.md\", w=\"100%\", h=\"200px\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a4f3397",
   "metadata": {},
   "source": [
    "### In line markdown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3e77412",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(Markdown(MDBox(w=\"100%\", h=\"200px\",\n",
    "                      contents='''\n",
    "#### lots of little programs that naturally feel like built-ins\n",
    "- `echo $PATH`\n",
    "   - `ls /bin`\n",
    "      - see `ls` there?\n",
    "      - notice `bash` ;-)\n",
    "   - how can we figure out where `ls` is coming from\n",
    "      - bash has handy built in called `type`\n",
    "        - `help type`\n",
    "        - `type -a ls`\n",
    "   - by tradition all preinstalled programs should have a manual page\n",
    "        - `man ls`\n",
    "        - `man bash`\n",
    "        - `man <something in /bin>`\n",
    "\n",
    "## extendent the shell is easy\n",
    "- set `PATH=\"$HOME/bin:$PATH`\n",
    "  - putting a program or script called `foobar` in `$HOME/bin`\n",
    "    - now `$ foobar` will feel like a built-in\n",
    "\n",
    "## overide what's there\n",
    "- putting `ls` in `$HOME/bin` will now overide the others\n",
    "- Even override a built in\n",
    "  - `function echo() { builtin echo -n \"myecho: \"; builtin echo $@ }`\n",
    "  - will talk about this one later\n",
    "\n",
    "## natural ways of composing and extending via programming\n",
    "Two important ones:\n",
    "\n",
    "1. shell scripts\n",
    "2. command pipelines\n",
    "\n",
    "### shell provide natural model for composition\n",
    "\n",
    "- put shell commands in a file : eg put this in `foobar`\n",
    "``` bash\n",
    "#!/bin/bash\n",
    "echo \"My first shell script\"\n",
    "echo \"foobar\"\n",
    "```\n",
    "- mark as executable\n",
    "- now shell will be able to run `foobar`\n",
    "\n",
    "### pipeline: allow programs to be easily composed\n",
    "\n",
    "- `ls -1` list files - one per line\n",
    "- `wc -l` counts lines\n",
    "- `ls -l | wc -l` - tells us how man files in this direcotyr\n",
    "- `ls -l /bin | wc -l`\n",
    "-  or to get really fancy\n",
    "  - exploit more knowledge about shell expansion abilities\n",
    "  - `echo $PATH`\n",
    "  - `echo ${PATH//:/ }`\n",
    "  - `ls -1 ${PATH//:/ } | wc -l`\n",
    "     - what do you think this did?                        \n",
    "                      '''\n",
    "                      )))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfd8f580",
   "metadata": {},
   "source": [
    "### A version to make displaying code FileCodeBox (uses MDBox) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e424688d",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(Markdown(FileCodeBox(\n",
    "    file=\"underthecovers/src/shlldirfile.sh\", \n",
    "    lang=\"bash\", \n",
    "    title='# File and Directories - Part I',\n",
    "    w=\"100%\",\n",
    "    h=\"200px\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4641d6d",
   "metadata": {},
   "source": [
    "# Rough notes and playing around\n",
    "\n",
    "Operate first url to directly spawn bu image\n",
    "\n",
    "https://jupyterhub-opf-jupyterhub.apps.zero.massopen.cloud/hub/spawn?bu-cs-jupyter-book:latest\n",
    "\n",
    "Working with Tom on operate first slack the following url git clone \n",
    "\n",
    "\n",
    "https://jupyterhub-opf-jupyterhub.apps.zero.massopen.cloud/hub/user-redirect/git-pull?bu-cs-jupyter-book%3Alatest=&repo=https%3A%2F%2Fgithub.com%2Fjappavoo%2FUndertheCovers&urlpath=tree%2FUndertheCovers%2Funderthecovers%2FL00_210_JA.ipynb&branch=main\n",
    "\n",
    "\n",
    "How to set \n",
    "```\n",
    "c.Spawner.default_url = '/data/jupyterhub/jupyterhub_notebooks'\n",
    "c.Spawner.notebook_dir = '/data/jupyterhub/jupyterhub_notebooks'\n",
    "```\n",
    "\n",
    "Hmmm it now seems that we are launching bu-cs container and home dir ends up being joyvan need to look at start up script as there is a lot of logic in it to adjust user and group\n",
    "\n",
    "At this point the running joyvan user on operatefirst seems to be some system assigned id with group root which leads to not having the right permissions to write home dir\n",
    "\n",
    "For the moment a hack might be to make the joyvan homdir and subdirs group root"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8529effb",
   "metadata": {},
   "source": [
    "\n",
    "Have managed to get the notebook running and now stuggling to get embedded terminals work.\n",
    "\n",
    "1) calling the api to create the terminals works after getting the api token from the environment variables\n",
    "2) however running into security setting need to be fixed\n",
    "  ```\n",
    "  \n",
    "  c.Spawner.args = ['--NotebookApp.tornado_settings={\"headers\":{\"Content-Security-Policy\": \"frame-ancestors * self host_ip:port\"}}']   \n",
    "\n",
    "c.JupyterHub.tornado_settings = { 'headers': { 'Content-Security-Policy': \"frame-ancestors * self host_ip:port\"} }\n",
    "```\n",
    "see: \n",
    "- https://github.com/jupyterhub/jupyterhub/issues/379#issuecomment-353607675\n",
    "- https://github.com/jupyterhub/the-littlest-jupyterhub/issues/312#issuecomment-590679516\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be2a2ba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(base_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af1d6835",
   "metadata": {},
   "outputs": [],
   "source": [
    "showET()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98c21c5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = \"foo\"\n",
    "a += \" bar\"\n",
    "print(a)\n",
    "img = {'src':\"file.jpg\", 'blah' : 'a b cm'}\n",
    "\n",
    "if img.get('src'):\n",
    "    print(img.get('src'))\n",
    "    \n",
    "#if img.get('blah'):\n",
    "blah=img.get('blah')\n",
    "if blah:\n",
    "    print(blah)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5135f73d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i underthecovers/python/common.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9732e96",
   "metadata": {},
   "outputs": [],
   "source": [
    "a=12"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48bd5783",
   "metadata": {
    "variables": {
     "a": "12"
    }
   },
   "source": [
    "what is the value{{a}}mean?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03f4dc8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "html=htmlFig(\n",
    "        imgs=[\n",
    "            [\n",
    "                {\n",
    "                'src' :\"underthecovers/images/shell.jpg\", \n",
    "                'border':\"1px solid red\",\n",
    "                'caption': \"A: Terminal 1\",\n",
    "                'capcolor': 'white',\n",
    "                'capbgcolor': 'red',\n",
    "                'extratxt': 'See <a href=\"https://en.wikipedia.org/wiki/VT220\">vt220</a>',\n",
    "                'extrafont': \"1vmin\",\n",
    "                'extracolor': \"green\"\n",
    "                }, \n",
    "                {\n",
    "                'src':\"underthecovers/images/shell.jpg\", \n",
    "                'border':\"1px solid green\",\n",
    "                'caption': \"B: Terminal 2\",\n",
    "                'capcolor': 'white',\n",
    "                'capbgcolor': 'green',\n",
    "                },\n",
    "                {\n",
    "                'src':\"underthecovers/images/shell.jpg\", \n",
    "                'border':\"1px solid orange\",\n",
    "                'caption': \"C: Terminal 3\",\n",
    "                'capcolor': 'white',\n",
    "                'capbgcolor': 'orange',\n",
    "                }\n",
    "            ],\n",
    "            { \n",
    "                'src': 'underthecovers/images/shell.jpg',\n",
    "                'border':\"1px solid blue\",\n",
    "                'caption': \"D: Terminal 4\",\n",
    "                'capcolor': 'white',\n",
    "                'capbgcolor': 'blue',\n",
    "                'colspan': '3',\n",
    "                'width': '33.3%'\n",
    "            }\n",
    "        ], \n",
    "        align=\"left\",\n",
    "        width=\"50%\",\n",
    "       caption='<center>Figure 1: Some <a href=\"https://en.wikipedia.org/wiki/VT220\">vt220</a> ASCII Terminals</center>',\n",
    "#        subcaption=\"B: Terminal 2\", \n",
    "#        extratxt=,\n",
    "#        border=\"1px solid red\"\n",
    "    )\n",
    "display(HTML(html))\n",
    "#print(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8c94e67",
   "metadata": {},
   "source": [
    "This is some text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "044a777d",
   "metadata": {},
   "outputs": [],
   "source": [
    "## def toImg(i):\n",
    "    # if a string the convert to img\n",
    "    if (type(i) == type(\"\")):\n",
    "        i={\"src\":i}\n",
    "\n",
    "    if ((not type(i) == type({})) or (not 'src' in i)):\n",
    "        raise ValueError('img must at have a src specified')\n",
    "        \n",
    "    return i\n",
    "\n",
    "# a list of imgs\n",
    "def toImgs(i):\n",
    "    # already a list so don't do anyting\n",
    "    if type(i) == type([]):\n",
    "        return i\n",
    "    \n",
    "    i=[toImg(i)]\n",
    "    \n",
    "    return i\n",
    "\n",
    "def testsrc(imgs):\n",
    "    imgs=toImgs(imgs)\n",
    "    rows = len(imgs)\n",
    "    maxcols = 1\n",
    "\n",
    "    # calculate the maximum number of columns\n",
    "    # and build new list\n",
    "    rows = []\n",
    "    for r in imgs:\n",
    "        r = toImgs(r)\n",
    "        rows.append(r)\n",
    "        if (len(r)>maxcols):\n",
    "            maxcols = len(r);\n",
    "    \n",
    "    for r in rows:\n",
    "        cols=len(r)\n",
    "        print(\"row: \" + str(cols) + \"/\" + str(maxcols))\n",
    "        for i in r:\n",
    "            print(\"cells:\");\n",
    "            img=toImg(i)\n",
    "            print(img)\n",
    "    \n",
    "testsrc(imgs=[[{'src':\"file.jpg\",'caption':\"a picture\"},\"file.jpg\"],\"file.jpg\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e770611",
   "metadata": {},
   "source": [
    "https://jupyterhub.github.io/nbgitpuller/link?hub=https://127.0.0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9083d1",
   "metadata": {},
   "source": [
    "<img src=\"underthecovers/images/SLS_WPIM.svg\" width=\"100%\" style=\"vertical-align:top; float:middle; border:0px; margin: 10px 0px; background-color: #191919;\">\n",
    "<img src=\"underthecovers/images/SLS_TheMachine.svg\" width=\"100%\" style=\"vertical-align:top; float:middle; border:0px; margin: 0px 0px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7eb75c4",
   "metadata": {},
   "source": [
    "In the <a href=\"#vt220Fig\">figure</a> we see a classic ASCII terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae6bc05",
   "metadata": {},
   "source": [
    "```{glossary}\n",
    "Term one\n",
    "  An indented explanation of term 1\n",
    "\n",
    "A second term\n",
    "  An indented explanation of term2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b25b3412",
   "metadata": {},
   "source": [
    "<div class=\"admonition note\" name=\"html-admonition\" style=\"background: lightgreen; padding: 10px\">\n",
    "<p class=\"title\">This is the **title**</p>\n",
    "This is the *content*\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "759a7746",
   "metadata": {},
   "source": [
    "<p id=\"farawayref\">\n",
    "testing a reference to a figure up in the page <a href=\"#fig:4Terms\">Four Terminal Figure</a>\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc3bab8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pexpect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9f7193b-20bc-44a4-8940-c8339d86eabe",
   "metadata": {},
   "outputs": [],
   "source": [
    "shell.close()\n",
    "shell = pexpect.spawn('/bin/bash -l -i')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4900e1c7-0f84-45cc-9735-97b5b6fc83ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(shell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d092e831-acd6-41a3-b4a2-1cbb7df3a2ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "shell.sendline()\n",
    "shell.expect('\\r\\n$ ')\n",
    "print(shell.before)\n",
    "print(shell.after)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c324d528-457c-47ea-b131-3ba4b4fc427b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(shell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a5d2d62-acac-4742-8be5-2c26e2e6db18",
   "metadata": {},
   "outputs": [],
   "source": [
    "shell.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9fb5565-3842-4847-acba-eaea14fe51a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(shell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3eed3ef-9f07-4e87-a933-7056e7e1acca",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, update_display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dff53a3c-d96b-462a-a2a2-190e445b0bb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "MIMETYPE = \"application/x-drawio\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c5e0718-c7aa-4e71-bf5e-f406b9430e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_drawio(xml, **kwargs):\n",
    "    \"\"\" Send some xml to the frontend\n",
    "    \n",
    "    Get a handle to update later by calling `display_drawio` with `display_id=True` \n",
    "    \"\"\"\n",
    "    return display({MIMETYPE: xml}, raw=True, **kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65aa4c50-574e-434b-9b9f-d9d0de7e1c95",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!ls\n",
    "handle = display_drawio(\"/Work/UndertheCovers/underthecovers/images/src/BashProcessExecution.drawio\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c114da5b-d6b2-487f-9d8d-d42fe47badad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipydrawio_widgets import Diagram\n",
    "\n",
    "diagram = Diagram()\n",
    "diagram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b6a3264-e6e3-431b-897e-9e896457bbda",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}