Table Transformer - DETR 2CSV.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"provenance":[{"file_id":"1lLRyBr7WraGdUJm-urUm_utArw6SkoCJ","timestamp":1664264535448},{"file_id":"https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb","timestamp":1634197440740}],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["\n","# **TABLE TRANSFORMER - DETR**\n","Two functionalities implemented: Table Detection (TD) and Table Structure Recognition (TSR)\n","\n","This notebook it is based in https://huggingface.co/spaces/SalML/TableTransformer2CSV/tree/main to obtain and postprocess the structure of the table obtained with table transformer from https://colab.research.google.com/drive/1lLRyBr7WraGdUJm-urUm_utArw6SkoCJ?usp=sharing. It is also based on https://github.com/microsoft/table-transformer "],"metadata":{"id":"HTd7TyeHYQi9"}},{"cell_type":"markdown","source":["## **1.- Dependencies**"],"metadata":{"id":"lmZZ1GSCakzS"}},{"cell_type":"code","source":["!git clone https://huggingface.co/spaces/SalML/TableTransformer2CSV"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-ryrXB1A_8AN","executionInfo":{"status":"ok","timestamp":1665993737361,"user_tz":-120,"elapsed":1277,"user":{"displayName":"Emilio Gómez García","userId":"12452742652222446865"}},"outputId":"530774cb-7359-4675-ab27-e627549decc0"},"execution_count":1,"outputs":[{"output_type":"stream","name":"stdout","text":["Cloning into 'TableTransformer2CSV'...\n","remote: Enumerating objects: 53, done.\u001b[K\n","remote: Counting objects: 100% (53/53), done.\u001b[K\n","remote: Compressing objects: 100% (52/52), done.\u001b[K\n","remote: Total 53 (delta 23), reused 2 (delta 1), pack-reused 0\u001b[K\n","Unpacking objects: 100% (53/53), done.\n"]}]},{"cell_type":"code","source":["!pip install -r ./TableTransformer2CSV/requirements.txt"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"id":"42A1EYOMAJQP","executionInfo":{"status":"ok","timestamp":1665994007275,"user_tz":-120,"elapsed":269916,"user":{"displayName":"Emilio Gómez García","userId":"12452742652222446865"}},"outputId":"cfe5f98e-3b77-42a7-cd5d-422f26bf9024"},"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n","Obtaining transformers from git+https://github.com/nielsrogge/transformers.git@d34f7e6ffbb911d39465173ef2b35ba147ef58a9#egg=transformers (from -r ./TableTransformer2CSV/requirements.txt (line 45))\n","  Cloning https://github.com/nielsrogge/transformers.git (to revision d34f7e6ffbb911d39465173ef2b35ba147ef58a9) to ./src/transformers\n","  Running command git clone -q https://github.com/nielsrogge/transformers.git /content/src/transformers\n","  Running command git rev-parse -q --verify 'sha^d34f7e6ffbb911d39465173ef2b35ba147ef58a9'\n","  Running command git fetch -q https://github.com/nielsrogge/transformers.git d34f7e6ffbb911d39465173ef2b35ba147ef58a9\n","  Running command git checkout -q d34f7e6ffbb911d39465173ef2b35ba147ef58a9\n","  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n","  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n","    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n","Collecting Cython==0.29.14\n","  Downloading Cython-0.29.14-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)\n","\u001b[K     |████████████████████████████████| 2.1 MB 4.3 MB/s \n","\u001b[?25hCollecting dask==2021.3.1\n","  Downloading dask-2021.3.1-py3-none-any.whl (935 kB)\n","\u001b[K     |████████████████████████████████| 935 kB 44.3 MB/s \n","\u001b[?25hCollecting datasets==1.18.3\n","  Downloading datasets-1.18.3-py3-none-any.whl (311 kB)\n","\u001b[K     |████████████████████████████████| 311 kB 57.4 MB/s \n","\u001b[?25hCollecting Flask==2.0.1\n","  Downloading Flask-2.0.1-py3-none-any.whl (94 kB)\n","\u001b[K     |████████████████████████████████| 94 kB 3.7 MB/s \n","\u001b[?25hCollecting GitPython==3.1.26\n","  Downloading GitPython-3.1.26-py3-none-any.whl (180 kB)\n","\u001b[K     |████████████████████████████████| 180 kB 63.2 MB/s \n","\u001b[?25hRequirement already satisfied: imutils==0.5.4 in /usr/local/lib/python3.7/dist-packages (from -r ./TableTransformer2CSV/requirements.txt (line 6)) (0.5.4)\n","Collecting multiprocess==0.70.12.2\n","  Downloading multiprocess-0.70.12.2-py37-none-any.whl (112 kB)\n","\u001b[K     |████████████████████████████████| 112 kB 59.8 MB/s \n","\u001b[?25hCollecting numba==0.54.1\n","  Downloading numba-0.54.1-cp37-cp37m-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.3 MB)\n","\u001b[K     |████████████████████████████████| 3.3 MB 51.3 MB/s \n","\u001b[?25hCollecting numexpr==2.7.3\n","  Downloading numexpr-2.7.3-cp37-cp37m-manylinux2010_x86_64.whl (471 kB)\n","\u001b[K     |████████████████████████████████| 471 kB 60.6 MB/s \n","\u001b[?25hCollecting numpy==1.20.3\n","  Downloading numpy-1.20.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.3 MB)\n","\u001b[K     |████████████████████████████████| 15.3 MB 35.3 MB/s \n","\u001b[?25hCollecting oauthlib==3.1.0\n","  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)\n","\u001b[K     |████████████████████████████████| 147 kB 59.4 MB/s \n","\u001b[?25hRequirement already satisfied: opencv-contrib-python==4.6.0.66 in /usr/local/lib/python3.7/dist-packages (from -r ./TableTransformer2CSV/requirements.txt (line 12)) (4.6.0.66)\n","Collecting openpyxl==3.0.7\n","  Downloading openpyxl-3.0.7-py2.py3-none-any.whl (243 kB)\n","\u001b[K     |████████████████████████████████| 243 kB 58.1 MB/s \n","\u001b[?25hCollecting Pillow==9.0.1\n","  Downloading Pillow-9.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)\n","\u001b[K     |████████████████████████████████| 4.3 MB 40.3 MB/s \n","\u001b[?25hCollecting plotly==4.14.3\n","  Downloading plotly-4.14.3-py2.py3-none-any.whl (13.2 MB)\n","\u001b[K     |████████████████████████████████| 13.2 MB 25.4 MB/s \n","\u001b[?25hCollecting ply==3.11\n","  Downloading ply-3.11-py2.py3-none-any.whl (49 kB)\n","\u001b[K     |████████████████████████████████| 49 kB 6.8 MB/s \n","\u001b[?25hCollecting protobuf==3.14.0\n","  Downloading protobuf-3.14.0-cp37-cp37m-manylinux1_x86_64.whl (1.0 MB)\n","\u001b[K     |████████████████████████████████| 1.0 MB 44.6 MB/s \n","\u001b[?25hCollecting psutil==5.8.0\n","  Downloading psutil-5.8.0-cp37-cp37m-manylinux2010_x86_64.whl (296 kB)\n","\u001b[K     |████████████████████████████████| 296 kB 69.8 MB/s \n","\u001b[?25hCollecting pyarrow==7.0.0\n","  Downloading pyarrow-7.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.7 MB)\n","\u001b[K     |████████████████████████████████| 26.7 MB 1.4 MB/s \n","\u001b[?25hCollecting pydantic==1.7.3\n","  Downloading pydantic-1.7.3-cp37-cp37m-manylinux2014_x86_64.whl (9.1 MB)\n","\u001b[K     |████████████████████████████████| 9.1 MB 39.7 MB/s \n","\u001b[?25hCollecting pydeck==0.7.1\n","  Downloading pydeck-0.7.1-py2.py3-none-any.whl (4.3 MB)\n","\u001b[K     |████████████████████████████████| 4.3 MB 52.1 MB/s \n","\u001b[?25hCollecting PyDictionary==2.0.1\n","  Downloading PyDictionary-2.0.1-py3-none-any.whl (6.1 kB)\n","Collecting pydot==1.4.2\n","  Downloading pydot-1.4.2-py2.py3-none-any.whl (21 kB)\n","Collecting pymongo==4.0.2\n","  Downloading pymongo-4.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (451 kB)\n","\u001b[K     |████████████████████████████████| 451 kB 64.1 MB/s \n","\u001b[?25hCollecting Pympler==1.0.1\n","  Downloading Pympler-1.0.1-py3-none-any.whl (164 kB)\n","\u001b[K     |████████████████████████████████| 164 kB 71.1 MB/s \n","\u001b[?25hCollecting PyMuPDF==1.20.2\n","  Downloading PyMuPDF-1.20.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.8 MB)\n","\u001b[K     |████████████████████████████████| 8.8 MB 47.1 MB/s \n","\u001b[?25hCollecting pyperclip==1.8.2\n","  Downloading pyperclip-1.8.2.tar.gz (20 kB)\n","Collecting pyppeteer==0.2.5\n","  Downloading pyppeteer-0.2.5-py3-none-any.whl (87 kB)\n","\u001b[K     |████████████████████████████████| 87 kB 6.3 MB/s \n","\u001b[?25hCollecting pyquery==1.4.3\n","  Downloading pyquery-1.4.3-py3-none-any.whl (22 kB)\n","Collecting pyreadline3==3.3\n","  Downloading pyreadline3-3.3-py3-none-any.whl (95 kB)\n","\u001b[K     |████████████████████████████████| 95 kB 4.5 MB/s \n","\u001b[?25hCollecting pytesseract==0.3.10\n","  Downloading pytesseract-0.3.10-py3-none-any.whl (14 kB)\n","Collecting pytz-deprecation-shim==0.1.0.post0\n","  Downloading pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)\n","Collecting PyWavelets==1.1.1\n","  Downloading PyWavelets-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (4.4 MB)\n","\u001b[K     |████████████████████████████████| 4.4 MB 49.9 MB/s \n","\u001b[?25hCollecting PyYAML==5.4.1\n","  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n","\u001b[K     |████████████████████████████████| 636 kB 64.1 MB/s \n","\u001b[?25hCollecting scipy==1.4.1\n","  Downloading scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1 MB)\n","\u001b[K     |████████████████████████████████| 26.1 MB 1.3 MB/s \n","\u001b[?25hCollecting seaborn==0.11.1\n","  Downloading seaborn-0.11.1-py3-none-any.whl (285 kB)\n","\u001b[K     |████████████████████████████████| 285 kB 52.0 MB/s \n","\u001b[?25hCollecting sklearn==0.0\n","  Downloading sklearn-0.0.tar.gz (1.1 kB)\n","Collecting streamlit==1.5.1\n","  Downloading streamlit-1.5.1-py2.py3-none-any.whl (9.7 MB)\n","\u001b[K     |████████████████████████████████| 9.7 MB 22.1 MB/s \n","\u001b[?25hCollecting timm==0.6.7\n","  Downloading timm-0.6.7-py3-none-any.whl (509 kB)\n","\u001b[K     |████████████████████████████████| 509 kB 64.0 MB/s \n","\u001b[?25hCollecting tokenizers==0.12.1\n","  Downloading tokenizers-0.12.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n","\u001b[K     |████████████████████████████████| 6.6 MB 34.0 MB/s \n","\u001b[?25hRequirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.7/dist-packages (from -r ./TableTransformer2CSV/requirements.txt (line 41)) (0.10.2)\n","Collecting toolz==0.11.1\n","  Downloading toolz-0.11.1-py3-none-any.whl (55 kB)\n","\u001b[K     |████████████████████████████████| 55 kB 3.8 MB/s \n","\u001b[?25hCollecting torch==1.10.0\n","  Downloading torch-1.10.0-cp37-cp37m-manylinux1_x86_64.whl (881.9 MB)\n","\u001b[K     |██████████████████████████████▎ | 834.1 MB 1.3 MB/s eta 0:00:38tcmalloc: large alloc 1147494400 bytes == 0x3a904000 @  0x7faa3a15b615 0x58e046 0x4f2e5e 0x4d19df 0x51b31c 0x5b41c5 0x58f49e 0x51b221 0x5b41c5 0x58f49e 0x51837f 0x4cfabb 0x517aa0 0x4cfabb 0x517aa0 0x4cfabb 0x517aa0 0x4ba70a 0x538136 0x590055 0x51b180 0x5b41c5 0x58f49e 0x51837f 0x5b41c5 0x58f49e 0x51740e 0x58f2a7 0x517947 0x5b41c5 0x58f49e\n","\u001b[K     |████████████████████████████████| 881.9 MB 18 kB/s \n","\u001b[?25hCollecting torchvision==0.11.1\n","  Downloading torchvision-0.11.1-cp37-cp37m-manylinux1_x86_64.whl (23.3 MB)\n","\u001b[K     |████████████████████████████████| 23.3 MB 86.2 MB/s \n","\u001b[?25hCollecting urllib3==1.26.7\n","  Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)\n","\u001b[K     |████████████████████████████████| 138 kB 58.2 MB/s \n","\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (3.8.0)\n","Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (2.23.0)\n","Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (21.3)\n","Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (2022.6.2)\n","Collecting huggingface-hub<1.0,>=0.9.0\n","  Downloading huggingface_hub-0.10.1-py3-none-any.whl (163 kB)\n","\u001b[K     |████████████████████████████████| 163 kB 78.2 MB/s \n","\u001b[?25hRequirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (5.0.0)\n","Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (4.64.1)\n","Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from dask==2021.3.1->-r ./TableTransformer2CSV/requirements.txt (line 2)) (1.5.0)\n","Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from dask==2021.3.1->-r ./TableTransformer2CSV/requirements.txt (line 2)) (2022.8.2)\n","Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.7/dist-packages (from dask==2021.3.1->-r ./TableTransformer2CSV/requirements.txt (line 2)) (1.3.0)\n","Collecting xxhash\n","  Downloading xxhash-3.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n","\u001b[K     |████████████████████████████████| 212 kB 45.3 MB/s \n","\u001b[?25hRequirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (3.8.3)\n","Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (1.3.5)\n","Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (0.3.5.1)\n","Collecting itsdangerous>=2.0\n","  Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)\n","Collecting Werkzeug>=2.0\n","  Downloading Werkzeug-2.2.2-py3-none-any.whl (232 kB)\n","\u001b[K     |████████████████████████████████| 232 kB 69.6 MB/s \n","\u001b[?25hRequirement already satisfied: click>=7.1.2 in /usr/local/lib/python3.7/dist-packages (from Flask==2.0.1->-r ./TableTransformer2CSV/requirements.txt (line 4)) (7.1.2)\n","Collecting Jinja2>=3.0\n","  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)\n","\u001b[K     |████████████████████████████████| 133 kB 60.8 MB/s \n","\u001b[?25hRequirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from GitPython==3.1.26->-r ./TableTransformer2CSV/requirements.txt (line 5)) (4.1.1)\n","Collecting gitdb<5,>=4.0.1\n","  Downloading gitdb-4.0.9-py3-none-any.whl (63 kB)\n","\u001b[K     |████████████████████████████████| 63 kB 2.0 MB/s \n","\u001b[?25hCollecting llvmlite<0.38,>=0.37.0rc1\n","  Downloading llvmlite-0.37.0-cp37-cp37m-manylinux2014_x86_64.whl (26.3 MB)\n","\u001b[K     |████████████████████████████████| 26.3 MB 1.3 MB/s \n","\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba==0.54.1->-r ./TableTransformer2CSV/requirements.txt (line 8)) (57.4.0)\n","Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.7/dist-packages (from openpyxl==3.0.7->-r ./TableTransformer2CSV/requirements.txt (line 13)) (1.1.0)\n","Collecting retrying>=1.3.3\n","  Downloading retrying-1.3.3.tar.gz (10 kB)\n","Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from plotly==4.14.3->-r ./TableTransformer2CSV/requirements.txt (line 15)) (1.15.0)\n","Requirement already satisfied: ipykernel>=5.1.2 in /usr/local/lib/python3.7/dist-packages (from pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (5.3.4)\n","Requirement already satisfied: ipywidgets>=7.0.0 in /usr/local/lib/python3.7/dist-packages (from pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (7.7.1)\n","Requirement already satisfied: traitlets>=4.3.2 in /usr/local/lib/python3.7/dist-packages (from pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (5.1.1)\n","Requirement already satisfied: bs4 in /usr/local/lib/python3.7/dist-packages (from PyDictionary==2.0.1->-r ./TableTransformer2CSV/requirements.txt (line 22)) (0.0.1)\n","Collecting goslate\n","  Downloading goslate-1.5.4.tar.gz (14 kB)\n","Requirement already satisfied: pyparsing>=2.1.4 in /usr/local/lib/python3.7/dist-packages (from pydot==1.4.2->-r ./TableTransformer2CSV/requirements.txt (line 23)) (3.0.9)\n","Collecting websockets<9.0,>=8.1\n","  Downloading websockets-8.1-cp37-cp37m-manylinux2010_x86_64.whl (79 kB)\n","\u001b[K     |████████████████████████████████| 79 kB 8.0 MB/s \n","\u001b[?25hCollecting importlib-metadata\n","  Downloading importlib_metadata-2.1.3-py2.py3-none-any.whl (10 kB)\n","Requirement already satisfied: appdirs<2.0.0,>=1.4.3 in /usr/local/lib/python3.7/dist-packages (from pyppeteer==0.2.5->-r ./TableTransformer2CSV/requirements.txt (line 28)) (1.4.4)\n","Collecting pyee<9.0.0,>=8.1.0\n","  Downloading pyee-8.2.2-py2.py3-none-any.whl (12 kB)\n","Collecting cssselect>0.7.9\n","  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)\n","Requirement already satisfied: lxml>=2.1 in /usr/local/lib/python3.7/dist-packages (from pyquery==1.4.3->-r ./TableTransformer2CSV/requirements.txt (line 29)) (4.9.1)\n","Collecting tzdata\n","  Downloading tzdata-2022.5-py2.py3-none-any.whl (336 kB)\n","\u001b[K     |████████████████████████████████| 336 kB 58.0 MB/s \n","\u001b[?25hCollecting backports.zoneinfo\n","  Downloading backports.zoneinfo-0.2.1-cp37-cp37m-manylinux1_x86_64.whl (70 kB)\n","\u001b[K     |████████████████████████████████| 70 kB 8.9 MB/s \n","\u001b[?25hRequirement already satisfied: matplotlib>=2.2 in /usr/local/lib/python3.7/dist-packages (from seaborn==0.11.1->-r ./TableTransformer2CSV/requirements.txt (line 36)) (3.2.2)\n","Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from sklearn==0.0->-r ./TableTransformer2CSV/requirements.txt (line 37)) (1.0.2)\n","Requirement already satisfied: astor in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (0.8.1)\n","Collecting watchdog\n","  Downloading watchdog-2.1.9-py3-none-manylinux2014_x86_64.whl (78 kB)\n","\u001b[K     |████████████████████████████████| 78 kB 7.2 MB/s \n","\u001b[?25hRequirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (2.8.2)\n","Requirement already satisfied: attrs in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (22.1.0)\n","Requirement already satisfied: tornado>=5.0 in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (5.1.1)\n","Collecting base58\n","  Downloading base58-2.1.1-py3-none-any.whl (5.6 kB)\n","Requirement already satisfied: cachetools>=4.0 in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (4.2.4)\n","Collecting validators\n","  Downloading validators-0.20.0.tar.gz (30 kB)\n","Requirement already satisfied: altair>=3.2.0 in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (4.2.0)\n","Collecting blinker\n","  Downloading blinker-1.5-py2.py3-none-any.whl (12 kB)\n","Requirement already satisfied: tzlocal in /usr/local/lib/python3.7/dist-packages (from streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (1.5.1)\n","Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.7/dist-packages (from altair>=3.2.0->streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (4.3.3)\n","Requirement already satisfied: entrypoints in /usr/local/lib/python3.7/dist-packages (from altair>=3.2.0->streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (0.4)\n","Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (4.0.2)\n","Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (6.0.2)\n","Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (1.8.1)\n","Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (1.2.0)\n","Requirement already satisfied: asynctest==0.13.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (0.13.0)\n","Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (1.3.1)\n","Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (2.1.1)\n","Collecting smmap<6,>=3.0.1\n","  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n","Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (3.9.0)\n","Requirement already satisfied: ipython>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (7.9.0)\n","Requirement already satisfied: jupyter-client in /usr/local/lib/python3.7/dist-packages (from ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (6.1.12)\n","Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.7.5)\n","Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (2.0.10)\n","Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (2.6.1)\n","Collecting jedi>=0.10\n","  Downloading jedi-0.18.1-py2.py3-none-any.whl (1.6 MB)\n","\u001b[K     |████████████████████████████████| 1.6 MB 49.9 MB/s \n","\u001b[?25hRequirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (4.8.0)\n","Requirement already satisfied: backcall in /usr/local/lib/python3.7/dist-packages (from ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.2.0)\n","Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (4.4.2)\n","Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (3.6.1)\n","Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.2.0)\n","Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (3.0.3)\n","Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.7/dist-packages (from jedi>=0.10->ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.8.3)\n","Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.7/dist-packages (from Jinja2>=3.0->Flask==2.0.1->-r ./TableTransformer2CSV/requirements.txt (line 4)) (2.0.1)\n","Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.0->altair>=3.2.0->streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (0.18.1)\n","Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.0->altair>=3.2.0->streamlit==1.5.1->-r ./TableTransformer2CSV/requirements.txt (line 38)) (5.10.0)\n","Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2->seaborn==0.11.1->-r ./TableTransformer2CSV/requirements.txt (line 36)) (0.11.0)\n","Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib>=2.2->seaborn==0.11.1->-r ./TableTransformer2CSV/requirements.txt (line 36)) (1.4.4)\n","Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets==1.18.3->-r ./TableTransformer2CSV/requirements.txt (line 3)) (2022.4)\n","Requirement already satisfied: locket in /usr/local/lib/python3.7/dist-packages (from partd>=0.3.10->dask==2021.3.1->-r ./TableTransformer2CSV/requirements.txt (line 2)) (1.0.0)\n","Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=5.0.0->ipykernel>=5.1.2->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.2.5)\n","Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (2022.9.24)\n","Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (3.0.4)\n","Collecting requests\n","  Downloading requests-2.28.1-py3-none-any.whl (62 kB)\n","\u001b[K     |████████████████████████████████| 62 kB 1.4 MB/s \n","\u001b[?25hRequirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers->-r ./TableTransformer2CSV/requirements.txt (line 45)) (2.10)\n","Collecting MarkupSafe>=2.0\n","  Downloading MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n","Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.7/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (5.5.0)\n","Requirement already satisfied: nbformat in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (5.7.0)\n","Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.13.3)\n","Requirement already satisfied: jupyter-core>=4.4.0 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (4.11.1)\n","Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (5.6.1)\n","Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (23.2.1)\n","Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (1.8.0)\n","Requirement already satisfied: ptyprocess in /usr/local/lib/python3.7/dist-packages (from terminado>=0.8.1->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.7.0)\n","Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from bs4->PyDictionary==2.0.1->-r ./TableTransformer2CSV/requirements.txt (line 22)) (4.6.3)\n","Collecting futures\n","  Downloading futures-3.0.5.tar.gz (25 kB)\n","\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/55/db/97c1ca37edab586a1ae03d6892b6633d8eaa23b23ac40c7e5bbc55423c78/futures-3.0.5.tar.gz#sha256=0542525145d5afc984c88f914a0c85c77527f65946617edb5274f72406f981df (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n","  Downloading futures-3.0.4.tar.gz (25 kB)\n","\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/8d/73/b5fff618482bc06c9711e7cdc0d5d7eb1904d35898f48f2d7f9696b08bef/futures-3.0.4.tar.gz#sha256=19485d83f7bd2151c0aeaf88fbba3ee50dadfb222ffc3b66a344ef4952b782a3 (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n","  Downloading futures-3.0.3.tar.gz (24 kB)\n","\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/4c/dc/f9473006d4c9c52d4a4e977173fbcbfb1a8ef3a57e32e885edf994fd4a45/futures-3.0.3.tar.gz#sha256=2fe2342bb4fe8b8e217f0d21b5921cbe5408bf966d9f92025e707e881b198bed (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n","  Downloading futures-3.0.2.tar.gz (24 kB)\n","\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/f8/e7/fc0fcbeb9193ba2d4de00b065e7fd5aecd0679e93ce95a07322b2b1434f4/futures-3.0.2.tar.gz#sha256=dc3fc91508e49e0fd2f8625f0132d16e49c80f882e7e1d565c56b0d5dfbae257 (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n","  Downloading futures-3.0.1.tar.gz (24 kB)\n","\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/b2/2c/6b6a57379e47031c6f52e625e0e2b8f6702a8d1f61b6e0daee391e82c187/futures-3.0.1.tar.gz#sha256=f78f2ef458639d72a625cf9c7643cf5442bb222ac11c12bcc445c6ad1cd862e2 (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n","  Downloading futures-3.0.0.tar.gz (24 kB)\n","\u001b[33mWARNING: Discarding https://files.pythonhosted.org/packages/ea/c9/35287369718fc05059e7a9d0d73c53745fe981010b4185b3858e7d46eff1/futures-3.0.0.tar.gz#sha256=d9cd7bb09aa01f0e4940af64c31fbd7045098b7b4354420d7838ea39e8b86ee3 (from https://pypi.org/simple/futures/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.\u001b[0m\n","  Downloading futures-2.2.0-py2.py3-none-any.whl (16 kB)\n","Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (1.5.0)\n","Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.7.1)\n","Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.6.0)\n","Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (5.0.1)\n","Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.8.4)\n","Collecting nbformat\n","  Downloading nbformat-5.6.1-py3-none-any.whl (77 kB)\n","\u001b[K     |████████████████████████████████| 77 kB 3.4 MB/s \n","\u001b[?25h  Downloading nbformat-5.6.0-py3-none-any.whl (77 kB)\n","\u001b[K     |████████████████████████████████| 77 kB 5.8 MB/s \n","\u001b[?25hRequirement already satisfied: fastjsonschema in /usr/local/lib/python3.7/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (2.16.2)\n","Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets>=7.0.0->pydeck==0.7.1->-r ./TableTransformer2CSV/requirements.txt (line 21)) (0.5.1)\n","Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn==0.0->-r ./TableTransformer2CSV/requirements.txt (line 37)) (1.2.0)\n","Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->sklearn==0.0->-r ./TableTransformer2CSV/requirements.txt (line 37)) (3.1.0)\n","Building wheels for collected packages: pyperclip, sklearn, retrying, goslate, validators\n","  Building wheel for pyperclip (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for pyperclip: filename=pyperclip-1.8.2-py3-none-any.whl size=11137 sha256=edc3b7d934aeb9a547301ac5c91093b1aaa826868f26e53219f3abfb189dceba\n","  Stored in directory: /root/.cache/pip/wheels/9f/18/84/8f69f8b08169c7bae2dde6bd7daf0c19fca8c8e500ee620a28\n","  Building wheel for sklearn (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for sklearn: filename=sklearn-0.0-py2.py3-none-any.whl size=1310 sha256=4a6caf005ba17fd0c7b2b6f453893bd79e9866046cb9ef2487bc550c17cef21b\n","  Stored in directory: /root/.cache/pip/wheels/46/ef/c3/157e41f5ee1372d1be90b09f74f82b10e391eaacca8f22d33e\n","  Building wheel for retrying (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for retrying: filename=retrying-1.3.3-py3-none-any.whl size=11447 sha256=94dd4ea918f956d77dde0dbb951ad97ab0bec12957abe88d2738cc9770aac85d\n","  Stored in directory: /root/.cache/pip/wheels/f9/8d/8d/f6af3f7f9eea3553bc2fe6d53e4b287dad18b06a861ac56ddf\n","  Building wheel for goslate (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for goslate: filename=goslate-1.5.4-py3-none-any.whl size=11597 sha256=5003f787f6ddded6758e3f0d381ccaf1f3a1a98cbdb7b741ce7c73c2a0471b77\n","  Stored in directory: /root/.cache/pip/wheels/7e/08/d5/88fc81ecccbff86db9b3dac7ca657c2cc64b5d10db421bd2a2\n","  Building wheel for validators (setup.py) ... \u001b[?25l\u001b[?25hdone\n","  Created wheel for validators: filename=validators-0.20.0-py3-none-any.whl size=19582 sha256=b51443110a435cbc613478932ad82a61f6a6038dc59924250b76287d25f3950c\n","  Stored in directory: /root/.cache/pip/wheels/5f/55/ab/36a76989f7f88d9ca7b1f68da6d94252bb6a8d6ad4f18e04e9\n","Successfully built pyperclip sklearn retrying goslate validators\n","Installing collected packages: importlib-metadata, MarkupSafe, jedi, nbformat, Jinja2, urllib3, smmap, numpy, torch, toolz, scipy, requests, PyYAML, Pillow, gitdb, futures, xxhash, Werkzeug, websockets, watchdog, validators, tzdata, torchvision, tokenizers, retrying, Pympler, pyee, pydeck, pyarrow, protobuf, multiprocess, llvmlite, itsdangerous, huggingface-hub, goslate, GitPython, cssselect, blinker, base58, backports.zoneinfo, transformers, timm, streamlit, sklearn, seaborn, PyWavelets, pytz-deprecation-shim, pytesseract, pyreadline3, pyquery, pyppeteer, pyperclip, PyMuPDF, pymongo, pydot, PyDictionary, pydantic, psutil, ply, plotly, openpyxl, oauthlib, numexpr, numba, Flask, datasets, dask, Cython\n","  Attempting uninstall: importlib-metadata\n","    Found existing installation: importlib-metadata 5.0.0\n","    Uninstalling importlib-metadata-5.0.0:\n","      Successfully uninstalled importlib-metadata-5.0.0\n","  Attempting uninstall: MarkupSafe\n","    Found existing installation: MarkupSafe 2.0.1\n","    Uninstalling MarkupSafe-2.0.1:\n","      Successfully uninstalled MarkupSafe-2.0.1\n","  Attempting uninstall: nbformat\n","    Found existing installation: nbformat 5.7.0\n","    Uninstalling nbformat-5.7.0:\n","      Successfully uninstalled nbformat-5.7.0\n","  Attempting uninstall: Jinja2\n","    Found existing installation: Jinja2 2.11.3\n","    Uninstalling Jinja2-2.11.3:\n","      Successfully uninstalled Jinja2-2.11.3\n","  Attempting uninstall: urllib3\n","    Found existing installation: urllib3 1.24.3\n","    Uninstalling urllib3-1.24.3:\n","      Successfully uninstalled urllib3-1.24.3\n","  Attempting uninstall: numpy\n","    Found existing installation: numpy 1.21.6\n","    Uninstalling numpy-1.21.6:\n","      Successfully uninstalled numpy-1.21.6\n","  Attempting uninstall: torch\n","    Found existing installation: torch 1.12.1+cu113\n","    Uninstalling torch-1.12.1+cu113:\n","      Successfully uninstalled torch-1.12.1+cu113\n","  Attempting uninstall: toolz\n","    Found existing installation: toolz 0.12.0\n","    Uninstalling toolz-0.12.0:\n","      Successfully uninstalled toolz-0.12.0\n","  Attempting uninstall: scipy\n","    Found existing installation: scipy 1.7.3\n","    Uninstalling scipy-1.7.3:\n","      Successfully uninstalled scipy-1.7.3\n","  Attempting uninstall: requests\n","    Found existing installation: requests 2.23.0\n","    Uninstalling requests-2.23.0:\n","      Successfully uninstalled requests-2.23.0\n","  Attempting uninstall: PyYAML\n","    Found existing installation: PyYAML 6.0\n","    Uninstalling PyYAML-6.0:\n","      Successfully uninstalled PyYAML-6.0\n","  Attempting uninstall: Pillow\n","    Found existing installation: Pillow 7.1.2\n","    Uninstalling Pillow-7.1.2:\n","      Successfully uninstalled Pillow-7.1.2\n","  Attempting uninstall: Werkzeug\n","    Found existing installation: Werkzeug 1.0.1\n","    Uninstalling Werkzeug-1.0.1:\n","      Successfully uninstalled Werkzeug-1.0.1\n","  Attempting uninstall: torchvision\n","    Found existing installation: torchvision 0.13.1+cu113\n","    Uninstalling torchvision-0.13.1+cu113:\n","      Successfully uninstalled torchvision-0.13.1+cu113\n","  Attempting uninstall: pyarrow\n","    Found existing installation: pyarrow 6.0.1\n","    Uninstalling pyarrow-6.0.1:\n","      Successfully uninstalled pyarrow-6.0.1\n","  Attempting uninstall: protobuf\n","    Found existing installation: protobuf 3.17.3\n","    Uninstalling protobuf-3.17.3:\n","      Successfully uninstalled protobuf-3.17.3\n","  Attempting uninstall: llvmlite\n","    Found existing installation: llvmlite 0.39.1\n","    Uninstalling llvmlite-0.39.1:\n","      Successfully uninstalled llvmlite-0.39.1\n","  Attempting uninstall: itsdangerous\n","    Found existing installation: itsdangerous 1.1.0\n","    Uninstalling itsdangerous-1.1.0:\n","      Successfully uninstalled itsdangerous-1.1.0\n","  Running setup.py develop for transformers\n","  Attempting uninstall: seaborn\n","    Found existing installation: seaborn 0.11.2\n","    Uninstalling seaborn-0.11.2:\n","      Successfully uninstalled seaborn-0.11.2\n","  Attempting uninstall: PyWavelets\n","    Found existing installation: PyWavelets 1.3.0\n","    Uninstalling PyWavelets-1.3.0:\n","      Successfully uninstalled PyWavelets-1.3.0\n","  Attempting uninstall: pymongo\n","    Found existing installation: pymongo 4.2.0\n","    Uninstalling pymongo-4.2.0:\n","      Successfully uninstalled pymongo-4.2.0\n","  Attempting uninstall: pydot\n","    Found existing installation: pydot 1.3.0\n","    Uninstalling pydot-1.3.0:\n","      Successfully uninstalled pydot-1.3.0\n","  Attempting uninstall: pydantic\n","    Found existing installation: pydantic 1.9.2\n","    Uninstalling pydantic-1.9.2:\n","      Successfully uninstalled pydantic-1.9.2\n","  Attempting uninstall: psutil\n","    Found existing installation: psutil 5.4.8\n","    Uninstalling psutil-5.4.8:\n","      Successfully uninstalled psutil-5.4.8\n","  Attempting uninstall: plotly\n","    Found existing installation: plotly 5.5.0\n","    Uninstalling plotly-5.5.0:\n","      Successfully uninstalled plotly-5.5.0\n","  Attempting uninstall: openpyxl\n","    Found existing installation: openpyxl 3.0.10\n","    Uninstalling openpyxl-3.0.10:\n","      Successfully uninstalled openpyxl-3.0.10\n","  Attempting uninstall: oauthlib\n","    Found existing installation: oauthlib 3.2.1\n","    Uninstalling oauthlib-3.2.1:\n","      Successfully uninstalled oauthlib-3.2.1\n","  Attempting uninstall: numexpr\n","    Found existing installation: numexpr 2.8.3\n","    Uninstalling numexpr-2.8.3:\n","      Successfully uninstalled numexpr-2.8.3\n","  Attempting uninstall: numba\n","    Found existing installation: numba 0.56.2\n","    Uninstalling numba-0.56.2:\n","      Successfully uninstalled numba-0.56.2\n","  Attempting uninstall: Flask\n","    Found existing installation: Flask 1.1.4\n","    Uninstalling Flask-1.1.4:\n","      Successfully uninstalled Flask-1.1.4\n","  Attempting uninstall: dask\n","    Found existing installation: dask 2022.2.0\n","    Uninstalling dask-2022.2.0:\n","      Successfully uninstalled dask-2022.2.0\n","  Attempting uninstall: Cython\n","    Found existing installation: Cython 0.29.32\n","    Uninstalling Cython-0.29.32:\n","      Successfully uninstalled Cython-0.29.32\n","\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n","xarray-einstats 0.2.2 requires numpy>=1.21, but you have numpy 1.20.3 which is incompatible.\n","torchtext 0.13.1 requires torch==1.12.1, but you have torch 1.10.0 which is incompatible.\n","torchaudio 0.12.1+cu113 requires torch==1.12.1, but you have torch 1.10.0 which is incompatible.\n","thinc 8.1.3 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 1.7.3 which is incompatible.\n","spacy 3.4.1 requires pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4, but you have pydantic 1.7.3 which is incompatible.\n","plotnine 0.8.0 requires scipy>=1.5.0, but you have scipy 1.4.1 which is incompatible.\n","markdown 3.4.1 requires importlib-metadata>=4.4; python_version < \"3.10\", but you have importlib-metadata 2.1.3 which is incompatible.\n","jaxlib 0.3.20+cuda11.cudnn805 requires scipy>=1.5, but you have scipy 1.4.1 which is incompatible.\n","jax 0.3.21 requires scipy>=1.5, but you have scipy 1.4.1 which is incompatible.\n","gym 0.25.2 requires importlib-metadata>=4.8.0; python_version < \"3.10\", but you have importlib-metadata 2.1.3 which is incompatible.\n","googleapis-common-protos 1.56.4 requires protobuf<5.0.0dev,>=3.15.0, but you have protobuf 3.14.0 which is incompatible.\n","distributed 2022.2.0 requires dask==2022.02.0, but you have dask 2021.3.1 which is incompatible.\n","confection 0.0.3 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 1.7.3 which is incompatible.\n","cmdstanpy 1.0.7 requires numpy>=1.21, but you have numpy 1.20.3 which is incompatible.\u001b[0m\n","Successfully installed Cython-0.29.14 Flask-2.0.1 GitPython-3.1.26 Jinja2-3.1.2 MarkupSafe-2.1.1 Pillow-9.0.1 PyDictionary-2.0.1 PyMuPDF-1.20.2 PyWavelets-1.1.1 PyYAML-5.4.1 Pympler-1.0.1 Werkzeug-2.2.2 backports.zoneinfo-0.2.1 base58-2.1.1 blinker-1.5 cssselect-1.1.0 dask-2021.3.1 datasets-1.18.3 futures-2.2.0 gitdb-4.0.9 goslate-1.5.4 huggingface-hub-0.10.1 importlib-metadata-2.1.3 itsdangerous-2.1.2 jedi-0.18.1 llvmlite-0.37.0 multiprocess-0.70.12.2 nbformat-5.6.0 numba-0.54.1 numexpr-2.7.3 numpy-1.20.3 oauthlib-3.1.0 openpyxl-3.0.7 plotly-4.14.3 ply-3.11 protobuf-3.14.0 psutil-5.8.0 pyarrow-7.0.0 pydantic-1.7.3 pydeck-0.7.1 pydot-1.4.2 pyee-8.2.2 pymongo-4.0.2 pyperclip-1.8.2 pyppeteer-0.2.5 pyquery-1.4.3 pyreadline3-3.3 pytesseract-0.3.10 pytz-deprecation-shim-0.1.0.post0 requests-2.28.1 retrying-1.3.3 scipy-1.4.1 seaborn-0.11.1 sklearn-0.0 smmap-5.0.0 streamlit-1.5.1 timm-0.6.7 tokenizers-0.12.1 toolz-0.11.1 torch-1.10.0 torchvision-0.11.1 transformers tzdata-2022.5 urllib3-1.26.7 validators-0.20.0 watchdog-2.1.9 websockets-8.1 xxhash-3.0.0\n"]},{"output_type":"display_data","data":{"application/vnd.colab-display-data+json":{"pip_warning":{"packages":["PIL","concurrent","google","numpy","psutil"]}}},"metadata":{}}]},{"cell_type":"markdown","source":["Restart the enviroment!!\n","\n","\n","Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference\n","\n"],"metadata":{"id":"CKl3rTyhAksQ"}},{"cell_type":"markdown","source":["## **2.- Download dataset**\n"],"metadata":{"id":"17l-y_d6atSr"}},{"cell_type":"code","source":["import os\n","from google.colab import drive\n","drive.mount('/content/drive/')\n","\n","!cp \"/content/drive/MyDrive/Colab Notebooks/oculist-sparkocr/dataset/oculist.zip\" \"/content/oculist.zip\"\n","!unzip \"/content/oculist.zip\"\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"QuiXg-rhPJ2O","executionInfo":{"status":"ok","timestamp":1665994714402,"user_tz":-120,"elapsed":25299,"user":{"displayName":"Emilio Gómez García","userId":"12452742652222446865"}},"outputId":"2ba41e56-c19f-49be-e9ba-14cb345f068a"},"execution_count":1,"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/drive/\n","Archive:  /content/oculist.zip\n","   creating: images/\n","  inflating: images/1.jpeg           \n","  inflating: images/10.jpeg          \n","  inflating: images/11-1.jpg         \n","  inflating: images/11-2.jpg         \n","  inflating: images/12.jpeg          \n","  inflating: images/13.jpg           \n","  inflating: images/14.jpg           \n","  inflating: images/15.jpg           \n","  inflating: images/16.jpeg          \n","  inflating: images/17.jpg           \n","  inflating: images/18.jpg           \n","  inflating: images/19.jpg           \n","  inflating: images/2.jpeg           \n","  inflating: images/20.jpg           \n","  inflating: images/21.jpg           \n","  inflating: images/22.jpg           \n","  inflating: images/23.jpg           \n","  inflating: images/24-1.jpg         \n","  inflating: images/24-2.jpg         \n","  inflating: images/24-3.jpg         \n","  inflating: images/24-4.jpg         \n","  inflating: images/24-5.jpg         \n","  inflating: images/24.jpeg          \n","  inflating: images/3.jpg            \n","  inflating: images/4.jpg            \n","  inflating: images/5.jpeg           \n","  inflating: images/6.jpg            \n","  inflating: images/7.jpg            \n","  inflating: images/8.jpeg           \n","  inflating: images/9.jpg            \n","   creating: img_clean/\n","  inflating: img_clean/1.jpg         \n","  inflating: img_clean/10.jpg        \n","  inflating: img_clean/11-1.jpg      \n","  inflating: img_clean/11-2.jpg      \n","  inflating: img_clean/12.jpg        \n","  inflating: img_clean/13.jpg        \n","  inflating: img_clean/14.jpg        \n","  inflating: img_clean/15.jpg        \n","  inflating: img_clean/16.jpg        \n","  inflating: img_clean/17.jpg        \n","  inflating: img_clean/18.jpg        \n","  inflating: img_clean/19.jpg        \n","  inflating: img_clean/2.jpg         \n","  inflating: img_clean/20.jpg        \n","  inflating: img_clean/21.jpg        \n","  inflating: img_clean/22.jpg        \n","  inflating: img_clean/23.jpg        \n","  inflating: img_clean/24-1.jpg      \n","  inflating: img_clean/24-2.jpg      \n","  inflating: img_clean/24-3.jpg      \n","  inflating: img_clean/24-4.jpg      \n","  inflating: img_clean/24-5.jpg      \n","  inflating: img_clean/24.jpg        \n","  inflating: img_clean/3.jpg         \n","  inflating: img_clean/4.jpg         \n","  inflating: img_clean/5.jpg         \n","  inflating: img_clean/6.jpg         \n","  inflating: img_clean/7.jpg         \n","  inflating: img_clean/8.jpg         \n","  inflating: img_clean/9.jpg         \n","   creating: img_tables/\n","  inflating: img_tables/1.jpg        \n","  inflating: img_tables/10.jpg       \n","  inflating: img_tables/11-1.jpg     \n","  inflating: img_tables/12.jpg       \n","  inflating: img_tables/13.jpg       \n","  inflating: img_tables/13_1.jpg     \n","  inflating: img_tables/14.jpg       \n","  inflating: img_tables/15.jpg       \n","  inflating: img_tables/15_1.jpg     \n","  inflating: img_tables/16.jpg       \n","  inflating: img_tables/16_1.jpg     \n","  inflating: img_tables/17.jpg       \n","  inflating: img_tables/18.jpg       \n","  inflating: img_tables/18_1.jpg     \n","  inflating: img_tables/19.jpg       \n","  inflating: img_tables/2.jpg        \n","  inflating: img_tables/20.jpg       \n","  inflating: img_tables/21.jpg       \n","  inflating: img_tables/22.jpg       \n","  inflating: img_tables/23.jpg       \n","  inflating: img_tables/24-1.jpg     \n","  inflating: img_tables/24-1_1.jpg   \n","  inflating: img_tables/24-2.jpg     \n","  inflating: img_tables/24-2_1.jpg   \n","  inflating: img_tables/24-3.jpg     \n","  inflating: img_tables/24-3_1.jpg   \n","  inflating: img_tables/24-4.jpg     \n","  inflating: img_tables/24-4_1.jpg   \n","  inflating: img_tables/24-5.jpg     \n","  inflating: img_tables/24-5_1.jpg   \n","  inflating: img_tables/24.jpg       \n","  inflating: img_tables/3.jpg        \n","  inflating: img_tables/3_1.jpg      \n","  inflating: img_tables/4.jpg        \n","  inflating: img_tables/5.jpg        \n","  inflating: img_tables/6.jpg        \n","  inflating: img_tables/7.jpg        \n","  inflating: img_tables/7_1.jpg      \n","  inflating: img_tables/8.jpg        \n","  inflating: img_tables/9.jpg        \n"]}]},{"cell_type":"markdown","source":["## **3.- Inference**"],"metadata":{"id":"QIWX_Ql_CjbQ"}},{"cell_type":"code","source":["#!python ./TableTransformer2CSV/app.py"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"drFyebqqCjCR","executionInfo":{"status":"ok","timestamp":1665994830526,"user_tz":-120,"elapsed":3284,"user":{"displayName":"Emilio Gómez García","userId":"12452742652222446865"}},"outputId":"c87927b0-22f0-406b-e9ae-70881efe081e"},"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["2022-10-17 08:20:26.275 INFO    numexpr.utils: NumExpr defaulting to 2 threads.\n","Traceback (most recent call last):\n","  File \"./TableTransformer2CSV/app.py\", line 18, in <module>\n","    from transformers import DetrFeatureExtractor\n","ModuleNotFoundError: No module named 'transformers'\n"]}]},{"cell_type":"code","source":["import streamlit as st\n","from TableTransformer2CSV import *\n","# pytesseract.pytesseract.tesseract_cmd = r'C:\\Program Files\\Tesseract-OCR\\tesseract.exe'\n","\n","\n","st.set_option('deprecation.showPyplotGlobalUse', False)\n","st.set_page_config(layout='wide')\n","st.title(\"Table Detection and Table Structure Recognition\")\n","st.write(\"Implemented by MSFT team: https://github.com/microsoft/table-transformer\")\n","\n","\n","img_name = st.file_uploader(\"Upload an image with table(s)\")\n","\n","padd_top = st.slider('Padding top', 0, 200, 20)\n","padd_left = st.slider('Padding left', 0, 200, 20)\n","padd_right = st.slider('Padding right', 0, 200, 20)\n","padd_bottom = st.slider('Padding bottom', 0, 200, 20)\n","\n","\n","te = TableExtractionPipeline()\n","# for img in image_list:\n","if img_name is not None:\n","    asyncio.run(te.start_process(img_name, TD_THRESHOLD=0.6, TSR_THRESHOLD=0.8, padd_top=padd_top, padd_left=padd_left, padd_bottom=padd_bottom, padd_right=padd_right, delta_xmin=0, delta_ymin=0, delta_xmax=0, delta_ymax=0, expand_rowcol_bbox_top=0, expand_rowcol_bbox_bottom=0))\n","\n","\n"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":463},"id":"72dREpRDCB8A","executionInfo":{"status":"error","timestamp":1665994976455,"user_tz":-120,"elapsed":1325,"user":{"displayName":"Emilio Gómez García","userId":"12452742652222446865"}},"outputId":"6258a211-ffbe-4d42-ea92-ffbac1bef28c"},"execution_count":4,"outputs":[{"output_type":"stream","name":"stderr","text":["INFO:numexpr.utils:NumExpr defaulting to 2 threads.\n","2022-10-17 08:22:53.359 INFO    numexpr.utils: NumExpr defaulting to 2 threads.\n","WARNING:root:\n","  \u001b[33m\u001b[1mWarning:\u001b[0m to view this Streamlit app on a browser, run it with the following\n","  command:\n","\n","    streamlit run /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py [ARGUMENTS]\n","2022-10-17 08:22:54.160 \n","  \u001b[33m\u001b[1mWarning:\u001b[0m to view this Streamlit app on a browser, run it with the following\n","  command:\n","\n","    streamlit run /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py [ARGUMENTS]\n"]},{"output_type":"error","ename":"NameError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-4-f43c2aa6ce7b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     18\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0mte\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mTableExtractionPipeline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     21\u001b[0m \u001b[0;31m# for img in image_list:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mimg_name\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mNameError\u001b[0m: name 'TableExtractionPipeline' is not defined"]}]},{"cell_type":"markdown","source":["# TSR Evaluation"],"metadata":{"id":"umnwLWanR4Dk"}},{"cell_type":"markdown","source":["There are two different evaluation of TSR, with the dataset TrackB1 or with the TrackB2, as explained in:  https://github.com/cndplab-founder/ctdar_measurement_tool "],"metadata":{"id":"h-aBi3Qi5OZu"}},{"cell_type":"markdown","source":["*   TRACK B1\n"],"metadata":{"id":"eRAhZeVFcVUa"}},{"cell_type":"code","source":["# Download dataset\n","!wget -O TrackB1_test_GT.zip https://zenodo.org/record/3239032/files/TrackB1_test_GT.zip?download=1\n","!unzip -d /content/ctdar_measurement_tool/annotations/ TrackB1_test_GT.zip"],"metadata":{"id":"w4REKnL45JQu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Remove images of dataset that are not in result, and remove result that there are not in dataset\n","processed = [get_fname(x) for x in glob.glob(\"/content/result/*\")]\n","len(processed)\n","for f in glob.glob(\"/content/ctdar_measurement_tool/annotations/trackB1/*\"):\n","    if get_fname(f) not in processed:\n","        os.remove(f)\n","dataset = [get_fname(x) for x in glob.glob(\"/content/ctdar_measurement_tool/annotations/trackB1/*\")]\n","len(dataset)\n","for f in glob.glob(\"/content/result/*\"):\n","    if get_fname(f) not in dataset:\n","        os.remove(f)\n","\n","!tar -cvzf reg.tar.gz result/\n","!tar -cvzf res_img.tar.gz result_img/"],"metadata":{"id":"xom9AeA4cuh_"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Run evaluation\n","%cd \"/content/ctdar_measurement_tool\"\n","!python evaluate.py -trackB1 \"/content/reg.tar.gz\"\n","%cd \"/content\""],"metadata":{"id":"o0i8Kka9R-Hw"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["*   TRACK B2"],"metadata":{"id":"rdc8XS3McbYf"}},{"cell_type":"code","source":["# Download dataset\n","!wget -O TrackB2_test_GT.zip https://zenodo.org/record/3239032/files/TrackB2_test_GT.zip?download=1\n","!unzip -d /content/ctdar_measurement_tool/annotations/ TrackB2_test_GT.zip"],"metadata":{"id":"6Yc2YjZacdFu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Remove images of dataset that are not in result, and remove result that there are not in dataset\n","processed = [get_fname(x) for x in glob.glob(\"/content/result/*\")]\n","len(processed)\n","for f in glob.glob(\"/content/ctdar_measurement_tool/annotations/trackB2/*\"):\n","    if get_fname(f) not in processed:\n","        os.remove(f)\n","dataset = [get_fname(x) for x in glob.glob(\"/content/ctdar_measurement_tool/annotations/trackB2/*\")]\n","len(dataset)\n","for f in glob.glob(\"/content/result/*\"):\n","    if get_fname(f) not in dataset:\n","        os.remove(f)\n","\n","!tar -cvzf reg.tar.gz result/\n","!tar -cvzf res_img.tar.gz result_img/"],"metadata":{"id":"r8TklorPc0iW"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["# Run evaluation\n","%cd \"/content/ctdar_measurement_tool\"\n","!python evaluate.py -trackB2 \"/content/reg.tar.gz\"\n","%cd \"/content\""],"metadata":{"id":"LV2SlxC8chD_"},"execution_count":null,"outputs":[]}]}