toolbox

Curated libraries for a faster workflow

Phase: Data

Annotation

Image: makesense.ai
Text: doccano, prodigy

Dataset

Text: nlp-datasets, curse-words, badwords, LDNOOBW, english-words (A text file containing over 466k English words), 10K most common words
Image: 1 million fake faces
Dataset search engine: datasetlist, UCI Machine Learning Datasets

Fetch data

Audio: pydub
Video: pytube (download youTube vidoes)
Image: py-image-dataset-generator (auto fetch images from web for certain search)
News: news-please
PDF: camelot, tabula-py
Remote file: smart_open
Crawling: pyppeteer (chrome automation), MechanicalSoup, libextract
Google sheets: gspread
Google drive: gdown
Python API for datasets: pydataset
Google maps location data: geo-heatmap

Data Augmentation

Text: nlpaug
Image: imgaug, albumentations, augmentor

Phase: Exploration

Data Preparation

Missing values: missingno
Split images into train/validation/test: split-folders
Class Imbalance: imblearn

Experiment in notebooks

View Jupyter notebooks through CLI: nbdime
Parametrize notebooks: papermill
Access notebooks programatically: nbformat
Convert notebooks to other formats: nbconvert
Extra utilities not present in frameworks: mlxtend
Maps in notebooks: ipyleaflet

Phase: Feature Engineering

Generate Features

Automatic feature engineering: featuretools, autopandas
Custom distance metric learning: metric-learn
List of holidays: python-holidays

Phase: Modeling

Model Selection

Bruteforce through all scikit-learn model and parameters: auto-sklearn, tpot
Autogenerate ML code: automl-gs, mindsdb
Pretrained models: modeldepot, pytorch-hub, papers-with-code
Find SOTA models: sotawhat

Framework extensions

Pytorch: Keras like summary for pytorch, skorch (wrap pytorch in scikit-learn compatible API)
Einstein notation: einops
Scikit-learn: scikit-lego

Algorithms

Gradient Boosting: catboost
Hidden Markov Models: hmmlearn
Genetic Programming: gplearn
Active Learning: modal
Rule based classifier: sklearn-expertsys

NLP

Preprocessing: textacy
Text Extraction from Image, Audio, PDF: textract
Text generation: gp2client, textgenrnn, gpt-2-simple
Text summarization: textrank, pytldr
Spelling correction: JamSpell, pyhunspell, pyspellchecker, cython_hunspell, hunspell-dictionaries, autocorrect (can add more languages)
Keyword extraction: rake, pke
Multiply Choice Question Answering: mcQA
Sequence to sequence models: headliner
Transfer learning: finetune
Translation: googletrans
Embeddings: pymagnitude (manage vector embeddings easily), chakin (download pre-trained word vectors), sentence-transformers, InferSent, bert-as-service, sent2vec
Multilingual support: polyglot, inltk (indic languages), indic_nlp
NLU: snips-nlu
Semantic parsing: quepy
Inflections: inflect
Contractions: pycontractions
Coreference Resolution: neuralcoref
Readability: homer
Language Detection: language-check
Topic Modeling: guidedlda, enstop
Clustering: spherecluster (kmeans with cosine distance), kneed (automatically find number of clusters from elbow curve), kmodes
Metrics: seqeval (NER, POS tagging)
String match: jellyfish (perform string and phonetic comparison),flashtext (superfast extract and replace keywords), pythonverbalexpressions: (verbally describe regex), commonregex (readymade regex for email/phone etc)
Sentiment: vaderSentiment (rule based)
Text distances: textdistance, editdistance
PID removal: scrubadub
Profanity detection: profanity-check
wordclouds: stylecloud

Speech Recognition

Library: speech_recognition

RecSys

Factorization machines (FM), and field-aware factorization machines (FFM): xlearn
Scikit-learn like API: surprise

Computer Vision

Image processing: scikit-image, imutils
Segmentation Models in Keras: segmentation_models
Face recognition: face_recognition, face-alignment (find facial landmarks)
Face swapping: faceit
Video summarization: videodigest
Semantic search over videos: scoper
OCR: keras-ocr, pytesseract

Timeseries

Predict Time Series: prophet
Scikit-learn like API: sktime

Phase: Monitoring

Monitor training process

Learning curve: lrcurve (plot realtime learning curve in Keras), livelossplot
Notifications: knockknock (get notified by slack/email), jupyter-notify (notify when task is completed in jupyter)

Phase: Optimization

Hyperparameter Optimization

Keras: keras-tuner
Scikit-learn: sklearn-deap (evolutionary algorithm for hyperparameter search)
General: hyperopt

Interpretability

Visualize keras models: keras-vis
Interpret models: eli5, lime, shap, alibi, tf-explain, treeinterpreter
Interpret BERT: exbert
Interpret word2vec: word2viz

Visualization

Draw CNN figures: nn-svg
Visualization for scikit-learn: yellowbrick, scikit-plot
XKCD like charts: chart.xkcd
Convert matplotlib charts to D3 charts: mpld3
Generate graphs using markdown: mermaid
Visualize topics models: pyldavis
High dimensional visualization: umap

Phase: Production

Scalability

Parallelize .apply in Pandas: pandarallel, swifter

Bechmarking

Profile pytorch layers: torchprof

API

Read config files: config, python-decouple
Data Validation: schema, jsonschema, cerebrus, pydantic, marshmallow
Enable CORS in Flask: flask-cors
Cache results of functions: cachetools, cachew (cache to local sqlite)
Authentication: pyjwt (JWT)
Task Queue: rq

Serialization

Transpiling: sklearn-porter (transpile sklearn model to C, Java, JavaScript and others), m2cgen

Dashboards

Generate frontend with python: streamlit

Adversarial testing

Generate images to fool model: foolbox
Generate phrases to fool NLP models: triggers

Python libraries

Datetime compatible API for Bikram Sambat: nepali-date
bloom filter: python-bloomfilter
Run python libraries in sandbox: pipx
Pretty print tables in CLI: tabulate
Leaflet maps from python: folium
Debugging: PySnooper
Pickling extended: cloudpickle