Home

Articles to understand theory better:

• How Does A Computer Calculate Eigenvalues?
• A new trick for calculating Jacobian vector products

Other articles not discussed in mainstream ML:

• Bayesian Committee Machine
• Predictive State Representations: A New Theory for Modeling Dynamical Systems
• Learning by transduction

Misc

• Mixtape: Breaking the Softmax Bottleneck Efficiently
• Understanding the Mixture of Softmaxes (MoS)

---

Reference only

Fourier Transforms in Neural Networks

• NeRF
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains Some background work:
  • Neural Tangent Kernels Blog Post
  • Neural Tangent Kernels main paper

Transformers

• Links to papers and code on various transformers by HuggingFace

Probabilistic Programming Languages

• An introduction to probabilistic programming
• (a more approachable resource) Automating Inference, Learning, and Designusing Probabilistic Programming

Geometric deep learning

• Good list of resources
• Deep Convolutional Networks on Graph-Structured Data

Applied Stochastic Differential Equations

• PDF file

Bayesian Models of Cognition

• A rational analysis of rule-based concept learning
• Learning overhypotheses with hierarchical Bayesian models

Important tricks used in recent papers

• Noise contrastive estimation
• Notes on noise contrastive estimation and negative sampling
• Density ratio trick
• Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation

---

NIPS 2019 papers

• Putting an end to end-to-end: Gradient-Isolated Learning of Representations
• Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs
• Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks
• Probabilistic Matrix Factorization for Automated Machine Learning.pdf
• Estimators for Multivariate Information Measures in General Probability Spaces
• Generalized Sliced Wasserstein Distances
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Learning Conditional Deformable Templates with Convolutional Networks
• SGD on Neural Networks Learns Functions of Increasing Complexity
• Full-Gradient Representation for Neural Network Visualization
• Understanding and Improving Layer Normalization
• Neural Similarity Learning
• A Primal-Dual Constraint for Deep Learning with Constraints
• Integer Discrete Flows and Lossless Compression

---

Other papers worth reading

• Bayesian Deep Learning and a Probabilistic Perspective of Generalization