-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathmetropolis.html
390 lines (336 loc) · 30.4 KB
/
metropolis.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="author" content="Jonathan Landy" />
<meta property="og:type" content="article" />
<meta name="twitter:card" content="summary">
<meta name="keywords" content=", Methods, Theory, " />
<meta property="og:title" content="Bayesian Statistics: MCMC "/>
<meta property="og:url" content="./metropolis" />
<meta property="og:description" content="We review the Metropolis algorithm — a simple Markov Chain Monte Carlo (MCMC) sampling method — and its application to estimating posteriors in Bayesian statistics. A simple python example is provided. Introduction One of the central aims of statistics is to identify good methods for fitting models to data. One way to …" />
<meta property="og:site_name" content="EFAVDB" />
<meta property="og:article:author" content="Jonathan Landy" />
<meta property="og:article:published_time" content="2016-08-07T18:37:00-07:00" />
<meta name="twitter:title" content="Bayesian Statistics: MCMC ">
<meta name="twitter:description" content="We review the Metropolis algorithm — a simple Markov Chain Monte Carlo (MCMC) sampling method — and its application to estimating posteriors in Bayesian statistics. A simple python example is provided. Introduction One of the central aims of statistics is to identify good methods for fitting models to data. One way to …">
<title>Bayesian Statistics: MCMC · EFAVDB
</title>
<link href="//netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/css/bootstrap-combined.min.css" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=Fira+Sans:300,400,700" rel="stylesheet" type='text/css' />
<link href="https://fonts.googleapis.com/css?family=Cardo:400,700" rel="stylesheet" type='text/css' />
<link rel="stylesheet" type="text/css" href="./theme/css/elegant.prod.css" media="screen">
<link rel="stylesheet" type="text/css" href="./theme/css/custom.css" media="screen">
<link rel="apple-touch-icon" sizes="180x180" href="./images/apple-touch-icon.png">
<link rel="icon" type="image/png" sizes="32x32" href="./images/favicon-32x32.png">
<link rel="icon" type="image/png" sizes="16x16" href="./images/favicon-16x16.png">
<link rel="manifest" href="./images/site.webmanifest">
<link rel="mask-icon" href="./images/safari-pinned-tab.svg" color="#5bbad5">
<meta name="msapplication-TileColor" content="#da532c">
<meta name="theme-color" content="#ffffff">
</head>
<body>
<div id="content">
<div class="navbar navbar-static-top">
<div class="navbar-inner">
<div class="container-fluid">
<a class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</a>
<a class="brand" href="./"><span class=site-name>EFAVDB</span></a>
<div class="nav-collapse collapse">
<ul class="nav pull-right top-menu">
<li >
<a href=
.
>Home</a>
</li>
<li ><a href="./pages/authors.html">Authors</a></li>
<li ><a href="./categories.html">Categories</a></li>
<li ><a href="./tags.html">Tags</a></li>
<li ><a href="./archives.html">Archives</a></li>
<li><form class="navbar-search" action="./search.html" onsubmit="return validateForm(this.elements['q'].value);"> <input type="text" class="search-query" placeholder="Search" name="q" id="tipue_search_input"></form></li>
</ul>
</div>
</div>
</div>
</div>
<div class="container-fluid">
<div class="row-fluid">
<div class="span1"></div>
<div class="span10">
<article itemscope>
<div class="row-fluid">
<header class="page-header span8 offset2">
<h1>
<a href="./metropolis">
Bayesian Statistics: <span class="caps">MCMC</span>
</a>
</h1>
</header>
<div class="span2"></div>
</div>
<div class="row-fluid">
<div class="span8 offset2 article-content">
<p>We review the Metropolis algorithm — a simple Markov Chain Monte Carlo (<span class="caps">MCMC</span>) sampling method — and its application to estimating posteriors in Bayesian statistics. A simple python example is provided.</p>
<h2>Introduction</h2>
<p>One of the central aims of statistics is to identify good methods for fitting models to data. One way to do this is through the use of Bayes’ rule: If <span class="math">\(\textbf{x}\)</span> is a vector of <span class="math">\(k\)</span> samples from a distribution and <span class="math">\(\textbf{z}\)</span> is a vector of model parameters, Bayes’ rule gives
</p>
<div class="math">\begin{align} \tag{1} \label{Bayes}
p(\textbf{z} \vert \textbf{x}) = \frac{p(\textbf{x} \vert \textbf{z}) p(\textbf{z})}{p(\textbf{x})}.
\end{align}</div>
<p>
Here, the probability at left, <span class="math">\(p(\textbf{z} \vert \textbf{x})\)</span> — the “posterior” — is a function that tells us how likely it is that the underlying true parameter values are <span class="math">\(\textbf{z}\)</span>, given the information provided by our observations <span class="math">\(\textbf{x}\)</span>. Notice that if we could solve for this function, we would be able to identify which parameter values are most likely — those that are good candidates for a fit. We could also use the posterior’s variance to quantify how uncertain we are about the true, underlying parameter values.</p>
<p>Bayes’ rule gives us a method for evaluating the posterior — now our goal: We need only evaluate the right side of (\ref{Bayes}). The quantities shown there are</p>
<p><span class="math">\(p(\textbf{x} \vert \textbf{z})\)</span> — This is the probability of seeing <span class="math">\(\textbf{x}\)</span> at fixed parameter values <span class="math">\(\textbf{z}\)</span>. Note that if the model is specified, we can often immediately write this part down. For example, if we have a Normal distribution model, specifying <span class="math">\(\textbf{z}\)</span> means that we have specified the Normal’s mean and variance. Given these, we can say how likely it is to observe any <span class="math">\(\textbf{x}\)</span>.</p>
<p><span class="math">\(p(\textbf{z})\)</span> — the “prior”. This is something we insert by hand before taking any data. We choose its form so that it covers the values we expect are reasonable for the parameters in question.</p>
<p><span class="math">\(p(\textbf{x})\)</span> — the denominator. Notice that this doesn’t depend on <span class="math">\(\textbf{z}\)</span>, and so represents a normalization constant for the posterior.</p>
<p>It turns out that the last term above can sometimes be difficult to evaluate analytically, and so we must often resort to numerical methods for estimating the posterior. Monte Carlo sampling is one of the most common approaches taken for doing this. The idea behind Monte Carlo is to take many samples <span class="math">\(\{\textbf{z}_i\}\)</span> from the posterior (\ref{Bayes}). Once these are obtained, we can approximate population averages by averages over the samples. For example, the true posterior average <span class="math">\(\langle\textbf{z} \rangle \equiv \int \textbf{z} p(\textbf{z} \vert \textbf{x}) d \textbf{z}\)</span> can be approximated by <span class="math">\(\overline{\textbf{z}} \equiv \frac{1}{N}\sum_i \textbf{z}_i\)</span>, the sample average. By the law of large numbers, the sample averages are guaranteed to approach the distribution averages as <span class="math">\(N \to \infty\)</span>. This means that Monte Carlo can always be used to obtain very accurate parameter estimates, provided we take <span class="math">\(N\)</span> sufficiently large — and that we can find a convenient way to sample from the posterior. In this post, we review one simple variant of Monte Carlo that allows for posterior sampling: the Metropolis algorithm.</p>
<h2>Metropolis Algorithm</h2>
<h3>Iterative Procedure</h3>
<p>Metropolis is an iterative, try-accept algorithm. We initialize the algorithm by selecting a parameter vector <span class="math">\(\textbf{z}\)</span> at random. Following this, we repeatedly carry out the following two steps to obtain additional posterior samples:</p>
<ol>
<li>Identify a next candidate sample <span class="math">\(\textbf{z}_j\)</span> via some random process. This candidate selection step can be informed by the current sample’s position, <span class="math">\(\textbf{z}_i\)</span>. For example, one could require that the next candidate be selected from those parameter vectors a given step-size distance from the current sample, <span class="math">\(\textbf{z}_j \in \{\textbf{z}_k: \vert \textbf{z}_i - \textbf{z}_k \vert = \delta \}\)</span>. However, while the candidate selected can depend on the current sample, it must not depend on any prior history of the sampling process. Whatever the process chosen (there’s some flexibility here), we write <span class="math">\(t_{i,j}\)</span> for the rate of selecting <span class="math">\(\textbf{z}_j\)</span> as the next candidate given the current sample is <span class="math">\(\textbf{z}_i\)</span>.</li>
<li>Once a candidate is identified, we either accept or reject it via a second random process. If it is accepted, we mark it down as the next sample, then go back to step one, using the current sample to inform the next candidate selection. Otherwise, we mark the current sample down again, taking it as a repeat sample, and then use it to return to candidate search step, as above. Here, we write <span class="math">\(A_{i,j}\)</span> for the rate of accepting <span class="math">\(\textbf{z}_j\)</span>, given that it was selected as the next candidate, starting from <span class="math">\(\textbf{z}_i\)</span>.</li>
</ol>
<h3>Selecting the trial and acceptance rates</h3>
<p><a href="./wp-content/uploads/2016/08/Untitled-1.jpg"><img alt="Untitled-1" src="./wp-content/uploads/2016/08/Untitled-1.jpg"></a></p>
<p>In order to ensure that our above process selects samples according to the distribution (\ref{Bayes}), we need to appropriately set the <span class="math">\(\{t_{i,j}\}\)</span> and <span class="math">\(\{A_{i,j}\}\)</span> values. To do that, note that at equilibrium one must see the same number of hops from <span class="math">\(\textbf{z}_i\)</span> to <span class="math">\(\textbf{z}_j\)</span> as hops from <span class="math">\(\textbf{z}_j\)</span> from <span class="math">\(\textbf{z}_i\)</span> (if this did not hold, one would see a net shifting of weight from one to the other over time, contradicting the assumption of equilibrium). If <span class="math">\(\rho_i\)</span> is the fraction of samples the process takes from state <span class="math">\(i\)</span>, this condition can be written as
</p>
<div class="math">\begin{align} \label{inter}
\rho_i t_{i,j} A_{i,j} = \rho_j t_{j,i} A_{j,i} \tag{3}
\end{align}</div>
<p>
To select a process that returns the desired sampling weight, we solve for <span class="math">\(\rho_i\)</span> over <span class="math">\(\rho_j\)</span> in (\ref{inter}) and then equate this to the ratio required by (\ref{Bayes}). This gives
</p>
<div class="math">\begin{align} \tag{4} \label{cond}
\frac{\rho_i}{\rho_j} = \frac{t_{j,i} A_{j,i}}{t_{i,j} A_{i,j}}
\equiv \frac{p(\textbf{x} \vert \textbf{z}_i)p(\textbf{z}_i)}{p(\textbf{x} \vert \textbf{z}_j)p(\textbf{z}_j)}.
\end{align}</div>
<p>
Now, the single constraint above is not sufficient to pin down all of our degrees of freedom. In the Metropolis case, we choose the following working balance: The trial rates between states are set equal, <span class="math">\(t_{i,j} = t_{j,i}\)</span> (but remain unspecified — left to the discretion of the coder on a case-by-case basis), and we set
</p>
<div class="math">$$ \tag{5}
A_{i,j} = \begin{cases}
1, & \text{if } p(\textbf{z}_j \vert \textbf{x}) > p(\textbf{z}_i \vert \textbf{x}) \\
\frac{p(\textbf{x} \vert \textbf{z}_j)p(\textbf{z}_j)}{p(\textbf{x} \vert \textbf{z}_i)p(\textbf{z}_i)} \equiv \frac{p(\textbf{z}_j \vert \textbf{x})}{p(\textbf{z}_i \vert \textbf{x})}, & \text{else}.
\end{cases}
$$</div>
<p>
This last equation says that we choose to always accept a candidate sample if it is more likely than the current one. However, if the candidate is less likely, we only accept a fraction of the time — with rate equal to the relative probability ratio of the two states. For example, if the candidate is only <span class="math">\(80%\)</span> as likely as the current sample, we accept it <span class="math">\(80%\)</span> of the time. That’s it for Metropolis — a simple <span class="caps">MCMC</span> algorithm, guaranteed to satisfy (\ref{cond}), and to therefore equilibrate to (\ref{Bayes})! An example follows.</p>
<h3>Coding example</h3>
<p>The following python snippet illustrates the Metropolis algorithm in action. Here, we take 15 samples from a Normal distribution of variance one and true mean also equal to one. We pretend not to know the mean (but assume we do know the variance), assume a uniform prior for the mean, and then run the algorithm to obtain two hundred thousand samples from the mean’s posterior. <a href="./wp-content/uploads/2016/08/result-1.png"><img alt="result" src="./wp-content/uploads/2016/08/result-1.png"></a> The histogram at right summarizes the results, obtained by dropping the first 1% of the samples (to protect against bias towards the initialization value). Averaging over the samples returns a mean estimate of <span class="math">\(\mu \approx 1.4 \pm 0.5\)</span> (95% confidence interval), consistent with the true value of <span class="math">\(1\)</span>.</p>
<div class="highlight"><pre><span></span><code><span class="o">%</span><span class="n">matplotlib</span> <span class="n">inline</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="c1"># Take some samples</span>
<span class="n">true_mean</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="n">loc</span><span class="o">=</span><span class="n">true_mean</span><span class="p">,</span> <span class="kp">size</span><span class="o">=</span><span class="mi">15</span><span class="p">)</span>
<span class="n">total_samples</span> <span class="o">=</span> <span class="mi">200000</span>
<span class="c1"># Function used to decide move acceptance</span>
<span class="k">def</span> <span class="nf">posterior_numerator</span><span class="p">(</span><span class="n">mu</span><span class="p">):</span>
<span class="kp">prod</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">X</span><span class="p">:</span>
<span class="kp">prod</span> <span class="o">*=</span> <span class="n">np</span><span class="o">.</span><span class="kp">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span> <span class="o">-</span> <span class="n">mu</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span> <span class="o">/</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">return</span> <span class="kp">prod</span>
<span class="c1"># Initialize MCMC, then iterate</span>
<span class="n">z1</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">posterior_samples</span> <span class="o">=</span> <span class="p">[</span><span class="n">z1</span><span class="p">]</span>
<span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">posterior_samples</span><span class="p">)</span> <span class="o"><</span> <span class="n">total_samples</span><span class="p">:</span>
<span class="n">z_current</span> <span class="o">=</span> <span class="n">posterior_samples</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">z_candidate</span> <span class="o">=</span> <span class="n">z_current</span> <span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span> <span class="o">-</span> <span class="mf">0.5</span>
<span class="n">rel_prob</span> <span class="o">=</span> <span class="n">posterior_numerator</span><span class="p">(</span>
<span class="n">z_candidate</span><span class="p">)</span> <span class="o">/</span> <span class="n">posterior_numerator</span><span class="p">(</span><span class="n">z_current</span><span class="p">)</span>
<span class="k">if</span> <span class="n">rel_prob</span> <span class="o">></span> <span class="mi">1</span><span class="p">:</span>
<span class="n">posterior_samples</span><span class="o">.</span><span class="kp">append</span><span class="p">(</span><span class="n">z_candidate</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">trial_toss</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">()</span>
<span class="k">if</span> <span class="n">trial_toss</span> <span class="o"><</span> <span class="n">rel_prob</span><span class="p">:</span>
<span class="n">posterior_samples</span><span class="o">.</span><span class="kp">append</span><span class="p">(</span><span class="n">z_candidate</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">posterior_samples</span><span class="o">.</span><span class="kp">append</span><span class="p">(</span><span class="n">z_current</span><span class="p">)</span>
<span class="c1"># Drop some initial samples and thin</span>
<span class="n">thinned_samples</span> <span class="o">=</span> <span class="n">posterior_samples</span><span class="p">[</span><span class="mi">2000</span><span class="p">:]</span>
<span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">thinned_samples</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">"Histogram of MCMC samples"</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</code></pre></div>
<h3>Summary</h3>
<p>To summarize, we have reviewed the application of <span class="caps">MCMC</span> to Bayesian statistics. <span class="caps">MCMC</span> is a general tool for obtaining samples from a probability distribution. It can be applied whenever one can conveniently specify the relative probability of two states — and so is particularly apt for situations where only the normalization constant of a distribution is difficult to evaluate, precisely the problem with the posterior (\ref{Bayes}). The method entails carrying out an iterative try-accept algorithm, where the rates of trial and acceptance can be adjusted, but must be balanced so that the equilibrium distribution that results approaches the desired form. The key equation enabling us to strike this balance is (\ref{inter}) — the zero flux condition (aka the <em>detailed balance</em> condition to physicists) that holds between states at equilibrium.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>
<p id="post-share-links">
Like this post? Share on:
<a href="https://twitter.com/intent/tweet?text=Bayesian%20Statistics%3A%20MCMC&url=/metropolis" target="_blank" rel="nofollow noopener noreferrer" title="Share on Twitter">Twitter</a>
|
<a href="https://www.facebook.com/sharer/sharer.php?u=/metropolis" target="_blank" rel="nofollow noopener noreferrer" title="Share on Facebook">Facebook</a>
|
<a href="mailto:?subject=Bayesian%20Statistics%3A%20MCMC&body=/metropolis" target="_blank" rel="nofollow noopener noreferrer" title="Share via Email">Email</a>
</p>
<hr />
<div class="author_blurb">
<a href="" target="_blank" rel="nofollow noopener noreferrer">
<img src=/wp-content/uploads/2014/12/JonathanLinkedIn.jpg alt="Jonathan Landy Avatar" title="Jonathan Landy">
<span class="author_name">Jonathan Landy</span>
</a>
Jonathan grew up in the midwest and then went to school at Caltech and UCLA. Following this, he did two postdocs, one at UCSB and one at UC Berkeley. His academic research focused primarily on applications of statistical mechanics, but his professional passion has always been in the mastering, development, and practical application of slick math methods/tools. He currently works as a data-scientist at Stitch Fix.
</div>
<hr/>
<aside>
<nav>
<ul class="articles-timeline">
<li class="previous-article">« <a href="./interpret-linear-regression" title="Previous: Interpreting the results of linear regression">Interpreting the results of linear regression</a></li>
<li class="next-article"><a href="./model-selection" title="Next: Hyperparameter sample-size dependence">Hyperparameter sample-size dependence</a> »</li>
</ul>
</nav>
</aside>
</div>
<section id="article-sidebar" class="span2">
<h4>Published</h4>
<time itemprop="dateCreated" datetime="2016-08-07T18:37:00-07:00">Aug 7, 2016</time>
<h4>Category</h4>
<a class="category-link" href="./categories.html#methods-theory-ref">Methods, Theory</a>
<h4>Contact</h4>
<div id="sidebar-social-link">
<a href="https://twitter.com/efavdb" title="" target="_blank" rel="nofollow noopener noreferrer">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="Twitter" role="img" viewBox="0 0 512 512"><rect width="512" height="512" rx="15%" fill="#1da1f3"/><path fill="#fff" d="M437 152a72 72 0 0 1-40 12 72 72 0 0 0 32-40 72 72 0 0 1-45 17 72 72 0 0 0-122 65 200 200 0 0 1-145-74 72 72 0 0 0 22 94 72 72 0 0 1-32-7 72 72 0 0 0 56 69 72 72 0 0 1-32 1 72 72 0 0 0 67 50 200 200 0 0 1-105 29 200 200 0 0 0 309-179 200 200 0 0 0 35-37"/></svg>
</a>
<a href="https://github.com/efavdb" title="" target="_blank" rel="nofollow noopener noreferrer">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="GitHub" role="img" viewBox="0 0 512 512"><rect width="512" height="512" rx="15%" fill="#1B1817"/><path fill="#fff" d="M335 499c14 0 12 17 12 17H165s-2-17 12-17c13 0 16-6 16-12l-1-50c-71 16-86-28-86-28-12-30-28-37-28-37-24-16 1-16 1-16 26 2 40 26 40 26 22 39 59 28 74 22 2-17 9-28 16-35-57-6-116-28-116-126 0-28 10-51 26-69-3-6-11-32 3-67 0 0 21-7 70 26 42-12 86-12 128 0 49-33 70-26 70-26 14 35 6 61 3 67 16 18 26 41 26 69 0 98-60 120-117 126 10 8 18 24 18 48l-1 70c0 6 3 12 16 12z"/></svg>
</a>
<a href="https://www.youtube.com/channel/UClfvjoSiu0VvWOh5OpnuusA" title="" target="_blank" rel="nofollow noopener noreferrer">
<svg xmlns="http://www.w3.org/2000/svg" aria-label="YouTube" role="img" viewBox="0 0 512 512" fill="#ed1d24"><rect width="512" height="512" rx="15%"/><path d="m427 169c-4-15-17-27-32-31-34-9-239-10-278 0-15 4-28 16-32 31-9 38-10 135 0 174 4 15 17 27 32 31 36 10 241 10 278 0 15-4 28-16 32-31 9-36 9-137 0-174" fill="#fff"/><path d="m220 203v106l93-53"/></svg>
</a>
</div>
</section>
</div>
</article>
<button onclick="topFunction()" id="myBtn" title="Go to top">▲</button>
</div>
<div class="span1"></div>
</div>
</div>
</div>
<footer>
<div>
<span class="site-name"><a href= .>EFAVDB</span> - Everybody's Favorite Data Blog</a>
</div>
<!-- <div id="fpowered"> -->
<!-- Powered by: <a href="http://getpelican.com/" title="Pelican Home Page" target="_blank" rel="nofollow noopener noreferrer">Pelican</a> -->
<!-- Theme: <a href="https://elegant.oncrashreboot.com/" title="Theme Elegant Home Page" target="_blank" rel="nofollow noopener noreferrer">Elegant</a> -->
<!-- -->
<!-- </div> -->
</footer> <script src="//code.jquery.com/jquery.min.js"></script>
<script src="//netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/js/bootstrap.min.js"></script>
<script>
function validateForm(query)
{
return (query.length > 0);
}
</script>
<script>
//** Scroll to top button **
//Get the button:
mybutton = document.getElementById("myBtn");
// When the user scrolls down 30px from the top of the document, show the button
window.onscroll = function() {scrollFunction()};
function scrollFunction() {
if (document.body.scrollTop > 30 || document.documentElement.scrollTop > 30) {
mybutton.style.display = "block";
} else {
mybutton.style.display = "none";
}
}
// When the user clicks on the button, scroll to the top of the document
function topFunction() {
document.body.scrollTop = 0; // For Safari
document.documentElement.scrollTop = 0; // For Chrome, Firefox, IE and Opera
}
</script>
<script>
(function () {
if (window.location.hash.match(/^#comment-\d+$/)) {
$('#comment_thread').collapse('show');
}
})();
window.onhashchange=function(){
if (window.location.hash.match(/^#comment-\d+$/))
window.location.reload(true);
}
$('#comment_thread').on('shown', function () {
var link = document.getElementById('comment-accordion-toggle');
var old_innerHTML = link.innerHTML;
$(link).fadeOut(200, function() {
$(this).text('Click here to hide comments').fadeIn(200);
});
$('#comment_thread').on('hidden', function () {
$(link).fadeOut(200, function() {
$(this).text(old_innerHTML).fadeIn(200);
});
})
})
</script>
</body>
<!-- Theme: Elegant built for Pelican
License : MIT -->
</html>