-
Notifications
You must be signed in to change notification settings - Fork 0
/
faster-rails-3-deployments-to-aws-elastic-beanstalk.html
316 lines (251 loc) · 17 KB
/
faster-rails-3-deployments-to-aws-elastic-beanstalk.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Faster Rails 3 deployments to AWS Elastic Beanstalk ← Vinicius Horewicz</title>
<meta name="viewport" content="width=device-width">
<meta name="author" content="Vinicius Horewicz (wicz)">
<meta name="description" content="wicz' personal notes">
<link rel="alternate" type="application/atom+xml" href="/atom.xml" title="RSS feed" />
<link rel="stylesheet" href="/assets/stylesheets/normalize.css">
<link rel="stylesheet" href="/assets/stylesheets/monokai.css">
<link rel="stylesheet" href="/assets/stylesheets/main.css">
<link rel="stylesheet" href="/assets/stylesheets/screen.css">
<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.0/css/font-awesome.css" rel="stylesheet">
<link href="//fonts.googleapis.com/css?family=Roboto:400,300,300italic|Roboto+Slab:300|Ubuntu+Mono" rel="stylesheet" type="text/css">
<script src="/assets/javascripts/modernizr-2.6.2.min.js"></script>
</head>
<body>
<div class="container">
<p><a href="/">← index</a></p>
<section class="post-content">
<header>
<h1>Faster Rails 3 deployments to AWS Elastic Beanstalk</h1>
<time datetime="2013-04-08T00:00:00+02:00" pubdate>08 April 2013</time>
</header>
<article>
<p>Web applications deployments cannot be slow. The definition of <em>slow</em>,
of course, is relative. I like to think <em>slow</em> is anything that breaks
the flow, that takes long enough to make me lose focus of what I am
doing.</p>
<p>Recently I moved a Rails 3 app from Heroku to Amazon Elastic Beanstalk
(EB). Deploying to Heroku was not showing the best performance results,
but to EB was taking enough time to make me lose my patience.</p>
<p>After understanding how EB works, identifying bottlenecks,
and caching things; I was able to reduce the time to deploy to near one
minute.</p>
<h3 id="eb-behind-the-scenes">EB: Behind the scenes</h3>
<p>EB allows deployments in a Heroku-esque way simply executing <code>git
aws.push</code>. This command is actually an alias to a ruby script located
at <code>.git/AWSDevTools/aws.elasticbeanstalk.push</code>. It just pushes yours
latest commit to the EB servers.</p>
<p>When EB receives the latest version of your app, it packs the source
files in a zip archive, uploads to a S3 bucket under your account, and
deploys to the EB environment.</p>
<h3 id="inspecting-the-ec2-plumbing">Inspecting the EC2 plumbing</h3>
<p>Analyzing the file <code>/var/log/eb-tools.log</code>, we can see the deployment
has hooks and scripts associated with it. The program at
<code>/usr/bin/directoryHooksExecutor.py</code> executes those scripts
sequentially in alphabetical order by name.</p>
<div class="highlight"><pre><code>2013-04-07 14:03:59,414 [INFO] (3056 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing directory: /opt/elasticbeanstalk/hooks/appdeploy/enact/
2013-04-07 14:03:59,511 [INFO] (3056 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/enact/01_flip.sh
2013-04-07 14:04:00,420 [INFO] (3056 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/enact/09clean.sh
2013-04-07 14:04:00,600 [INFO] (3056 MainThread) [directoryHooksExecutor.py-29] [root directoryHooksExecutor info] Executing script: /opt/elasticbeanstalk/hooks/appdeploy/enact/99_reload_app_server.sh
</code></pre></div>
<p>Given EC2 (non-EBS) does not persist data, and EB auto-scale our EC2 instances,
we cannot change these deploys scripts. However, we can <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html">customize the
EC2</a>
with <code>commands</code> and <code>container_commands</code>.</p>
<h3 id="finding-the-bottlenecks">Finding the bottlenecks</h3>
<p>To customize the EC2 correctly, I first had to understand how exactly
the commands specified in my <code>.ebextensions/*.config</code> files act in the
whole deployment.</p>
<p>I created the two commands below, and added to the top of each
<code>/opt/elasticbeanstalk/hooks/appdeploy/**/*.sh</code> script an instruction to
create a file with the timestamp of its execution.</p>
<div class="highlight"><pre><code class="language-yaml"><span class="c1"># .ebextensions/app.config</span>
<span class="l-Scalar-Plain">commands</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">01_first_command</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">command</span><span class="p-Indicator">:</span> <span class="s">"touch</span><span class="nv"> </span><span class="s">/tmp/$(date</span><span class="nv"> </span><span class="s">+'%T.%N').command"</span>
<span class="l-Scalar-Plain">container_commands</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">01_first_container_command</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">command</span><span class="p-Indicator">:</span> <span class="s">"touch</span><span class="nv"> </span><span class="s">/tmp/$(date</span><span class="nv"> </span><span class="s">+'%T.%N').container_command"</span></code></pre></div>
<div class="highlight"><pre><code class="language-sh"><span class="c"># /opt/elasticbeanstalk/hooks/appdeploy/pre/01_unzip.sh</span>
<span class="c">#!/usr/bin/env bash</span>
touch /tmp/<span class="k">$(</span>date +<span class="s2">"%T.%N"</span><span class="k">)</span>.<span class="k">$(</span>basename <span class="nv">$0</span><span class="k">)</span>
. /opt/elasticbeanstalk/support/envvars
mkdir -p <span class="nv">$EB_CONFIG_APP_BASE</span> <span class="o">&&</span> chown <span class="nv">$EB_CONFIG_APP_USER</span>:<span class="nv">$EB_CONFIG_APP_USER</span> <span class="nv">$EB_CONFIG_APP_BASE</span>
<span class="o">[</span> -d <span class="nv">$EB_CONFIG_APP_ONDECK</span> <span class="o">]</span> <span class="o">&&</span> rm -rf <span class="nv">$EB_CONFIG_APP_ONDECK</span>
su -c <span class="s2">"/usr/bin/unzip -d $EB_CONFIG_APP_ONDECK $EB_CONFIG_SOURCE_BUNDLE"</span> <span class="nv">$EB_CONFIG_APP_USER</span>
chmod <span class="m">775</span> <span class="nv">$EB_CONFIG_APP_ONDECK</span></code></pre></div>
<p>After a deploy, I was able to see the exact sequence of the
commands executed, and how long each one was taking. The whole process
was taking about unconceivable <strong>8 minutes</strong> to complete. The culprits
for such slowness were, as suspected, gems installation and assets
compilation.</p>
<div class="highlight"><pre><code>$ ls -1 /tmp
19:56:22.524828173.command
19:56:23.188093045.01_unzip.sh
19:56:23.524226323.02_setup_envvars.sh
19:56:23.575868688.10_bundle_install.sh
19:59:32.932137842.11_asset_compilation.sh
20:04:29.588351276.12_db_migration.sh
20:04:30.618231911.container_command
20:04:38.656611666.01_flip.sh
20:04:39.683074030.09clean.sh
20:04:39.706735596.99_reload_app_server.sh
</code></pre></div>
<h3 id="caching-gems-and-assets">Caching gems and assets</h3>
<p>We need to cache the gems to optimize <code>bundle install</code>, and to boost
the asset compilation we install <a href="https://github.com/ndbroadbent/turbo-sprockets-rails3">Turbo
Sprockets</a> and
cache the compiled assets as well.</p>
<p>Unfortunately, we cannot use <code>container_commands</code>
because we need to set up our cached files between <code>01_unzip.sh</code> and
<code>10_bundle_install.sh</code>. Nor can we use <code>commands</code>, because
<code>01_unzip.sh</code> cleans up the target directory before unpacking your
source files. We can use <code>files</code> option to inject a script to set up
the cache.</p>
<p>Since the hook scripts are executed in alphabetical order, we need to
name the injected script correctly to be executed just after
<code>01_unzip.sh</code>. We will name it <code>01a_bootstrap.sh</code>.</p>
<div class="highlight"><pre><code class="language-yaml"><span class="l-Scalar-Plain">files</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">/opt/elasticbeanstalk/hooks/appdeploy/pre/01a_bootstrap.sh</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">mode</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">00755</span>
<span class="l-Scalar-Plain">owner</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">root</span>
<span class="l-Scalar-Plain">group</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">root</span>
<span class="l-Scalar-Plain">source</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">http://s3.amazonaws.com/mybucket/bootstrap.sh</span></code></pre></div>
<p>The cached files were previously moved to <code>/var/app/support</code>. The
bootstrap script will create symbolic links to the directory being
deployed.</p>
<div class="highlight"><pre><code class="language-bash"><span class="c"># /opt/elasticbeanstalk/hooks/appdeploy/pre/01a_bootstrap.sh</span>
<span class="c">#!/usr/bin/env bash</span>
mkdir /var/app/ondeck/vendor /var/app/ondeck/public /var/app/support/bundle /var/app/support/assets
ln -s /var/app/support/bundle /var/app/ondeck/vendor
ln -s /var/app/support/assets /var/app/ondeck/public</code></pre></div>
<p>Ultimately, we could drop the deployment time near to <strong>1 minute and 4
seconds</strong>—the time to download the packages is not counted here, but
since it is in S3 in the same region, it takes no longer than 3 seconds
to get 30MB. This is the kind of slowness I can withstand.</p>
<div class="highlight"><pre><code>$ ls -1 /tmp
01:49:05.696544004.01_unzip.sh
01:49:06.029004848.01a_bootstrap.sh
01:49:06.055676774.02_setup_envvars.sh
01:49:06.120822495.10_bundle_install.sh
01:49:07.156151174.11_asset_compilation.sh
01:50:07.243165374.12_db_migration.sh
01:50:08.454668833.01_flip.sh
01:50:09.648283462.09clean.sh
01:50:09.889387126.99_reload_app_server.sh
</code></pre></div>
<h3 id="cleaning-up-and-updating-the-cache">Cleaning up and updating the cache</h3>
<p>Given the deployment is no more an issue, we need to make sure our
cached files correspond to the latest version of our application. We
can use a <code>post</code> hook to execute another script to update the cache.</p>
<div class="highlight"><pre><code class="language-yaml"><span class="l-Scalar-Plain">files</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">/opt/elasticbeanstalk/hooks/appdeploy/post/01_update_cache.sh</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">mode</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">00755</span>
<span class="l-Scalar-Plain">owner</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">root</span>
<span class="l-Scalar-Plain">group</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">root</span>
<span class="l-Scalar-Plain">source</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">http://s3.amazonaws.com/mybucket/update_cache.sh</span></code></pre></div>
<div class="highlight"><pre><code class="language-sh"><span class="c"># /opt/elasticbeanstalk/hooks/appdeploy/post/01_update_cache.sh</span>
<span class="c">#!/usr/bin/env bash</span>
. /opt/elasticbeanstalk/support/envvars
<span class="k">if</span> <span class="o">[</span> <span class="s2">"$RAILS_ENV"</span> !<span class="o">=</span> <span class="s2">"staging"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">exit</span>
<span class="k">fi</span>
<span class="nb">cd</span> /var/app/support
tar zcf bundle.tar.gz bundle
s3put -b mybucket -p /var/app/support -g public-read bundle.tar.gz
tar zcf assets.tar.gz assets
s3put -b mybucket -p /var/app/support -g public-read assets.tar.gz</code></pre></div>
<p>Thanks to staging-production parity, we can safely use our
staging server to keep the cache updated, and leave the production
server exclusively to our application.</p>
<p>We send the cache packages to our S3 bucket, so it will be available to
any new EC2 instance EB starts. The only thing left is to configure
the EC2 to download and unpack those packages. We can easily accomplish
this using the <code>sources</code> key.</p>
<div class="highlight"><pre><code class="language-yaml"><span class="l-Scalar-Plain">sources</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">/var/app/support</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">http://s3.amazonaws.com/mybucket/bundle.tar.gz</span>
<span class="l-Scalar-Plain">/var/app/support</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">http://s3.amazonaws.com/mybucket/assets.tar.gz</span></code></pre></div>
<p>We have made great changes to our deploy time. Nevertheless, there is
room for improvements. For example, the <code>update_cache.sh</code> script could
check for changes in the cached packages and upload a new version only
when necessary.</p>
<h3 id="update-apr-09">Update Apr, 09</h3>
<p>I had some issues with the asset compilation. The problem is that the
script
<code>/opt/elasticbeanstalk/hooks/appdeploy/pre/11_asset_compilation.sh</code>
executes <code>rake</code> directly, therefore it was not using the bundled gems.
I set my bootstrap script to change it to <code>bundle exec rake</code>.</p>
<div class="highlight"><pre><code class="language-console"><span class="gp">$</span> sed -i <span class="s1">'s/"rake/"bundle exec rake/'</span> /opt/elasticbeanstalk/hooks/appdeploy/pre/11_asset_compilation.sh</code></pre></div>
<p>Another issue happened with passenger and git backed libraries.
Thanks to <a href="http://stackoverflow.com/a/13657473">these</a>
<a href="http://stackoverflow.com/a/13656775">answers</a>, I was able to fix the
problem injecting another script right after <code>10_bundle_install.sh</code> to
pack all the gems.</p>
<div class="highlight"><pre><code class="language-yaml"><span class="l-Scalar-Plain">files</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">/opt/elasticbeanstalk/hooks/appdeploy/pre/10a_bundle_pack.sh</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">mode</span><span class="p-Indicator">:</span> <span class="s">"00755"</span>
<span class="l-Scalar-Plain">owner</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">root</span>
<span class="l-Scalar-Plain">group</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">root</span>
<span class="l-Scalar-Plain">source</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">https://s3.amazonaws.com/mybucket/bundle_pack.sh</span></code></pre></div>
<div class="highlight"><pre><code class="language-sh"><span class="c"># /opt/elasticbeanstalk/hooks/appdeploy/pre/10a_bundle_pack.sh</span>
<span class="c">#!/usr/bin/env bash</span>
. /opt/elasticbeanstalk/support/envvars
<span class="nb">cd</span> /var/app/ondeck
bundle pack --all</code></pre></div>
<p>I also added the <code>vendor/cache</code> directory to the <code>bundle.tar.gz</code> to
avoid any delays in the deployment.</p>
<p>I have put all my configuration files and scripts in this <a href="https://gist.github.com/wicz/5345688">gist</a>.</p>
</article>
</section>
<!-- Enable Disqus comments -->
<div id="disqus_thread"></div>
<script type="text/javascript">
var disqus_shortname = 'horewicz';
var disqus_identifier = '/faster-rails-3-deployments-to-aws-elastic-beanstalk';
var disqus_url = 'http://horewi.cz/faster-rails-3-deployments-to-aws-elastic-beanstalk.html';
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<footer>
<a href="about.html">
<i class="fa fa-info-circle fa-fw fa-lg"></i>
</a>
<a href="https://www.linkedin.com/in/viniciushorewicz">
<i class="fa fa-l1nkedin fa-fw fa-lg"></i>
</a>
<a href="https://github.com/wicz">
<i class="fa fa-github fa-fw fa-lg"></i>
</a>
<a href="https://twitter.com/wicz">
<i class="fa fa-tw1tter fa-fw fa-lg"></i>
</a>
<a href="https://stackoverflow.com/users/3243455/wicz">
<i class="fa fa-stack-overflow fa-fw fa-lg"></i>
</a>
<a href="/assets/wicz.pub.asc">
<i class="fa fa-lock fa-fw fa-lg"></i>
</a>
<a href="/atom.xml">
<i class="fa fa-rss fa-fw fa-lg"></i>
</a>
</footer>
</div>
<script type="text/javascript">
var _gaq=[['_setAccount','UA-8480243-3'],['_trackPageview']];
(function(d,t){var g=d.createElement(t),s=d.getElementsByTagName(t)[0];
g.src=('https:'==location.protocol?'//ssl':'//www')+'.google-analytics.com/ga.js';
s.parentNode.insertBefore(g,s)}(document,'script'));
</script>
</body>
</html>