-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathdetaching-resources-detached-search-service.html
218 lines (176 loc) · 10.4 KB
/
detaching-resources-detached-search-service.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
<title>Detaching Resources: Detached Search Service ← Vinicius Horewicz</title>
<meta name="viewport" content="width=device-width">
<meta name="author" content="Vinicius Horewicz (wicz)">
<meta name="description" content="wicz' personal notes">
<link rel="alternate" type="application/atom+xml" href="/atom.xml" title="RSS feed" />
<link rel="stylesheet" href="/assets/stylesheets/normalize.css">
<link rel="stylesheet" href="/assets/stylesheets/monokai.css">
<link rel="stylesheet" href="/assets/stylesheets/main.css">
<link rel="stylesheet" href="/assets/stylesheets/screen.css">
<link href="//netdna.bootstrapcdn.com/font-awesome/4.0.0/css/font-awesome.css" rel="stylesheet">
<link href="//fonts.googleapis.com/css?family=Roboto:400,300,300italic|Roboto+Slab:300|Ubuntu+Mono" rel="stylesheet" type="text/css">
<script src="/assets/javascripts/modernizr-2.6.2.min.js"></script>
</head>
<body>
<div class="container">
<p><a href="/">← index</a></p>
<section class="post-content">
<header>
<h1>Detaching Resources: Detached Search Service</h1>
<time datetime="2013-11-01T00:00:00+01:00" pubdate>01 November 2013</time>
</header>
<article>
<p>In the <a href="detaching-resources-architecture-foundation.html">previous
post</a>, we have
reviewed the basics about virtualization, learned about Docker and set
up a container running RabbitMQ. In this post, we will create a web app
with a search service. The interesting part is that the former will be
completely decoupled from the latter.</p>
<p>Regardless of being very contrived, the following example gives us a
pretty good bird’s-eye view and the basics for building web apps towards
a service-oriented architecture.</p>
<h2 id="a-bit-more-docker">A bit more Docker</h2>
<p>First we will need to build a container able to run our ruby services.
Thanks to docker’s abilities of reusing images, we can create a new
image to suit our needs pretty easily by inheriting from another image.</p>
<p>I used <a href="https://index.docker.io/u/binaryphile/ruby/">this image</a> as the
base for my experiment, and added on top of it essential packages like
<code>curl</code>, <code>monit</code> and <code>openssh-server</code>. You can my new image
<a href="https://github.com/wicz/detaching-resources/tree/master/v1/docker-ruby">here</a>.</p>
<p>Given our goal is to create a fully distributed system, we will start
three docker containers, each for the following services: the message
<em>queue</em>, the <em>web</em> app and the <em>search</em> engine.</p>
<p>Every time you start a container, docker assigns it a new IP address.
This means we have to get the IP address—via <code>docker inspect</code>—of the
<em>queue</em> container, to set it on the other containers.</p>
<p>To avoid this intense manual labor, we can use
<a href="https://github.com/jpetazzo/pipework/">pipework</a> to set up a private
network. We will define static IP addresses for the containers and share
them using environment variables.</p>
<div class="highlight"><pre><code class="language-console"><span class="gp">#</span> spin up containers
<span class="gp">$</span> <span class="nv">QUEUE</span><span class="o">=</span><span class="k">$(</span>docker run -d wicz/rabbitmq<span class="k">)</span>
<span class="gp">$</span> <span class="nv">WEB</span><span class="o">=</span><span class="k">$(</span>docker run -e <span class="nv">QUEUE_HOST</span><span class="o">=</span>10.11.12.1 -d wicz/ruby<span class="k">)</span>
<span class="gp">$</span> <span class="nv">SEARCH</span><span class="o">=</span><span class="k">$(</span>docker run -e <span class="nv">QUEUE_HOST</span><span class="o">=</span>10.11.12.1 -d wicz/ruby<span class="k">)</span>
<span class="gp">#</span> <span class="nb">set </span>IP addresses <span class="k">for</span> containers
<span class="gp">$</span> pipework br1 <span class="nv">$QUEUE</span> 10.11.12.1/24
<span class="gp">$</span> pipework br1 <span class="nv">$WEB</span> 10.11.12.2/24
<span class="gp">$</span> pipework br1 <span class="nv">$SEARCH</span> 10.11.12.3/24
<span class="gp">#</span> <span class="nb">enable </span>host to access the private network
<span class="gp">$</span> ifconfig br1 10.11.12.254/24</code></pre></div>
<p>Docker 0.7 has a new feature to
<a href="http://docs.docker.io/en/latest/examples/linking_into_redis/">“link”</a>
containers together. With this, we can skip the pipework set up, because
docker will populate the environments variables with the IP addresses it
automatically assigns and with the exposed ports.</p>
<h2 id="testing-the-services">Testing the services</h2>
<p>With the ruby containers and the message queue running, we can set up
the web and search services. We will clone the repository in both
containers and start the respective services.</p>
<p>First we need to clone the repository in both <em>web</em> and <em>search</em>
containers:</p>
<div class="highlight"><pre><code class="language-console"><span class="gp">$</span> ssh [email protected].<span class="o">{</span>2,3<span class="o">}</span> git clone https://github.com/wicz/detaching-resources.git /opt/dr</code></pre></div>
<p>For the <em>search</em> container we need to start the consumer daemon:</p>
<div class="highlight"><pre><code class="language-console"><span class="go">search$ cd /opt/dr/v2/search</span>
<span class="go">search$ bundle exec ruby bin/search_consumer.rb</span></code></pre></div>
<p>And for the <em>web</em> container we need to start the consumer and the web
server:</p>
<div class="highlight"><pre><code class="language-console"><span class="go">web$ cd /opt/dr/v2/web</span>
<span class="go">web$ bundle exec ruby bin/web_consumer.rb</span>
<span class="go">web$ bundle exec ruby web.rb -o 0.0.0.0 &</span></code></pre></div>
<p>Now point your browser to the web server and you should see a simple
page with a search form. Whatever search you do will redirect to another
page with “Searching…” written on it. Wait a few seconds, refresh this
page and you should see “It works!”.</p>
<p>Since I am running the docker container inside a Vagrant VM, I
forwarded a local port to the web server in the <em>web</em> container to be
able to use my browser in the OS X.</p>
<div class="highlight"><pre><code class="language-console"><span class="gp">$</span> vagrant ssh -- -L 8001:10.11.12.2:4567
<span class="gp">#</span> open http://localhost:8001</code></pre></div>
<h2 id="behind-the-scenes">Behind the scenes</h2>
<p>The example is naive, not to mention it always returns the same result
for every search. But the significance in it, is to note <strong>how</strong> the
results are generated and the services exchange messages, and not
<strong>what</strong> they are.</p>
<p>When you hit the “Submit” button, the web app creates a new <code>Search</code>
object and invokes the <code>SearchService</code> to send it to a queue named
<code>dr.searches</code>. Then the client is redirect to a page with the search
results. Since it has not been processed yet, it shows the
“Searching…” placeholder.</p>
<p>On the other container, the search daemon consumes messages from
<code>dr.searches</code> queue. It unserializes the message, changes its content
and send it back to another queue named <code>dr.searches.results</code>.</p>
<p>The <code>SearchConsumer</code> knows nothing about the web app, the database
<code>SearchRepository</code> or even the <code>Search</code> object. All it knows is how to
handle JSON data. Its perfect because we have created a completely
decoupled solution. As long as we do not change the JSON specification,
we can change whatever we want in the web app and the search engine will
continue working. We can also create other apps that use the same search
engine.</p>
<p>Back in the <em>web</em> container, the <code>SearchResultsConsumer</code> receives the
search results from the <code>dr.searches.results</code> queue. It unserializes the
JSON and updates the results in the database.</p>
<h2 id="conclusion">Conclusion</h2>
<p>We have successfully set up our detached services solution. With a few
amendments in the business logic, I think it could be used in production
environments. If you need a similar solution and do not have the time or
energy to improve this one, I suggest giving
<a href="https://github.com/gocardless/hutch">Hutch</a> from GoCardless a try.</p>
<p>Given the message queue is fully working, we could easily set up another
services like long running jobs, e.g. image manipulation or reports
generation, by just creating new consumers.</p>
<p>Try this examples yourself. I would love to hear your thoughts on the
subject. Comments are open!</p>
</article>
</section>
<!-- Enable Disqus comments -->
<div id="disqus_thread"></div>
<script type="text/javascript">
var disqus_shortname = 'horewicz';
var disqus_identifier = '/detaching-resources-detached-search-service';
var disqus_url = 'http://horewi.cz/detaching-resources-detached-search-service.html';
(function() {
var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
})();
</script>
<footer>
<a href="about.html">
<i class="fa fa-info-circle fa-fw fa-lg"></i>
</a>
<a href="https://www.linkedin.com/in/viniciushorewicz">
<i class="fa fa-l1nkedin fa-fw fa-lg"></i>
</a>
<a href="https://github.com/wicz">
<i class="fa fa-github fa-fw fa-lg"></i>
</a>
<a href="https://twitter.com/wicz">
<i class="fa fa-tw1tter fa-fw fa-lg"></i>
</a>
<a href="https://stackoverflow.com/users/3243455/wicz">
<i class="fa fa-stack-overflow fa-fw fa-lg"></i>
</a>
<a href="/assets/wicz.pub.asc">
<i class="fa fa-lock fa-fw fa-lg"></i>
</a>
<a href="/atom.xml">
<i class="fa fa-rss fa-fw fa-lg"></i>
</a>
</footer>
</div>
<script type="text/javascript">
var _gaq=[['_setAccount','UA-8480243-3'],['_trackPageview']];
(function(d,t){var g=d.createElement(t),s=d.getElementsByTagName(t)[0];
g.src=('https:'==location.protocol?'//ssl':'//www')+'.google-analytics.com/ga.js';
s.parentNode.insertBefore(g,s)}(document,'script'));
</script>
</body>
</html>