Skip to content
This repository has been archived by the owner on Nov 30, 2021. It is now read-only.

Commit

Permalink
Ability to use http proxy with StreamClient
Browse files Browse the repository at this point in the history
  • Loading branch information
corinnekunze committed Sep 17, 2018
1 parent 5c538b5 commit 8b93592
Show file tree
Hide file tree
Showing 3 changed files with 34 additions and 13 deletions.
12 changes: 8 additions & 4 deletions lib/looker-sdk/client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -346,16 +346,20 @@ def build_response(connection, request)
http_request.body = request.body

connect_opts = {
:use_ssl => !!connection.ssl,
:verify_mode => (connection.ssl.verify rescue true) ?
use_ssl: !!connection.ssl,
verify_mode: (connection.ssl.verify rescue true) ?
OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE,
}

# TODO: figure out how/if to support proxies
# TODO: figure out how to test this comprehensively

progress = nil
Net::HTTP.start(uri.host, uri.port, connect_opts) do |http|
http_klass = if connection.proxy.present?
Net::HTTP::Proxy(connection.proxy.uri.host, connection.proxy.uri.port)
else
Net::HTTP
end
http_klass.start(uri.host, uri.port, connect_opts) do |http|
http.open_timeout = connection.options.open_timeout rescue 30
http.read_timeout = connection.options.timeout rescue 60

Expand Down
26 changes: 17 additions & 9 deletions streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,23 @@

#### Beta Feature - Experimental!

This SDK makes it easy to fetch a response from a Looker API and hydrate it into a Ruby object.This convenience is great for working with configuration and administrative data. However, when the response is gigabytes of row data, pulling it all into memory doesn't work so well - you can't begin processing the data until after it has all downloaded, for example, and chewing up tons of memory will put a serious strain on the entire system - even crash it.
This SDK makes it easy to fetch a response from a Looker API and hydrate it into a Ruby object.This convenience is great for working with configuration and administrative data. However, when the response is gigabytes of row data, pulling it all into memory doesn't work so well - you can't begin processing the data until after it has all downloaded, for example, and chewing up tons of memory will put a serious strain on the entire system - even crash it.

One solution to all this is to use streaming downloads to process the data in chunks as it is downloaded. Streaming requires a little more code to set up but the benefits can be significant.
One solution to all this is to use streaming downloads to process the data in chunks as it is downloaded. Streaming requires a little more code to set up but the benefits can be significant.

To use streaming downloads with the Looker SDK, simply add a block to an SDK call. The block will be called with chunks of data as they are downloaded instead of the method returning a complete result.

For example:

```ruby
```ruby
def run_look_to_file(look_id, filename, format, opts = {})
File.open(filename, 'w') do |file|
sdk.run_look(look_id, format, opts) do |data, progress|
file.write(data)
puts "Wrote #{data.length} bytes of #{progress.length} total"
end
end
end
end
end
end

run_look_to_file(38, 'out.csv', 'csv', limit: 10000)
```
Expand All @@ -40,7 +40,7 @@ You can also abort a streaming download by calling `progress.stop` within the bl
else
process_data(data)
end
end
end

```

Expand All @@ -53,7 +53,15 @@ These caveats can be mitigated by knowing the structure of the data being stream

#### A Tale of Two Stacks

The Looker Ruby SDK is built on top of Sawyer, and Sawyer sits on top of Faraday. Faraday does not support HTTP Streaming, so to do our streaming stuff we have to bypass Faraday and talk directly to the Net:HTTP stack. Our streaming implementation gathers connection settings from the Faraday stack so there's no additional config required for http streaming.
The Looker Ruby SDK is built on top of Sawyer, and Sawyer sits on top of Faraday. Faraday does not support HTTP Streaming, so to do our streaming stuff we have to bypass Faraday and talk directly to the Net:HTTP stack. Our streaming implementation gathers connection settings from the Faraday stack so there's no additional config required for http streaming.

Streaming downloads have not been tested with proxy connections. We assume attempting to stream across an http proxy will not work.
#### Proxy Connections
In order to stream with a proxy connection, you can define your proxy url in your client configs like so:
```
LookerSDK::Client.new(
client_id: ENV['CLIENT_ID'],
client_secret: ENV['CLIENT_SECRET'],
proxy: ENV['HTTP_PROXY']
)
```

9 changes: 9 additions & 0 deletions test/looker/test_client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,15 @@
client.client_secret.wont_be_nil
end
end

it "sets proxy with module methods" do
client = LookerSDK::Client.new
test_proxy = "http://my-test-proxy.com:232"
client.configure do |config|
config.proxy = test_proxy
end
client.instance_variable_get(:'@proxy').must_equal test_proxy
end
end

describe "config tests" do
Expand Down

0 comments on commit 8b93592

Please sign in to comment.