Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added support for ClickHouse #377

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add support for ActiveRecord driver for ClickHouse
foxy-eyed committed Jan 20, 2022
commit 3910dee5f8e177491aabe2d1a914ba69186fa740
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -716,7 +716,7 @@ data_sources:

### ClickHouse

Add [ClickHouse Ruby driver](https://github.com/shlima/click_house) to your Gemfile and set:
Add [ClickHouse Ruby driver](https://github.com/shlima/click_house) OR [Clickhouse::Activerecord](https://github.com/PNixx/clickhouse-activerecord) to your Gemfile and set:
```yml
data_sources:
my_source:
82 changes: 64 additions & 18 deletions lib/blazer/adapters/clickhouse_adapter.rb
Original file line number Diff line number Diff line change
@@ -1,14 +1,55 @@
module Blazer
module Adapters
class ClickhouseAdapter < BaseAdapter
SUPPORTED_DRIVERS_MAPPING = {
"click_house" => ClickHouseDriver,
"clickhouse-activerecord" => ClickhouseActiverecordDriver
}.freeze

delegate :tables, to: :driver

# Wrapper for ClickHouse Ruby driver (https://github.com/shlima/click_house)
class ClickHouseDriver
delegate :tables, to: :connection

def initialize(config)
@config = ClickHouse::Config.new(**config)
end

def connection
@connection ||= ClickHouse::Connection.new(@config)
end

def execute(statement, format)
connection.post(query: { query: statement, default_format: format }).body
end
end

# Wrapper for Clickhouse::Activerecord driver (https://github.com/PNixx/clickhouse-activerecord)
class ClickhouseActiverecordDriver
delegate :tables, to: :connection

def initialize(config)
@config = config
end

def connection
@connection ||= ActiveRecord::Base.clickhouse_connection(@config)
end

def execute(statement, format)
body = connection.do_execute(statement, format: format)
format.in?(%w[CSV CSVWithNames]) ? CSV.parse(body) : body
end
end

def run_statement(statement, _comment)
columns = []
rows = []
error = nil

begin
response = connection.post(query: { query: statement, default_format: "CSVWithNames" })
rows = response.body
rows = driver.execute(statement, "CSVWithNames")
columns = rows.shift
rescue => e
error = e.message
@@ -17,54 +58,59 @@ def run_statement(statement, _comment)
[columns, rows, error]
end

def tables
connection.tables
end

def schema
statement = <<-SQL
SELECT table, name, type
FROM system.columns
WHERE database = '#{connection.config.database}'
WHERE database = '#{config[:database]}'
ORDER BY table, position
SQL

response = connection.post(query: { query: statement, default_format: "CSV" })
response.body
.group_by { |row| row[0] }
.transform_values { |columns| columns.map { |c| { name: c[1], data_type: c[2] } } }
.map { |table, columns| { schema: "public", table: table, columns: columns } }
rows = driver.execute(statement, "CSV")
rows.group_by { |row| row[0] }
.transform_values { |columns| columns.map { |c| { name: c[1], data_type: c[2] } } }
.map { |table, columns| { schema: "public", table: table, columns: columns } }
end

def preview_statement
"SELECT * FROM {table} LIMIT 10"
end

def explain(statement)
connection.explain(statement)
driver.execute("EXPLAIN #{statement.gsub(/\A(\s*EXPLAIN)/io, '')}", "TSV")
end

protected

def connection
@connection ||= ClickHouse::Connection.new(config)
def driver
@driver ||= begin
driver = SUPPORTED_DRIVERS_MAPPING.keys.find { |driver| installed?(driver) }
raise Blazer::Error, "ClickHouse driver not installed!" unless driver

SUPPORTED_DRIVERS_MAPPING[driver].new(config)
end
end

def config
@config ||= begin
uri = URI.parse(settings["url"])
options = {
{
scheme: uri.scheme,
host: uri.host,
port: uri.port,
username: uri.user,
password: uri.password,
database: uri.path.split("/").last
}.compact

ClickHouse::Config.new(**options)
end
end

def installed?(driver_name)
Gem::Specification.find_by_name(driver_name)
true
rescue Gem::LoadError
false
end
end
end
end