Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Third homework #5

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
- `localhost:3000` - main
- `localhost:3000/oauth/applications` - oauth app managment
- `localhost:8000' - task tracker
- `localhost:8080` - accounting

## How to start oAuth with kafka broker

Expand Down Expand Up @@ -133,4 +134,16 @@ docker-compose up
- c: Task management: create account
- Task.Created:
- pr: Task management,
- c: Balance create task with costs
- c: Balance create task with costs

## Эволюция схемы данных

1. Делаем новую схему для нашего события и добавляем его в библиотеку
2. Обновить библиотеку на консьюмере, подтвердить, что появилась новая версия нашего события, после чего написать новый консьюмер, который будет работать с событиями новой версии
3. Добавляем в сервис инвентаризации продьюсер для новой версии события и начинаем продьюсить событие
4. После проверки, что всё работает в продакшене, мы можем удалить старые продьюсер (в первую очередь) и консьюмер.

## Стратегия обработки ошибок

1. Делаем отдельный топик со всеми сообщениями связанные с таск апплай и резолв, которые мы не смогли обработать, и отправляем невалидное событие туда.
2. Просматриваем вручную такие сообщения, исправляем и возвращаем в изначальный топик.
106 changes: 106 additions & 0 deletions schema_registory/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Event schema registry

This repository is an example of how to make event schema registry for JSON schema events using only github. The general idea - how to share schemas across different services plus how to validate data for specific events.

## Setup
### Ruby
Add this line into your Gemfile:

```
gem "schema_registry", git: "https://github.com/davydovanton/event_schema_registry.git"
```

## How to add a new event schema

For example, you want to create `billing.refund` event. For make it happen you need:

1. Create a new file `domain/event_name/version.json` in `schemas/` folder. For `billing.refund` it will be `schemas/billing/refund/1.json` (because all new events should be first version;
2. Create a new json schema file like this:

```
{
"$schema": "http://json-schema.org/draft-04/schema#",

"title": "Billing.Refund.v1",
"description": "json schema for billing refund event (version 1)",

"definitions": {
"event_data": {
"type": "object",
"properties": {
// event specific information here
},
"required": [
]
}
},

"type": "object",

"properties": {
"event_id": { "type": "string" },
"event_version": { "enum": [1] },
"event_name": { "type": "string" },
"event_time": { "type": "string" },
"producer": { "type": "string" },

"data": { "$ref": "#/definitions/event_data" }
},

"required": [
"event_id",
"event_version",
"event_name",
"event_time",
"producer",
"data"
]
}
```

## How to validate an event data by specific schema

### Ruby

For validating event data you need to use `SchemaRegistry#validate_event` method with following options:

* `data` - event data
* `name` - name of event which you will use for getting schema
* `version` - version of event data schema (default `1`)

Example:

```ruby
message = {
# ...
}

# will try to search `schemas/Billing/CompliteCycle/1.json` file
result = SchemaRegistry.validate_event(data, 'Billing.CompliteCycle', version: 1)
# will try to search `schemas/billing/complite_cycle/1.json` file
result = SchemaRegistry.validate_event(data, 'billing.complite_cycle', version: 1)

# After you can work with result object
result.success?
result.failure?
result.failure
```

## How to use this library with producer
### Option one: with event object
```ruby
result = SchemaRegistry.validate_event(event, 'billing.refund', version: 1)

if result.success?
kafka.produce('topic', event.to_json)
end
```

### Option two: with pure hash
```ruby
result = SchemaRegistry.validate_event(event, 'billing.refund', version: 1)

if result.success?
kafka.produce('topic', event.to_json)
end
```
5 changes: 5 additions & 0 deletions schema_registory/ruby/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# frozen_string_literal: true

source 'https://rubygems.org'

gemspec
47 changes: 47 additions & 0 deletions schema_registory/ruby/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
PATH
remote: .
specs:
schema_registry (1)
json-schema

GEM
remote: https://rubygems.org/
specs:
addressable (2.7.0)
public_suffix (>= 2.0.2, < 5.0)
diff-lcs (1.3)
highline (2.0.3)
json-schema (2.8.1)
addressable (>= 2.4)
options (2.3.2)
progress_bar (1.3.1)
highline (>= 1.6, < 3)
options (~> 2.3.0)
public_suffix (4.0.5)
rake (13.0.1)
rspec (3.9.0)
rspec-core (~> 3.9.0)
rspec-expectations (~> 3.9.0)
rspec-mocks (~> 3.9.0)
rspec-core (3.9.0)
rspec-support (~> 3.9.0)
rspec-expectations (3.9.0)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.9.0)
rspec-mocks (3.9.0)
diff-lcs (>= 1.2.0, < 2.0)
rspec-support (~> 3.9.0)
rspec-support (3.9.0)

PLATFORMS
ruby

DEPENDENCIES
bundler
progress_bar
rake
rspec
schema_registry!

BUNDLED WITH
2.1.4
24 changes: 24 additions & 0 deletions schema_registory/ruby/schema_registry.gemspec
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# frozen_string_literal: true

Gem::Specification.new do |spec|
spec.name = 'schema_registry'
spec.version = 1
spec.authors = ['Anton Davydov']
spec.email = ['[email protected]']
spec.summary = 'Event schema registry example'
spec.homepage = 'https://github.com/davydovanton/event_schema_registry'
spec.license = 'MIT'

spec.files = Dir['../schemas/**/*', 'lib/**/*']
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
spec.require_paths = ['../schemas', 'lib/']

spec.required_ruby_version = '>= 2.4.0'

spec.add_dependency 'json-schema'

spec.add_development_dependency 'bundler'
spec.add_development_dependency 'progress_bar'
spec.add_development_dependency 'rake'
spec.add_development_dependency 'rspec'
end
49 changes: 49 additions & 0 deletions schema_registory/ruby/spec/schema_registry/validator_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# frozen_string_literal: true

require 'schema_registry'

RSpec.describe SchemaRegistry::Validator do
describe '#validate' do
subject { validator.validate(data, event_name, version: version) }

let(:validator) { described_class.new(loader: loader) }
let(:loader) { SchemaRegistry::Loader.new(schemas_root_path: schemas_root_path) }
let(:schemas_root_path) { File.expand_path('../support/schemas', __dir__) }

let(:event_name) { 'domain.event' }
let(:category) { 'general' }
let(:version) { 1 }

context 'when event schema is valid' do
context 'when type schema is valid' do
let(:data) do
{
event_id: SecureRandom.uuid,
event_version: 1,
event_name: 'Domain.Event',
event_time: Time.now.to_s,
producer: 'rspec',
data_version: 1,
data: {
order_id: 1,
account_uuid: SecureRandom.uuid,
},
}
end

it { expect(subject).to be_success }
end

context 'when event schema is invalid' do
let(:data) { { data: { name: nil } } }

it { expect(subject).to be_failure }

it 'returns errors' do
expect(subject.failure.count).to eq(7)
expect(subject.failure.last).to include("The property '#/' did not contain a required property of 'producer' in schema")
end
end
end
end
end
28 changes: 28 additions & 0 deletions schema_registory/ruby/spec/schema_registry_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# frozen_string_literal: true

require 'schema_registry'
require 'securerandom'

RSpec.describe SchemaRegistry do
describe '#validate_event' do
subject { described_class.validate_event(data, 'billing.refund', version: 1) }

let(:data) do
{
event_id: SecureRandom.uuid,
event_version: 1,
event_name: 'Domain.Event',
event_time: Time.now.to_s,
producer: 'rspec',
data_version: 1,
data: {
order_id: 1,
account_uuid: SecureRandom.uuid,
timestamp: Time.now.to_s
},
}
end

it { is_expected.to eq(SchemaRegistry::Result.new([])) }
end
end
95 changes: 95 additions & 0 deletions schema_registory/ruby/spec/spec_helper.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# frozen_string_literal: true

require 'schema_registry'

# This file was generated by the `rspec --init` command. Conventionally, all
# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
# The generated `.rspec` file contains `--require spec_helper` which will cause
# this file to always be loaded, without a need to explicitly require it in any
# files.
#
# Given that it is always loaded, you are encouraged to keep this file as
# light-weight as possible. Requiring heavyweight dependencies from this file
# will add to the boot time of your test suite on EVERY test run, even for an
# individual file that may not need all of that loaded. Instead, consider making
# a separate helper file that requires the additional dependencies and performs
# the additional setup, and require it from the spec files that actually need
# it.
#
# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
RSpec.configure do |config|
# rspec-expectations config goes here. You can use an alternate
# assertion/expectation library such as wrong or the stdlib/minitest
# assertions if you prefer.
config.expect_with :rspec do |expectations|
# This option will default to `true` in RSpec 4. It makes the `description`
# and `failure_message` of custom matchers include text for helper methods
# defined using `chain`, e.g.:
# be_bigger_than(2).and_smaller_than(4).description
# # => "be bigger than 2 and smaller than 4"
# ...rather than:
# # => "be bigger than 2"
expectations.include_chain_clauses_in_custom_matcher_descriptions = true
end

# rspec-mocks config goes here. You can use an alternate test double
# library (such as bogus or mocha) by changing the `mock_with` option here.
config.mock_with :rspec do |mocks|
# Prevents you from mocking or stubbing a method that does not exist on
# a real object. This is generally recommended, and will default to
# `true` in RSpec 4.
mocks.verify_partial_doubles = true
end

# This option will default to `:apply_to_host_groups` in RSpec 4 (and will
# have no way to turn it off -- the option exists only for backwards
# compatibility in RSpec 3). It causes shared context metadata to be
# inherited by the metadata hash of host groups and examples, rather than
# triggering implicit auto-inclusion in groups with matching metadata.
config.shared_context_metadata_behavior = :apply_to_host_groups

# This allows you to limit a spec run to individual examples or groups
# you care about by tagging them with `:focus` metadata. When nothing
# is tagged with `:focus`, all examples get run. RSpec also provides
# aliases for `it`, `describe`, and `context` that include `:focus`
# metadata: `fit`, `fdescribe` and `fcontext`, respectively.
config.filter_run_when_matching :focus

# Limits the available syntax to the non-monkey patched syntax that is
# recommended. For more details, see:
# - http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/
# - http://www.teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/
# - http://rspec.info/blog/2014/05/notable-changes-in-rspec-3/#zero-monkey-patching-mode
config.disable_monkey_patching!

# This setting enables warnings. It's recommended, but in some cases may
# be too noisy due to issues in dependencies.
config.warnings = true

# Many RSpec users commonly either run the entire suite or an individual
# file, and it's useful to allow more verbose output when running an
# individual spec file.
if config.files_to_run.one?
# Use the documentation formatter for detailed output,
# unless a formatter has already been configured
# (e.g. via a command-line flag).
config.default_formatter = 'doc'
end

# Print the 10 slowest examples and example groups at the
# end of the spec run, to help surface which specs are running
# particularly slow.
config.profile_examples = 10

# Run specs in random order to surface order dependencies. If you find an
# order dependency and want to debug it, you can fix the order by providing
# the seed, which is printed after each run.
# --seed 1234
config.order = :random

# Seed global randomization in this process using the `--seed` CLI option.
# Setting this allows you to use `--seed` to deterministically reproduce
# test failures related to randomization by passing the same `--seed` value
# as the one that triggered the failure.
Kernel.srand config.seed
end
Loading