Skip to content

Commit

Permalink
Merge pull request #21 from zerlok/feature/brokrpc-amqp-options
Browse files Browse the repository at this point in the history 
add amqp options support in brokrpc
  • Loading branch information
@zerlok
zerlok authored Nov 28, 2024
2 parents 9f30b24 + 6c09798 commit 9a31c09
Show file tree
Hide file tree
Showing 39 changed files with 1,434 additions and 458 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/pull-request.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
path: ./.venv
key: venv-${{ hashFiles('poetry.lock') }}
- name: Install the project dependencies
run: poetry install --all-extras
run: poetry install --without=examples --all-extras
- name: Run mypy
run: poetry run mypy
- name: Run ruff check
Expand All @@ -49,7 +49,7 @@ jobs:
matrix:
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13" ]
protoc-version: [ "26.1" ]
buf-version: [ "1.31.0" ]
buf-version: [ "1.38.0" ]
steps:
- uses: actions/checkout@v4
- name: Install Python ${{ matrix.python-version }}
Expand All @@ -70,7 +70,7 @@ jobs:
path: ./.venv
key: venv-${{ hashFiles('poetry.lock') }}
- name: Install the project dependencies
run: poetry install --all-extras
run: poetry install --without=examples --all-extras
- name: Download protoc ${{ matrix.protoc-version }}
run: wget -q -O protoc.zip "https://github.com/protocolbuffers/protobuf/releases/download/v${{ matrix.protoc-version }}/protoc-${{ matrix.protoc-version }}-linux-x86_64.zip"
- name: Unzip protoc ${{ matrix.protoc-version }}
Expand Down
304 changes: 7 additions & 297 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

Generate Python modules from protobuf files.

## installation
## usage

[pypi package](https://pypi.python.org/pypi/pyprotostuben)

Expand All @@ -19,11 +19,13 @@ install with your favorite python package manager
pip install pyprotostuben
```

Then use protoc plugins to generate python code. See [greeting examples](examples/greeting/README.md) for more info.

## protoc plugins

### protoc-gen-mypy-stub

Generates python stubs (`*_pb2.pyi` & `*_pb2_grpc.pyi` files which then used by MyPy type checker / IDE syntax highlits
Generates python stubs (`*_pb2.pyi` & `*_pb2_grpc.pyi` files which then used by MyPy type checker / IDE syntax highlits
& suggestions).

**features:**
Expand All @@ -36,7 +38,7 @@ Generates python stubs (`*_pb2.pyi` & `*_pb2_grpc.pyi` files which then used by
**plugin options:**

* `message-mutable` -- add setters for fields, use mutable containers
* `message-all-init-args-optional` -- each field is optional in message constructor (event if required)
* `message-all-init-args-optional` -- each field is optional in message constructor (even if field is not optional)
* `grpc-sync` -- use sync grpc stubs instead of grpc.aio module and async defs
* `grpc-skip-servicer` -- don't generate code for servicers
* `grpc-skip-stub` -- don't generate code for stubs
Expand All @@ -45,7 +47,7 @@ Generates python stubs (`*_pb2.pyi` & `*_pb2_grpc.pyi` files which then used by

### protoc-gen-brokrpc

Generates `*_brokrpc.py` modules for [BrokRPC](https://github.com/zerlok/BrokRPC) framework. This is similar to gRPC
Generates `*_brokrpc.py` modules for [BrokRPC](https://github.com/zerlok/BrokRPC) framework. This is similar to gRPC
codegen (`*_pb2_grpc.py` modules).

**plugin options:**
Expand All @@ -59,297 +61,5 @@ Saves protoc plugin input to a file. Helps develop protoc plugins.

**plugin options:**

* `format={binary|json}` (default = `json`) -- specify output format
* `format={raw|binary|json}` (default = `raw`) -- specify output format
* `dest={path}` (default = `request.json`) -- specify file destination

## examples

requirements:

* [buf CLI](https://buf.build/product/cli)
* [protoc](https://grpc.io/docs/protoc-installation/)

### mypy stubs

the following project structure with protobuf file in `proto3` syntax & buf `v1beta1` condigs:

* `/`
* `src/`
* `greeting.proto`
```protobuf
syntax = "proto3";

package greeting;

// RPC request for greeting
message GreetRequest {
string name = 1;
}

// RPC response for greeting
message GreetResponse {
string text = 1;
}

// RPC service that provides greet functionality
service Greeter {
// RPC method for greeting
rpc Greet(GreetRequest) returns (GreetResponse) {}
}
```
* `buf.yaml`
```yaml
version: v1beta1
build:
roots:
- src
```
* `buf.gen.yaml`
```yaml
version: v1
managed:
enabled: true
plugins:
- plugin: pyprotostuben
out: out
strategy: all
```

run codegen

```bash
buf generate
```

the output:

* `src/greeting_pb2.pyi`
```python
import builtins
import google.protobuf.message
import typing
class GreetRequest(google.protobuf.message.Message):
"""RPC request for greeting"""
def __init__(self, *, name: builtins.str) -> None:...
@builtins.property
def name(self) -> builtins.str:...
def HasField(self, field_name: typing.NoReturn) -> typing.NoReturn:...
def WhichOneof(self, oneof_group: typing.NoReturn) -> typing.NoReturn:...
class GreetResponse(google.protobuf.message.Message):
"""RPC response for greeting"""
def __init__(self, *, text: builtins.str) -> None:...
@builtins.property
def text(self) -> builtins.str:...
def HasField(self, field_name: typing.NoReturn) -> typing.NoReturn:...
def WhichOneof(self, oneof_group: typing.NoReturn) -> typing.NoReturn:...
```
* `src/greeting_pb2_grpc.pyi`
```python
import abc
import builtins
import greeting_pb2
import grpc
import grpc.aio
import typing
class GreeterServicer(metaclass=abc.ABCMeta):
"""RPC service that provides greet functionality"""
@abc.abstractmethod
async def Greet(self, request: greeting_pb2.GreetRequest, context: grpc.aio.ServicerContext[greeting_pb2.GreetRequest, greeting_pb2.GreetResponse]) -> greeting_pb2.GreetResponse:
"""RPC method for greeting"""
...
def add_GreeterServicer_to_server(servicer: GreeterServicer, server: grpc.aio.Server) -> None:...
class GreeterStub:
"""RPC service that provides greet functionality"""
def __init__(self, channel: grpc.aio.Channel) -> None:...
def Greet(self, request: greeting_pb2.GreetRequest, *, timeout: typing.Optional[builtins.float]=None, metadata: typing.Optional[grpc.aio.MetadataType]=None, credentials: typing.Optional[grpc.CallCredentials]=None, wait_for_ready: typing.Optional[builtins.bool]=None, compression: typing.Optional[grpc.Compression]=None) -> grpc.aio.UnaryUnaryCall[greeting_pb2.GreetRequest, greeting_pb2.GreetResponse]:
"""RPC method for greeting"""
...
```

### BrokRPC

requirements:

* [BrokRPC](https://github.com/zerlok/BrokRPC) -- to run RPC server & client code. Installed with pip `pip install BrokRPC[aiormq]`.

create files:

* `buf.yaml`
```yaml
version: v2
modules:
- path: src
lint:
use:
- DEFAULT
breaking:
use:
- FILE
```
* `buf.gen.yaml`
```yaml
version: v2
managed:
enabled: true
plugins:
# generates python protobuf message modules (the builtin protoc python codegen)
- protoc_builtin: python
out: out
# generates python mypy stubs for protobuf messages (stub generator from pyprotostuben)
- protoc_builtin: mypy-stub
out: out
# generates brokrpc files with server interface and client factory (also from pyprotostuben)
- protoc_builtin: brokrpc
out: out
strategy: all
```
* `src/greeting.proto`
```protobuf
syntax = "proto3";
package greeting;
// the greet request
message GreetRequest {
string name = 1;
}
// the greet response
message GreetResponse {
string text = 1;
}
// the greeter service
service Greeter {
// the greet method
rpc Greet(GreetRequest) returns (GreetResponse) {}
}
```

then run codegen

```shell
buf generate
```

output:

* `out/greeting_pb2.py` -- generated protobuf messages
* `out/greeting_pb2.pyi` -- generated mypy stubs for messages
* `out/greeting_brokrpc.py`
```python
"""Source: greeting.proto"""
import abc
import brokrpc.rpc.abc
import brokrpc.rpc.client
import brokrpc.rpc.model
import brokrpc.rpc.server
import brokrpc.serializer.protobuf
import contextlib
import greeting_pb2
import typing
class GreeterService(metaclass=abc.ABCMeta):
"""the greeter service"""
@abc.abstractmethod
async def greet(self, request: brokrpc.rpc.model.Request[greeting_pb2.GreetRequest]) -> greeting_pb2.GreetResponse:
"""the greet method"""
raise NotImplementedError
def add_greeter_service_to_server(service: GreeterService, server: brokrpc.rpc.server.Server) -> None:
server.register_unary_unary_handler(func=service.greet, routing_key='/greeting/Greeter/greet', serializer=brokrpc.serializer.protobuf.RPCProtobufSerializer(greeting_pb2.GreetRequest, greeting_pb2.GreetResponse))
class GreeterClient:
"""the greeter service"""
def __init__(self, greet: brokrpc.rpc.abc.Caller[greeting_pb2.GreetRequest, greeting_pb2.GreetResponse]) -> None:
self.__greet = greet
async def greet(self, request: greeting_pb2.GreetRequest) -> brokrpc.rpc.model.Response[greeting_pb2.GreetResponse]:
"""the greet method"""
return await self.__greet.invoke(request)
@contextlib.asynccontextmanager
async def create_client(client: brokrpc.rpc.client.Client) -> typing.AsyncIterator[GreeterClient]:
async with client.unary_unary_caller(routing_key='/greeting/Greeter/greet', serializer=brokrpc.serializer.protobuf.RPCProtobufSerializer(greeting_pb2.GreetRequest, greeting_pb2.GreetResponse)) as greet:
yield GreeterClient(greet=greet)
```

now on the server side you can implement the `GreeterService`, pass it to `add_greeter_service_to_server` and run the
**RPC server**.

```python
import asyncio
from brokrpc.broker import Broker
from brokrpc.options import ExchangeOptions
from brokrpc.rpc.model import Request
from brokrpc.rpc.server import Server
from greeting_pb2 import GreetRequest, GreetResponse
from greeting_brokrpc import GreeterService, add_greeter_service_to_server
class MyService(GreeterService):
async def greet(self, request: Request[GreetRequest]) -> GreetResponse:
return GreetResponse(text=f"hello, {request.body.name}")
async def main() -> None:
broker = Broker("amqp://guest:guest@localhost:5672/", default_exchange=ExchangeOptions(name="test-greet-codegen"))
server = Server(broker)
add_greeter_service_to_server(MyService(), server)
await server.run_until_terminated()
if __name__ == "__main__":
asyncio.run(main())
```

on the client side you just have to get the client from `create_client` using **RPC client** and you ready to send
requests to greeter RPC server.

```python
import asyncio
from brokrpc.broker import Broker
from brokrpc.options import ExchangeOptions
from brokrpc.rpc.client import Client
from greeting_pb2 import GreetRequest, GreetResponse
from greeting_brokrpc import GreeterClient, create_client
async def main() -> None:
async with Broker("amqp://guest:guest@localhost:5672/", default_exchange=ExchangeOptions(name="test-greet-codegen")) as broker:
client_session = Client(broker)
greeting_client: GreeterClient
async with create_client(client_session) as greeting_client:
response = await greeting_client.greet(GreetRequest(name="Bob"))
print(response)
assert isinstance(response.body, GreetResponse)
print(response.body.text) # hello, Bob
if __name__ == "__main__":
asyncio.run(main())
```
Loading

0 comments on commit 9a31c09

Please sign in to comment.