-
-
Notifications
You must be signed in to change notification settings - Fork 28
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
b47a41f
commit de8c49b
Showing
20 changed files
with
446 additions
and
49 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
# Yii Queue Change Log | ||
# Yii Queue Extension Change Log | ||
|
||
## 1.0.0 under development | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,3 @@ | ||
Upgrading Instructions | ||
====================== | ||
|
||
This file contains the upgrade notes. These notes highlight changes that could break your | ||
application when you upgrade the package from one version to another. | ||
# Yii Queue Extension Upgrading Instructions | ||
|
||
## Changes summary |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Yii Queue extension | ||
|
||
An extension for running tasks asynchronously via queues. | ||
|
||
## Guides and concept explanations | ||
|
||
* [Usage basics](usage.md) | ||
* [Migrating from `yii2-queue`](migrating-from-yii2-queue.md) | ||
* [Errors and retryable jobs](error-handling.md) | ||
* [Workers](worker.md) | ||
* [Adapter list](adapter-list.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Queue Adapters | ||
------------- | ||
|
||
* [Synchronous](adapter-sync.md) - adapter for development and test environments | ||
* [AMQP](https://github.com/yiisoft/queue-amqp) - adapter over AMQP protocol via [amqplib](https://github.com/php-amqplib/php-amqplib) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
Synchronous Adapter | ||
================== | ||
|
||
Run tasks synchronously in the same process. It could be used when developing and debugging an application. | ||
|
||
Configuration example: | ||
|
||
```php | ||
$logger = $DIContainer->get(\Psr\Log\LoggerInterface::class); | ||
|
||
$worker = $DIContainer->get(\Yiisoft\Queue\Worker\WorkerInterface::class); | ||
$loop = $DIContainer->get(\Yiisoft\Queue\Cli\LoopInterface::class); | ||
$adapter = new Yiisoft\Queue\Adapter\SynchronousAdapter($loop, $worker); | ||
|
||
$queue = new Yiisoft\Queue\Queue( | ||
$adapter, | ||
$worker, | ||
$loop, | ||
$logger | ||
); | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,90 @@ | ||
# Error handling on message processing | ||
|
||
Often when some message handling is failing, we want to retry its execution a couple more times or redirect it to another queue channel. This can be done in `yiisoft/queue` with _Failure Handling Middleware Pipeline_. It is triggered each time message processing via Consume Middleware Pipeline is interrupted with any `Throwable`. | ||
|
||
## Configuration | ||
|
||
Here below is configuration via `yiisoft/config`. If you don't use it - you should add middleware definition list (in the `middlewares-fail` key here) to the `FailureMiddlewareDispatcher` by your own. | ||
|
||
Configuration should be passed to the `yiisoft/queue.fail-strategy-pipelines` key of the `params` config to work with the `yiisoft/config`. You can define different failure handling pipelines for each queue channel. Let's see and describe an example: | ||
|
||
```php | ||
'yiisoft/queue' => [ | ||
'middlewares-fail' => [ | ||
FailureMiddlewareDispatcher::DEFAULT_PIPELINE => [ | ||
[ | ||
'class' => SendAgainMiddleware::class, | ||
'__construct()' => ['id' => 'default-first-resend', 'queue' => null], | ||
], | ||
static fn (QueueFactoryInterface $factory) => new SendAgainMiddleware( | ||
id: 'default-second-resend', | ||
queue: $factory->get('failed-messages'), | ||
), | ||
], | ||
|
||
'failed-messages' => [ | ||
[ | ||
'class' => ExponentialDelayMiddleware::class, | ||
'__construct()' => [ | ||
'id' => 'failed-messages', | ||
'maxAttempts' => 30, | ||
'delayInitial' => 5, | ||
'delayMaximum' => 60, | ||
'exponent' => 1.5, | ||
'queue' => null, | ||
], | ||
], | ||
], | ||
], | ||
] | ||
``` | ||
|
||
Keys here except `FailureMiddlewareDispatcher::DEFAULT_PIPELINE` are queue channel names, and values are lists of `FailureMiddlewareInterface` definitions. `FailureMiddlewareDispatcher::DEFAULT_PIPELINE` defines a default pipeline to apply to channels without explicitly defined failure strategy pipeline. Each middleware definition must be one of: | ||
- A ready-to-use `MiddlewareFailureInterface` object like `new FooMiddleware()`. | ||
- A valid definition for the [yiisoft/definitions](https://github.com/yiisoft/definitions). It must describe an object, implementing the `MiddlewareFailureInterface`. | ||
- A callable: `fn() => // do stuff`, `$object->foo(...)`, etc. It will be executed through the `yiisoft/injector`, so all the dependencies of your callable will be resolved. You can also define a "callable-looking" array, where object will be instantiated with a DI container: `[FooMiddleware::class, 'handle']`. | ||
- A string for your DI container to resolve the middleware, e.g. `FooMiddleware::class`. | ||
|
||
In the example above failures will be handled this way (look the concrete middleware description below): | ||
|
||
1. For the first time message will be resent to the same queue channel immediately. | ||
2. If it fails again, it will be resent to the queue channel named `failed-messages`. | ||
3. From now on it will be resent to the same queue channel (`failed-messages`) up to 30 times with a delay from 5 to 60 seconds, increased 1.5 times each time the message fails again. | ||
4. If the message handler throw an exception one more (33rd) time, the exception will not be caught. | ||
|
||
Failures of messages, which are initially sent to the `failed-messages` channel, will only be handled by the 3rd and the 4th points of this list. | ||
|
||
## Default failure handling strategies | ||
|
||
Let's see the built-in defaults. | ||
|
||
### SendAgainMiddleware | ||
|
||
This strategy simply resends the given message to a queue. Let's see the constructor parameters through which it's configured: | ||
|
||
- `id` - A unique string. Allows to use this strategy more than once for the same message, just like in example above. | ||
- `maxAttempts` - Maximum attempts count for this strategy with the given $id before it will give up. | ||
- `queue` - The strategy will send the message to the given queue when it's not `null`. That means you can use this strategy to push a message not to the same queue channel it came from. When the `queue` parameter is set to `null`, a message will be sent to the same channel it came from. | ||
|
||
### ExponentialDelayMiddleware | ||
|
||
This strategy does the same thing as the `SendAgainMiddleware` with a single difference: it resends a message with an exponentially increasing delay. The delay **must** be implemented by the used `AdapterInterface` implementation. | ||
|
||
It's configured via constructor parameters, too. Here they are: | ||
|
||
- `id` - A unique string allows to use this strategy more than once for the same message, just like in example above. | ||
- `maxAttempts` - Maximum attempts count for this strategy with the given $id before it will give up. | ||
- `delayInitial` - The initial delay that will be applied to a message for the first time. It must be a positive float. | ||
- `delayMaximum` - The maximum delay which can be applied to a single message. Must be above the `delayInitial`. | ||
- `exponent` - Message handling delay will be muliplied by exponent each time it fails. | ||
- `queue` - The strategy will send the message to the given queue when it's not `null`. That means you can use this strategy to push a message not to the same queue channel it came from. When the `queue` parameter is set to `null`, a message will be sent to the same channel it came from. | ||
|
||
## How to create a custom Failure Middleware? | ||
|
||
All you need is to implement the `MiddlewareFailureInterface` and add your implementation definition to the [configuration](#configuration). | ||
This interface has the only method `handle`. And the method has these parameters: | ||
- `ConsumeRequest $request` - a request for a message handling. It consists of a message and a queue the message came from. | ||
- `Throwable $exception` - an exception thrown on the `request` handling | ||
- `MessageFailureHandlerInterface $handler` - failure strategy pipeline continuation. Your Middleware should call `$pipeline->handle()` when it shouldn't interrupt failure pipeline execution. | ||
|
||
> Note: your strategy have to check by its own if it should be applied. Look into [`SendAgainMiddleware::suites()`](../../src/Middleware/Implementation/FailureMiddleware/Middleware/SendAgainMiddleware.php#L52) for an example. |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# Migrating from yii2-queue | ||
|
||
This package is similar to [yiisoft/yii2-queue] but with improved design and code style. The new package less coupled | ||
and more structured than the old one allowing better maintenance. | ||
|
||
## Adapters | ||
|
||
- Individual adapters are now separate packages. This means each adapter must be `require`d with composer in order to | ||
be available in your application. | ||
- Adapter may be any class which implements `AdapterInterface`. This means you can replace one adapter with another without | ||
changing any code in your app. For example, you can use `db` adapter in development while using `amqp` in production, | ||
then you may seamlessly switch to `redis` if needed. This also means you can write your own adapter implementation | ||
if necessary. | ||
|
||
## Jobs (Messages and Handlers) | ||
|
||
There was a concept in [yiisoft/yii2-queue] called `Job`: you had to push it to the queue, and it was executed after | ||
being consumed. In the new package it is divided into two different concepts: a message and a handler. | ||
|
||
- A `Message` is a class implementing `MessageInterface`. It contains 2 types of data: | ||
- Name. Worker uses it to find the right handler for a message. | ||
- Data. Any serializable data which should be used by the message handler. | ||
|
||
All the message data is fully serializable (that means message `data` must be serializable too). It allows you to | ||
freely choose where and how to send and process jobs. Both can be implemented in a single application, or | ||
separated into multiple applications, or you can do sending/processing only leaving part of the job to another | ||
system including non-PHP ones. It is fairly popular to process heavy jobs with Go. | ||
|
||
- A `Handler` is called by a `Worker` when a message comes. Default `Worker` finds a corresponding message handler | ||
by the message name. [See more](worker.md#handler-format). | ||
|
||
[yiisoft/yii2-queue]: (https://github.com/yiisoft/yii2-queue) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
# Usage basics | ||
|
||
## Configuration | ||
|
||
You can configure it with a DI container in the following way: | ||
|
||
```php | ||
$logger = $DIContainer->get(\Psr\Log\LoggerInterface::class); | ||
|
||
$worker = $DIContainer->get(\Yiisoft\Queue\Worker\WorkerInterface::class); | ||
$loop = $DIContainer->get(\Yiisoft\Queue\Cli\LoopInterface::class); | ||
$adapter = $DIContainer->get(\Yiisoft\Queue\Adapter\AdapterInterface::class); | ||
|
||
$queue = new Queue( | ||
$adapter, | ||
$worker, | ||
$loop, | ||
$logger | ||
); | ||
``` | ||
|
||
See also the documentation for concrete adapters ([synchronous adapter](adapter-sync.md), | ||
[AMQP adapter](https://github.com/yiisoft/queue-amqp)) and [workers](worker.md) | ||
|
||
|
||
## Usage | ||
|
||
Each job sent to the queue should be defined as a separate class. | ||
For example, if you need to download and save a file the class may look like the following: | ||
|
||
```php | ||
$data = [ | ||
'url' => $url, | ||
'destinationFile' => $filename, | ||
]; | ||
$message = new \Yiisoft\Queue\Message\Message('file-download', $data); | ||
``` | ||
|
||
Here's how to send a task to the queue: | ||
|
||
```php | ||
$queue->push($message); | ||
``` | ||
|
||
To push a job into the queue that should run after 5 minutes: | ||
|
||
```php | ||
// TODO | ||
``` | ||
|
||
**Important:** Not every adapter (such as synchronous adapter) supports delayed execution. | ||
|
||
|
||
## Queue handling | ||
|
||
The exact way how a job is executed depends on the adapter used. Most adapters can be run using | ||
console commands, which the component registers in your application. For more details check the respective | ||
adapter documentation. | ||
|
||
|
||
## Job status | ||
|
||
```php | ||
// Push a job into the queue and get a message ID. | ||
$id = $queue->push(new SomeJob()); | ||
|
||
// Get job status. | ||
$status = $queue->status($id); | ||
|
||
// Check whether the job is waiting for execution. | ||
$status->isWaiting(); | ||
|
||
// Check whether a worker got the job from the queue and executes it. | ||
$status->isReserved($id); | ||
|
||
// Check whether a worker has executed the job. | ||
$status->isDone($id); | ||
``` | ||
|
||
## Limitations | ||
|
||
When using queues it is important to remember that tasks are put into and obtained from the queue in separate | ||
processes. Therefore, avoid external dependencies when executing a task if you're not sure if they are available in | ||
the environment where the worker does its job. | ||
|
||
All the data to process the task should be provided with your payload `getData()` method. |
Oops, something went wrong.