From b3a6882c21ccffd9408b103d8a4c273cfbaae39b Mon Sep 17 00:00:00 2001 From: "Nikola Svitlica a.k.a TheCelavi" Date: Wed, 14 Aug 2024 10:32:30 +0200 Subject: [PATCH] Completed version 8 --- CHANGELOG.md | 27 ++++ README.md | 5 +- docs/doctrine-dbal-executor-result.md | 2 +- docs/faq.md | 11 ++ docs/index.md | 2 +- docs/using-manager.md | 174 -------------------------- 6 files changed, 43 insertions(+), 178 deletions(-) create mode 100644 CHANGELOG.md delete mode 100644 docs/using-manager.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..f4ccf57 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,27 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](http://keepachangelog.com/)and this project adheres to +[Semantic Versioning](http://semver.org/). + +## Changelog + +## [8.0.0] - 2024-08-14 + +### Added + +- New API for interacting with query resources loader, + `RunOpenCode\Bundle\QueryResourcesLoader\Contract\QueryResourcesLoaderInterface`. +- Support for middleware in query resources loader. +- Support for caching of query results. + +### Deprecated + +- `RunOpenCode\Bundle\QueryResourcesLoader\Contract\ManagerInterface` is deprecated and will be removed in version 9. +- `RunOpenCode\Bundle\QueryResourcesLoader\Contract\ExecutorInterface` is deprecated and will be removed in version 9. +- No support for `iterate()` method in `ManagerInterface`. + +### Removed + +- Support for iterating over records/tables in library. \ No newline at end of file diff --git a/README.md b/README.md index 2c3daa2..c541730 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,9 @@ application that deals with reporting. You can control transaction isolation level for current statements within transaction. - **Distributed transactions**. You can execute multiple queries within same transaction against different databases. If - **Caching**. You can cache your query results, so they are not loaded from database on each execution. +- **Middlewares**. You can use middlewares to manipulate query before execution, or to manipulate result after + execution. You can switch to other database if query fails, you can add monitoring, logging, load balancing on several + databases, etc... Read the documentation [here](docs/index.md). @@ -141,5 +144,3 @@ For other details about this bundle, as well as for tips on how to use it, read ## TODO - Add profiling for middlewares and query execution. -- Add changelog. -- Improve documentation. diff --git a/docs/doctrine-dbal-executor-result.md b/docs/doctrine-dbal-executor-result.md index 7565a48..ab0f1ac 100644 --- a/docs/doctrine-dbal-executor-result.md +++ b/docs/doctrine-dbal-executor-result.md @@ -26,4 +26,4 @@ for more details. Note that `ExecutionResultInterface` implements `\Traversable` and `\Countable`. -[<< Using manager](using-manager.md) | [Table of contents](index.md) | [Transaction support >>](transactions.md) +[<< Twig support](legacy-support) | [Table of contents](index.md) | [Transaction support >>](transactions.md) diff --git a/docs/faq.md b/docs/faq.md index 968bca8..0a9d347 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -18,5 +18,16 @@ That being said, `dmaicher/doctrine-test-bundle`tracks transactions of `Connecti `START TRANSACTION` statement in your query, `dmaicher/doctrine-test-bundle` is not able to roll back them, because those are not explicitly tracked by `Connection` object. +## Legacy support + +Prior to version 8, main interface of the library was +`RunOpenCode\Bundle\QueryResourcesLoader\Contract\ManagerInterface`. This interface is replaced with +`RunOpenCode\Bundle\QueryResourcesLoader\Contract\QueryResourcesLoaderInterface` which should be used from now on. + +However, to support gradual migration, interface `ManagerInterface` is still available and can be used, but without +`iterate()` method which is not supported in legacy interface. API for iterating over records/tables will not be +introduced in future versions as part of this library. + +Support for `ManagerInterface` will be removed in version 9. [FAQ](faq.md) | [Table of contents](index.md) diff --git a/docs/index.md b/docs/index.md index 820c734..2299d93 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,7 +16,7 @@ of entities first, and then use those identifiers to fetch entities themselves. - [Introduction](introduction.md) - [Proposed solution](proposed-solution.md) -- [Using manager](using-manager.md) +- [Using manager](legacy-support) - [Twig support](twig-support.md) - [DoctrineDbalExecutorResult](doctrine-dbal-executor-result.md) - [Transaction support](transactions.md) diff --git a/docs/using-manager.md b/docs/using-manager.md deleted file mode 100644 index b56c00e..0000000 --- a/docs/using-manager.md +++ /dev/null @@ -1,174 +0,0 @@ -# Using manager - -Manager defines 3 public methods: - - /** - * Manager service provides Query source code from loaders, modifying it, if needed, as per concrete implementation of - * relevant manager and supported scripting language. Manager can execute a Query as well. - */ - interface ManagerInterface - { - /** - * Check if manager have the Query source code by its given name. - * - * @param string $name The name of the Query source to check if can be loaded. - * - * @return bool TRUE If the Query source code is handled by this manager or not. - */ - public function has(string $name): bool; - - /** - * Get Query source by its name. - * - * @param string $name Name of Query source code. - * @param array $args Arguments for modification/compilation of Query source code. - * - * @return string SQL statement. - */ - public function get(string $name, array $args = []): string; - - /** - * Execute Query source. - * - * @param string $name Name of Query source code. - * @param array $args Arguments for modification/compilation of Query source code, as well as params for query statement. - * @param array $types Types of parameters for prepared statement. - * @param array $options Any executor specific options (depending on concrete driver). - * @param null|string $executor Executor name. - * - * @return ExecutionResultInterface Execution results. - */ - public function execute(string $name, array $args = [], array $types = [], array $options = [], ?string $executor = null): ExecutionResultInterface; - - /** - * Execute query and iterate results in batches. - * - * Query is modified in order to accommodate LIMIT/OFFSET clauses, - * provided query must not contain mentioned statements. Purpose is to - * iterate rows without using table/database cursor and achieving small - * memory footprint on both application and database side. - * - * Options may contain additional keys, depending on concrete driver, - * but all contains the following: - * - * - iterate: string, how values should be yielded for each row. - * - batch_size: int, how many rows to process per query. - * - on_batch_end: callable, callable to invoke when batch is fully processed. - * - * Executor may provide for prepared statement "last_batch_row" with last row - * of previous batch which may be used for building of query for next batch. - * - * @param string $query Query to execute. - * @param array $parameters Parameters required for query. - * @param array $types Parameter types required for query. - * @param array|array{iterate?:string, batch_size?:int, on_batch_end?: callable} $options Any executor specific options (depending on concrete driver). - * - * @return IterateResultInterface Result of execution. - * - * @see \RunOpenCode\Bundle\QueryResourcesLoader\Contract\IterateResultInterface::ITERATE_* - */ - public function iterate(string $name, array $args = [], array $types = [], array $options = [], ?string $executor = null): IterateResultInterface; - } - - -While first parameter `$name` is bundle resource locator syntax of query -which you are trying to load/execute, second parameter allows you to use -parametrised query statements (prepared statements), as well as building -complex queries depending on passed arguments (see [Twig support](twig-support.md)). - -Third, optional, parameter `$types` allows you to explicitly specify type -of parameters for prepared statements, which is useful when using, per example, -`WHERE IN` in your SQL statements. - -## Query executor - -Method `get()` will just provide you with loaded and parsed query string, -while method `execute()` will execute the query and provide you with the -query result as instance of -`RunOpenCode\Bundle\QueryResourcesLoader\Contract\ExecutionResultInterface`. - -Note that used Manager implementation is not database agnostic, nor it -is intended to be. Library provides you, for now, with -`RunOpenCode\Bundle\QueryResourcesLoader\Executor\DoctrineDbalExecutor` -which can be used for executing queries against relational databases -supported by Doctrine Dbal. - -When executing SQL statements with `DoctrineDbalExecutor` (Dbal), result set is -returned as instance of -`RunOpenCode\Bundle\QueryResourcesLoader\Executor\DoctrineDbalExecutorResult` -class, which is a proxy to a `Doctrine\DBAL\Driver\Statement`. - -You can, of course, implement your own query executor, by implementing -`RunOpenCode\Bundle\QueryResourcesLoader\Contract\ExecutorInterface` interface -and registering it in service container with tag name -`runopencode.query_resources_loader.executor` with attribute `name` -which you can use when executing query in multi-executor environment. - -Default executor is the first registered executor, or, it can be -configured under `runopencode_query_resources_loader.default_executor` -key where you should state the service name of your executor, example: - - runopencode_query_resources_loader: - default_executor: my_executor_service_name - -## How to use manager - -It is strongly advised to inject manager into your repository class/service -as in example in a previous chapter, however, you can get service from service -container as well: - - $this->get('runopencode.query_loader')->execute('@MyAppReporting/common.ledger.sql', array( - 'year' => '2016', - 'limit' => '10000' - )); - -While in `MyAppReportingBundle/Resources/query/common.ledger.sql` there can -be a query that uses, per example, prepared statement: - - SELECT * FROM expenses_table ET - - WHERE - - ET.year = :year - - AND - - ET.budget <= :limit; - - -By using Twig within this bundle, you can do a really serious and complex -query building as well. - -## Iterate - -Consider that you have a table (or dataset) which you want to iterate trough with the smallest possible -memory footprint on both application and database side. In general, your motivation is batch processing. -There are several methods to achieve similar, starting from using plain statement, Doctrine ORM method -stated in documentation -[https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/batch-processing.html](https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/batch-processing.html), -special libraries for that purpose (like: [https://github.com/Ocramius/DoctrineBatchUtils](https://github.com/Ocramius/DoctrineBatchUtils)), -cursors on database levels, you name it. - -This library allows you to use `iterate()` method which will modify your query, appending `LIMIT` and `OFFSET` -and iterate your recordset in pages, offloading pressure from booth database and application level. You may -configure a batch size (number of items per page), how rows should be yielded (whole row or just first column) as -well as you may pass a callable which you want to invoke after each bach of records, per example: - -- you iterate through IDs of some entities -- you load each entity with ORM -- you modify each entity -- you flush your changes and clear entity manager for each batch - -(as in example given here: [https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/batch-processing.html](https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/batch-processing.html)) - -However, note that quality of application of this method depends on many factors and data consistency may be -jeopardised. Some useful recommendations: - -- always write queries without `LIMIT`/`OFFSET`. -- make sorting stable (per example, add at the end `ORDER BY id ASC` and, if possible, add order by some timestamped -field, like "created_at") -- iterate() is not executed within transaction, if data consistency can be impacted by that, wrap everything in -transaction, or lock affected tables, or use some other method to batch process data. - - -[<< Proposed solution](proposed-solution.md) | [Table of contents](index.md) | [Twig support >>](twig-support.md)