You can use the batch aggregator scope to accumulate a subset of records from a batch step and bulk upsert them to an external source or service.
You can aggregate records to process in two ways:
-
Aggregating a fixed amount of records
-
Streaming all records
The batch aggregator is mutable. This means that you can access the payloads and variables of the records grouped on your batch aggregator scope.
When aggregating a fixed amount of records, you can access each record sequentially, or you can specify a random record to modify. However, if you configured your batch aggregator to stream its content, you can only access those records sequentially.
See the links below for an explanation of each aggregator configuration.
The batch aggregator scope can only exist in batch steps which, in turn, are only usable within the batch process phase of a batch job. You cannot use batch aggregators during the On Complete phase of a batch job.
An aggregator can only wrap the final element within the batch step in which it resides.
Several Anypoint Connectors have the ability to handle record-level errors without failing a whole batch aggregation. At runtime, these connectors keep track of which records were successfully accepted by the target resource, and which failed to upsert. Thus, rather than failing a complete group of records, the connector upserts as many records as it can, and tracks any failures for notification. These connectors are:
-
Salesforce
-
NetSuite
-
Database
The batch aggregator scope does not support job-instance-wide transactions. You can define a transaction inside a batch step that processes each record in a separate transaction. Think of it like a step within a step.
Such a transaction must start and end within the step’s boundaries.
You cannot share a transaction between a batch step and a batch aggregator that exists within the step. Any transaction that the batch step starts, ends before the batch aggregator begins processes. In other words, a transaction cannot cross the barrier between a batch step and the batch aggregator scope it contains.