Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to use bulk api #26

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add option to use bulk api #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,18 +90,27 @@ Website used to generate the data https://www.mockaroo.com/
You can customize some options by editing MaskSObjectSettings__mdt Default Custom Metadata
- Allow execution on prod : enable this option to be able to run the batch on PROD (otherwise soql query returns no rows)
- Configure the number of digits to preserve in standard Phone fields.
- Use Bulk API to perform faster updates


<img alt="Customize options" src="./screenshots/settings.png" />

## Bulk API
Make sure to check for the [Limits](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/bulk_common_limits.htm) and [compatible objects](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/how_requests_are_processed.htm).

We recommend to start building your mask configuration with the bulk api option disable, it will be easier to check for errors at the beginning.
Once your configuration is OK, you can enable Bulk API to perform faster updates.

- Check for errors : sue the REST request to check for errors for a particular jobId "/services/data/v56.0/jobs/ingest/jobID/failedResults/"
> **Warning** At the moment, the number of errors displayed in the MaskSObject record is not updated when using Bulk API

## Manage Errors
Mask SObject Framework may throw some errors during the run, due to validation rules or implementation specificity in your org.
To be able to track and manage those errors, we implemented the MaskSObjectError__c object.

<img alt="Errors" src="./screenshots/error.png" />

If you want to, you can disable the error's record creation in the custom metadata settings.
> **Note** If you want to, you can disable the error's record creation in the custom metadata settings.

### Errors Purge
You can manage logs purge with the framework [SObject Purge Framework](https://github.com/tprouvot/purge-sobject)
Expand Down
83 changes: 83 additions & 0 deletions force-app/main/default/classes/BulkApiDml.cls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
/**
* This class exposes a method which allows to perform DML operations throught Bulk API
*/
public with sharing class BulkApiDml {
public static final String API_VERSION = '56.0';
public static Id jobId;
public static String SObjectName;

/**
* @description This method creates a Bulk job, uploads job data and close the job when upload is complete.
* @param fields List of fields to update
* @param records List of SObject to update
* @param operation DML operation ('insert', 'delete', 'hardDelete', 'update', 'upsert')
* @return Job ID
*/
public static Id importBulkdata(List<String> fields, List<SObject> records, String operation) {
String sessId = UserInfo.getSessionId();
SObjectName = records.getSObjectType().getDescribe().getName();

try {
if(!records.isEmpty()) {

//create new job
HttpResponse respJobId = getHttpResponse('POST', '', 'application/json', sessId, getJobIdBody(operation));

if(respJobId.getStatusCode() <= 299) {
Map<String, Object> respMap = (Map<String, Object>) JSON.deserializeUntyped(respJobId.getBody());
jobId = (String)respMap.get('id');

//send records to the server
HttpResponse resp = getHttpResponse('PUT', '/' + jobId + '/batches', 'text/csv', sessId, getCsvExport(fields, records));

if(resp.getStatusCode() <= 299) {
//close job
getHttpResponse('PATCH', '/' + jobId, 'application/json', sessId, '{ "state" : "UploadComplete" }');
}
}
}
} catch (Exception e) {
MaskSObjectUtils.saveError(e, jobId, SObjectName);
}
return jobId;
}

private static String getCsvExport(List<String> fields, List<SObject> records) {
String csv = String.join(fields, ',');
csv += '\n';

for (SObject record : records) {
for(String key : fields){
csv += record.get(key) + ',';
}
csv = csv.removeEnd(',');
csv += '\n';
}
return csv;
}

private static String getJobIdBody(String operation){
return '{ "externalIdFieldName": "Id", "lineEnding": "LF", "operation": "' + operation + '",' +
'"object": "' + SObjectName + '", "contentType": "CSV"}';
}

private static void saveError(HttpResponse resp){
String error = resp.toString() + ' : There was an error. Please contact your admin.';
MaskSObjectUtils.saveError(error, jobId, SObjectName);
}

private static HttpResponse getHttpResponse(String method, String endpoint, String contentType, String token, String body){
HttpRequest req = new HttpRequest();
req.setMethod(method);
req.setEndpoint(URl.getOrgDomainUrl().toExternalForm() + '/services/data/v' + API_VERSION + '/jobs/ingest' + endpoint);
req.setHeader('content-type', contentType);
req.setHeader('Authorization', 'Bearer ' + token);

req.setBody(body);
HttpResponse resp = new Http().send(req);
if(resp.getStatusCode() >= 299) {
saveError(resp);
}
return resp;
}
}
5 changes: 5 additions & 0 deletions force-app/main/default/classes/BulkApiDml.cls-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
<?xml version="1.0" encoding="UTF-8"?>
<ApexClass xmlns="http://soap.sforce.com/2006/04/metadata">
<apiVersion>56.0</apiVersion>
<status>Active</status>
</ApexClass>
17 changes: 12 additions & 5 deletions force-app/main/default/classes/MaskSObjectBatch.cls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/**
* @description This class is used to mask the database by performing records updates
*/
public with sharing class MaskSObjectBatch implements Database.Batchable<sObject>{
public with sharing class MaskSObjectBatch implements Database.Batchable<SObject>, Database.AllowsCallouts{

private List<MaskSObjectDictionaryModel> dictionary;

Expand Down Expand Up @@ -80,12 +80,14 @@ public with sharing class MaskSObjectBatch implements Database.Batchable<sObject
return Database.getQueryLocator(query);
} catch (Exception e) {
MaskSObjectUtils.saveError(e, job.Id, this.sobjectName);
return Database.getQueryLocator(query);
//return a empty queryLocator
return Database.getQueryLocator('SELECT Id FROM ' + this.sobjectName + ' WHERE Id=null');
}
}

public void execute(Database.BatchableContext BC, List<SObject> scope){
AsyncApexJob job;
Id bulkJobId;
try {
job = getJob(BC.getJobId());

Expand All @@ -102,11 +104,16 @@ public with sharing class MaskSObjectBatch implements Database.Batchable<sObject

SObjectType sObjType = ((SObject) Type.forName(this.sobjectName).newInstance()).getSObjectType();
if (sObjType.getDescribe().isUpdateable()) {
List<Database.SaveResult> srList = Database.update(scope, false);
MaskSObjectUtils.saveErrors(srList, job.Id, this.sobjectName);
if(MaskSObjectUtils.getOptions().UseBulkAPI__c){
this.queryFields.add('Id');
bulkJobId = BulkApiDml.importBulkdata(this.queryFields, scope, 'update');
} else{
List<Database.SaveResult> srList = Database.update(scope, false);
MaskSObjectUtils.saveErrors(srList, job.Id, this.sobjectName);
}
}

upsert new MaskSObject__c(APIName__c = this.sobjectName, TotalJobItems__c = job.TotalJobItems,
upsert new MaskSObject__c(APIName__c = this.sobjectName, TotalJobItems__c = job.TotalJobItems, LastBulkJobId__c = bulkJobId,
JobItemsProcessed__c = job.JobItemsProcessed + 1, NumberOfErrors__c = job.NumberOfErrors) APIName__c;
} catch (Exception e) {
MaskSObjectUtils.saveError(e, job.Id, this.sobjectName);
Expand Down
7 changes: 5 additions & 2 deletions force-app/main/default/classes/MaskSObjectUtils.cls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,8 +182,11 @@ public with sharing class MaskSObjectUtils {
}

public static void saveError(Exception e, Id jobId, String sobj){
if(!options.DisableLogError__c){
String error = e.getMessage() + ' ' + e.getStackTraceString();
saveError(e.getMessage() + ' ' + e.getStackTraceString(), jobId, sobj);
}

public static void saveError(String error, Id jobId, String sobj){
if(!getOptions().DisableLogError__c){
insert createError(null, error, jobId, sobj);
}
}
Expand Down
10 changes: 9 additions & 1 deletion force-app/main/default/customMetadata/MaskSObjectSettings.Default.md-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
<protected>false</protected>
<values>
<field>AllowExecutionOnProd__c</field>
<value xsi:type="xsd:boolean">true</value>
</values>
<values>
<field>DisableLogError__c</field>
<value xsi:type="xsd:boolean">false</value>
</values>
<values>
Expand All @@ -22,4 +26,8 @@
<field>ObfuscatedCharacters__c</field>
<value xsi:type="xsd:string">a,e,i,o,1,2,5,6</value>
</values>
</CustomMetadata>
<values>
<field>UseBulkAPI__c</field>
<value xsi:type="xsd:boolean">true</value>
</values>
</CustomMetadata>
99 changes: 98 additions & 1 deletion force-app/main/default/layouts/MaskSObject__c-Mask SObject Layout.layout-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@
<behavior>Edit</behavior>
<field>LastEnd__c</field>
</layoutItems>
<layoutItems>
<behavior>Edit</behavior>
<field>LastBulkJobId__c</field>
</layoutItems>
<layoutItems>
<behavior>Readonly</behavior>
<field>Duration__c</field>
Expand All @@ -85,6 +89,99 @@
<layoutColumns/>
<style>CustomLinks</style>
</layoutSections>
<platformActionList>
<actionListContext>Record</actionListContext>
<platformActionListItems>
<actionName>FeedItem.TextPost</actionName>
<actionType>QuickAction</actionType>
<sortOrder>0</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>LogACall</actionName>
<actionType>QuickAction</actionType>
<sortOrder>1</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_Send_Email</actionName>
<actionType>QuickAction</actionType>
<sortOrder>2</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_New_Account</actionName>
<actionType>QuickAction</actionType>
<sortOrder>3</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_Service_New_Article</actionName>
<actionType>QuickAction</actionType>
<sortOrder>4</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_Sales_New_Lead</actionName>
<actionType>QuickAction</actionType>
<sortOrder>5</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_New_Task</actionName>
<actionType>QuickAction</actionType>
<sortOrder>6</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_New_Event</actionName>
<actionType>QuickAction</actionType>
<sortOrder>7</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>SDO_Sales_NewOpportunity</actionName>
<actionType>QuickAction</actionType>
<sortOrder>8</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>RecordShareHierarchy</actionName>
<actionType>StandardButton</actionType>
<sortOrder>9</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>Clone</actionName>
<actionType>StandardButton</actionType>
<sortOrder>10</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>ChangeOwnerOne</actionName>
<actionType>StandardButton</actionType>
<sortOrder>11</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>Delete</actionName>
<actionType>StandardButton</actionType>
<sortOrder>12</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>Submit</actionName>
<actionType>StandardButton</actionType>
<sortOrder>13</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>Edit</actionName>
<actionType>StandardButton</actionType>
<sortOrder>14</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>ChangeRecordType</actionName>
<actionType>StandardButton</actionType>
<sortOrder>15</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>Share</actionName>
<actionType>StandardButton</actionType>
<sortOrder>16</sortOrder>
</platformActionListItems>
<platformActionListItems>
<actionName>PrintableView</actionName>
<actionType>StandardButton</actionType>
<sortOrder>17</sortOrder>
</platformActionListItems>
</platformActionList>
<relatedLists>
<fields>Name__c</fields>
<fields>APIName__c</fields>
Expand All @@ -97,7 +194,7 @@
<showRunAssignmentRulesCheckbox>false</showRunAssignmentRulesCheckbox>
<showSubmitAndAttachButton>false</showSubmitAndAttachButton>
<summaryLayout>
<masterLabel>00h1X000004kwJm</masterLabel>
<masterLabel>00h7Q00000BZIwJ</masterLabel>
<sizeX>4</sizeX>
<sizeY>0</sizeY>
<summaryLayoutStyle>Default</summaryLayoutStyle>
Expand Down
2 changes: 1 addition & 1 deletion ...fault/objects/MaskSObjectSettings__mdt/fields/ObfuscateDigitReplacement__c.field-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@
<required>true</required>
<type>Text</type>
<unique>false</unique>
</CustomField>
</CustomField>
11 changes: 11 additions & 0 deletions force-app/main/default/objects/MaskSObjectSettings__mdt/fields/UseBulkAPI__c.field-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>UseBulkAPI__c</fullName>
<defaultValue>true</defaultValue>
<description>Enable this checkbox to use the Bulk API to perform the updates.</description>
<externalId>false</externalId>
<fieldManageability>DeveloperControlled</fieldManageability>
<inlineHelpText>Enable this checkbox to use the Bulk API to perform the updates.</inlineHelpText>
<label>Use Bulk API</label>
<type>Checkbox</type>
</CustomField>
12 changes: 12 additions & 0 deletions force-app/main/default/objects/MaskSObject__c/fields/LastBulkJobId__c.field-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<?xml version="1.0" encoding="UTF-8"?>
<CustomField xmlns="http://soap.sforce.com/2006/04/metadata">
<fullName>LastBulkJobId__c</fullName>
<description>Last Bulk job Id</description>
<externalId>false</externalId>
<label>Last Bulk Job Id</label>
<length>18</length>
<required>false</required>
<trackTrending>false</trackTrending>
<type>Text</type>
<unique>false</unique>
</CustomField>
7 changes: 6 additions & 1 deletion force-app/main/default/permissionsets/MaskSObjectAdmin.permissionset-meta.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,11 @@
<field>MaskSObject__c.LastEnd__c</field>
<readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>MaskSObject__c.LastBulkJobId__c</field>
<readable>true</readable>
</fieldPermissions>
<fieldPermissions>
<editable>true</editable>
<field>MaskSObject__c.LastJobId__c</field>
Expand Down Expand Up @@ -120,4 +125,4 @@
<object>MaskSObject__c</object>
<viewAllRecords>true</viewAllRecords>
</objectPermissions>
</PermissionSet>
</PermissionSet>
Binary file modified screenshots/MaskSObject.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified screenshots/settings.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.