Support for tooling API composite request

[pgonzaleznetwork]

HappySoup.io uses the sfdc-soup library behind the scenes. In turn, this library uses the sfdc-happy-api library (this repo), which is a tiny wrapper around some of the API operations that HappySoup needs.

Here's an example of how sfdc-soup uses the rest api exposed by sfdc-happy-api

https://github.com/pgonzaleznetwork/sfdc-soup/blob/bd1827cd1eec39094a4176558faf95536babb4b5/lib/sfdc_apis/metadata-types/CustomField.js#L107

Rest API: https://github.com/pgonzaleznetwork/sfdc-happy-api/blob/main/lib/rest.js

We should build a function inside the rest API module to support a composite request.

Spec:

• The function should be a closure, like all other functions inside the rest API module. This is so that the function can use the connection object that is passed to the outer function.

• The function should take a list of URLs. The list can be of any size, i.e 100 URLs should be allowed. Each URL represents an endpoint that can be passed to the composite request.

• Because the composite requests accepts a max of 25 URLs, the function should split the 100 (or whatever number) URLS, into batches of 25 or less (in case there are left overs). We already have similar functionality here

sfdc-happy-api/lib/rest.js

Line 173 in 698d6aa

async function tryWithSmallerQueries(queryString,endpoint,options){

and here

sfdc-happy-api/lib/rest.js

Line 303 in 698d6aa

function splitInBatchesOf(items,batchSize){

• Once we have batches of up to 25 URLs each, we should use Promise.all to issue all the composite requests. So essentially we are issuing multiple composite requests at the same time, each with 25 URLs max. You can see a similar implementation of Promise.all here

sfdc-happy-api/lib/rest.js

Line 189 in 698d6aa

let data = await Promise.all(

The function should return an array of all the responses.

• Proper error handling should be used to allow one or more requests to fail for whatever reason. Those that succeed should have their return value added to the returned array. In other words, if we issue 5 composite request and one fails, the response for the other 4 should be returned.

[brandonrwind]

@pgonzaleznetwork Had a few quick questions about how flexible we want this method to be.

Can you confirm that we only would want this method to create composite requests for the tooling API? That means I can assume that we'll always hit /services/data/v${apiVersion}/tooling/composite for these? Or would there ever be a use case to leverage the composite resource at /services/data/v${apiVersion}/composite?

And to follow up on that, if we can assume that we'll be using the tooling composite resource only, then can I hard code the required subrequest Method (would always be GET) and referenceId (some sort of incremented variable)?

[pgonzaleznetwork]

@brandonrwind Good questions. The client code will call the function passing a requests object, as follows

requests = {
	useToolingApi:true,
	urls:[url1,url2,url3]
}

If useToolingApi is true, then we should use the tooling endpoint. We might have some scenarios where issuing a composite request to the non-tooling endpoint will be necessary.

We already use this pattern when issuing queries against the REST API

sfdc-happy-api/lib/rest.js

Line 36 in 698d6aa

if(soqlQuery.useToolingApi){

That said, HappySoup will never (at least not any time soon) edit metadata/data in the customer's org, so we should never issue a PATCH/POST request (except for the composite request itself), so it is safe at this point to hardcode the requests to use GET.

Also, what is the referenceId used for?

[pgonzaleznetwork]

@brandonrwind another thing is that the API version to use is derived from the connection object that is used to instantiate the API, i.e

/**
* @token A session id or oauth token with API access
* @url Your instance url i.e login.salesforce.com or mydomain.my.salesforce.com
* @apiVersion the version of the Salesforce API. If not specified or if it's lower than 49.0, we use 49.0 by default
*/
let connection = {
    token: '00D0O000000Z9Ab!AQMAQMRoLQKh_uBxWEvz3as7V...',
    url:'https://resourceful-moose-448750-dev-ed.my.salesforce.com',
    apiVersion:'49.0'
};

This object is passed (and destructured) to the endpoints function and that returns the API endpoint along with the API version specified in the connection object

sfdc-happy-api/lib/rest.js

Line 9 in 698d6aa

let restEndpoint = endpoints(connection).restApi;

I recommend browsing the other module sfdc-soup to see how the REST API is instantiated and used

https://github.com/pgonzaleznetwork/sfdc-soup/blob/master/lib/sfdc_apis/usage.js

Our new function would live inside this REST API module.