Order workflow discussion [FEATURE]

[cgsmith]

Is your feature request related to a problem? Please describe.
Currently there is a separate set of scripts that pulls from Shopify, Amazon, WooCommerce, etc. This request will resolve this by creating the necessary jobs to allow webhooks to push orders into the Shipwise system. We will also be able to leverage this logic in the future for any ecommerce customers that still need us to pull in data (FTP adapters for example).

Describe the solution you'd like
The solution should take into account the following that I have seen over the years. For example, WooCommerce will fire only one webhook - even if it has a 500, it will not retry. Here are the other points:

1. Can be unreliable (Hookdeck will assist with retries on webhook requests).
2. May require additional calls (BigCommerce for example sends a webhook but requires 3 separate API calls to get all data)
3. If the order cannot be updated on Shipwise we will need to notify the customer
4. If the order can be updated on Shipwise we should update it with the upstream warehouse (notification if not possible)
5. API calls are sometimes rate-limited
6. Job failures MUST notify someone. If the order doesn't go through it can be expensive to track and find out why

Additional context
Here is a current workflow I'd like your comments on.

https://whimsical.com/shipwise-integration-flow-8NqqbeSAWSHWL4uCahqj6Z

Considerations

• Keep as separate jobs (queuing a job, processing a job, updating upstream)
• Possibly create a summary email if there are a lot of failed jobs
• Finish Add warehouse identifier to orders #98

Possible Schema

• id
• timestamp
• received from
• type: order create, order cancel, etc...
• payload
• status ( received, processing, success, failed)
• error log / general log messages to log details about the processing
• optional: retry counter

[cgsmith]

Adding a warehouse identifier will be helpful to making sure upstream orders can be updated or deleted: #98

[samdark]

Some questions about the schema:

1. In "Update Delete Order Job" what if upstread service isn't available? What if Cancelling failed?
2. In the "General Job Workflow":

• what's meant by "Order Open?"
• are webooks sent by upstream services? If so, won't an update at the upstream cause a webhook to fire that would cause update at the upstread etc.?
• What's "3pl"?

[cebe]

For stability and traceability it may be useful to decouple webhook receiving and webhook processing. This is also good for performance in case we receive a lot of webhook calls. They do not need to be processed directly.

1. Webhook endpoint just receives and validates the webhook input, stores it into a table or queue.
   This way we automatically have a log of webhooks and can monitor the state of it.
2. Queue jobs are triggered to process received web hooks and update their state, e.g. received -> processing -> success/failed
3. On failure someone gets notified to check the job and can manually trigger a retry or fix something.

Webhook table should include:

• id
• timestamp
• received from
• type: order create, order cancel, etc...
• payload
• status ( received, processing, success, failed)
• error log / general log messages to log details about the processing
• optional: retry counter

[cgsmith]

@samdark

1. In "Update Delete Order Job" what if upstread service isn't available? What if Cancelling failed?
2. In the "General Job Workflow":

• what's meant by "Order Open?"
• are webooks sent by upstream services? If so, won't an update at the upstream cause a webhook to fire that would cause update at the upstread etc.?
• What's "3pl"?

Update/Delete Order Job: If cancelling or updating fails upstream then a notification should be sent (maybe a summary of failures since this may be spammy?

Webhooks: Webhooks are currently not sent by upstream services but will send if an order is updated. We would then need to send an update to the 3PL or service upstream from us. Might be best as a separate job.

3PL: Third party logistics center. These are warehouses that Shipwise customers contract to ship their products
Order Open: Means the order was created in Shipwise and needs to be sent to the warehouse

[cgsmith]

For stability and traceability it may be useful to decouple webhook receiving and webhook processing.

Yes I'll include this in the main description. I was just thinking this after reading @samdark reply.

Do you think adding a rawContents to the webhook table would make sense? This may help troubleshooting in the future. Could always truncate or purge data older than 90 days.

[cgsmith]

Just saw the payload. Ignore my last question 🙂

[samdark]

Separate webhooks is a very good idea as well as queue for processing these.