Should a subscriber process crash on callback error?

[asfaltboy]

I am in the process of gathering information on the differences around GCP libraries, between python and node.js. I am doing this to help my team have a unified set of tools and to simplify the abstractions.

I'd like to discuss one major difference in 'default' behavior between the clients: handling callback errors.

The python client currently catches exceptions, logs the error and calls message.nack().

The node.js client on the other hand, simply allows it to bubble up, which crashes the node.js process and stops the program.

There seems to be a fundamental (philosophical?) divide between the two approaches. The Node.js client assumes it's users will simply always add a catch handler if they want to handle errors. The Python client considers it's a very common scenario, and tries to protect it's users by making a choice for them.

Both of these default behaviors have a different set of impacts on developers:

With Node.js:

1. Workloads will crash (incurring downtime, e.g in the form of CrashLoopBackOff)
2. Application developers need to remember to add the catch, or tool builders need to ship a wrapper library to protect their dev users
3. Automated observability wrappers (e.g Datadog's auto-instrumentation) is ineffective in tracking the errors, as it never sees the errors if the application catches them. Which leads to needing the observability tool to be manually called for each error being handled

With Python, uncaught failures are nack-ed, causing pubsub server to retry immediately. Which means

1. The redelivery incurs an egress cost
2. The workload re-processing incurs a resource cost - which is fine for recoverable errors, but a waste in terminal errors.

My personal opinion is that the generally cost of development (and maintenance) is higher than the GCP costs associated with redelivery, but I don't have no exact numbers to back this up.

I would love to hear what everyone thinks (both users of GCP, and code owners ideally)? Is this by design, or an arbitrary choice? Should the libraries align or keep to their separate default behaviors? Are there any other repercussions or nuance to the behaviors that I've not registered?