Annotate panics with provider metadata and origin #2871

t0yv0 · 2025-01-27T20:10:58Z

Recover panics in the bridged providers and emit them as engine error events with sufficient annotations about which provider, provider version, resource and method was responsible for the panic.

Fixes #2844

While the bridged providers should never panic, some WIP semantic discrepancy between TF and Pulumi still occasionally results in panics. These changes accomplish two purposes. First, it is easier for maintainers to immediately narrow down a panic to a resource or function involved to attempt a repro. Second, it is easier to scan for panics in internal logs and attribute them to providers unambiguously.

With these changes, an injected panic in pulumi-random looks as follows. It is unfortunately double-stated but the new error message correctly attributes it to the provider, resource and method:

pulumi up --yes                                 ~/code/pulumi-random/examples/simple/ts
Previewing update (anton-test-logging)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/simple-random/anton-test-logging/previews/3f77e6b6-287b-4115-8439-8a3e5f328f24

     Type                           Name                              Plan       Info
 +   pulumi:pulumi:Stack            simple-random-anton-test-logging  create     4 errors; 3 warnings
 +   ├─ random:index:RandomInteger  integer                           create     
 +   ├─ random:index:RandomUuid     uuid                              create     
     ├─ random:index:RandomString   string                                       2 errors
 +   └─ random:index:RandomPet      pet                               create     

Diagnostics:
  pulumi:pulumi:Stack (simple-random-anton-test-logging):
    warning: using pulumi-resource-random from $PATH at /Users/anton/code/pulumi-random/bin/pulumi-resource-random
    warning: using pulumi-resource-random from $PATH at /Users/anton/code/pulumi-random/bin/pulumi-resource-random
    warning: resource plugin random is expected to have version >=4.16.8, but has 4.0.0-alpha.0+dev; the wrong version may be on your path, or this may be a bug in the plugin

    error: 
    
             Detected that /Users/anton/code/pulumi-random/bin/pulumi-resource-random exited prematurely.
             This is *always* a bug in the provider. Please report the issue to the provider author as appropriate.
    
    To assist with debugging we have dumped the STDOUT and STDERR streams of the plugin:
    
    panic: FAILUREZ [recovered]
        panic: FAILUREZ
    
    goroutine 27 [running]:
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/providerserver.(*PanicRecoveringProviderServer).Check.func1()
        /Users/anton/code/pulumi-terraform-bridge/pkg/providerserver/panic_recovering_provider.go:306 +0xc8
    panic({0x1023e15e0?, 0x102779e50?})
        /nix/store/lb8wx7xsk7vryjxbbf9wlj820pdfvnbj-go-1.23.3/share/go/src/runtime/panic.go:785 +0x124
    github.com/pulumi/pulumi-random/provider/v4.Provider.func1({0x14000bbf440?, 0x14000783e00?}, 0x140008bc0c0?, 0x7?)
        /Users/anton/code/pulumi-random/provider/resources.go:69 +0x2c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/pf/tfbridge.(*provider).CheckWithContext(0x14000a3a388, {0x102799b98?, 0x140007822a0?}, {0x14000720000, 0x5c}, 0x140007822d0, 0x14000782300, 0x0?, {0x140005000c0, 0x20, ...}, ...)
        /Users/anton/code/pulumi-terraform-bridge/pkg/pf/tfbridge/provider_check.go:60 +0x518
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/pf/internal/plugin.(*providerServer).Check(0x14000a2bb00, {0x102799b98, 0x140007822a0}, 0x140007a0000)
        /Users/anton/code/pulumi-terraform-bridge/pkg/pf/internal/plugin/provider_server.go:366 +0x20c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/providerserver.(*PanicRecoveringProviderServer).Check(0x1035943d0?, {0x102799b98?, 0x140007822a0?}, 0x1024133a0?)
        /Users/anton/code/pulumi-terraform-bridge/pkg/providerserver/panic_recovering_provider.go:309 +0x6c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Check_Handler.func1({0x102799b98?, 0x140007822a0?}, {0x1026a33a0?, 0x140007a0000?})
        /Users/anton/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:855 +0xd0
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x102799b98, 0x140007821b0}, {0x1026a33a0, 0x140007a0000}, 0x1400017c0a0, 0x14000c0a0f0)
        /Users/anton/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x2d4
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Check_Handler({0x10272b000, 0x14000ba6780}, {0x102799b98, 0x140007821b0}, 0x14000284000, 0x14000a267c0)
        /Users/anton/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:857 +0x148
    google.golang.org/grpc.(*Server).processUnaryRPC(0x14000b9c000, {0x102799b98, 0x14000782120}, {0x1027aabe0, 0x140007b8820}, 0x1400078c000, 0x14000a2bb90, 0x1035c0960, 0x0)
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1394 +0xb64
    google.golang.org/grpc.(*Server).handleStream(0x14000b9c000, {0x1027aabe0, 0x140007b8820}, 0x1400078c000)
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1805 +0xb20
    google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1029 +0x84
    created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 51
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1040 +0x13c

  random:index:RandomString (string):
    error: Bridged provider panic (provider=random v=4.0.0-alpha.0+dev resourceURN=urn:pulumi:anton-test-logging::simple-random::random:index/randomString:RandomString::string method=Check): FAILUREZ
    goroutine 27 [running]:
    runtime/debug.Stack()
        /nix/store/lb8wx7xsk7vryjxbbf9wlj820pdfvnbj-go-1.23.3/share/go/src/runtime/debug/stack.go:26 +0x64
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/providerserver.(*PanicRecoveringProviderServer).Check.func1()
        /Users/anton/code/pulumi-terraform-bridge/pkg/providerserver/panic_recovering_provider.go:303 +0x64
    panic({0x1023e15e0?, 0x102779e50?})
        /nix/store/lb8wx7xsk7vryjxbbf9wlj820pdfvnbj-go-1.23.3/share/go/src/runtime/panic.go:785 +0x124
    github.com/pulumi/pulumi-random/provider/v4.Provider.func1({0x14000bbf440?, 0x14000783e00?}, 0x140008bc0c0?, 0x7?)
        /Users/anton/code/pulumi-random/provider/resources.go:69 +0x2c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/pf/tfbridge.(*provider).CheckWithContext(0x14000a3a388, {0x102799b98?, 0x140007822a0?}, {0x14000720000, 0x5c}, 0x140007822d0, 0x14000782300, 0x0?, {0x140005000c0, 0x20, ...}, ...)
        /Users/anton/code/pulumi-terraform-bridge/pkg/pf/tfbridge/provider_check.go:60 +0x518
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/pf/internal/plugin.(*providerServer).Check(0x14000a2bb00, {0x102799b98, 0x140007822a0}, 0x140007a0000)
        /Users/anton/code/pulumi-terraform-bridge/pkg/pf/internal/plugin/provider_server.go:366 +0x20c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/providerserver.(*PanicRecoveringProviderServer).Check(0x1035943d0?, {0x102799b98?, 0x140007822a0?}, 0x1024133a0?)
        /Users/anton/code/pulumi-terraform-bridge/pkg/providerserver/panic_recovering_provider.go:309 +0x6c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Check_Handler.func1({0x102799b98?, 0x140007822a0?}, {0x1026a33a0?, 0x140007a0000?})
        /Users/anton/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:855 +0xd0
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x102799b98, 0x140007821b0}, {0x1026a33a0, 0x140007a0000}, 0x1400017c0a0, 0x14000c0a0f0)
        /Users/anton/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x2d4
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Check_Handler({0x10272b000, 0x14000ba6780}, {0x102799b98, 0x140007821b0}, 0x14000284000, 0x14000a267c0)
        /Users/anton/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:857 +0x148
    google.golang.org/grpc.(*Server).processUnaryRPC(0x14000b9c000, {0x102799b98, 0x14000782120}, {0x1027aabe0, 0x140007b8820}, 0x1400078c000, 0x14000a2bb90, 0x1035c0960, 0x0)
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1394 +0xb64
    google.golang.org/grpc.(*Server).handleStream(0x14000b9c000, {0x1027aabe0, 0x140007b8820}, 0x1400078c000)
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1805 +0xb20
    google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1029 +0x84
    created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 51
        /Users/anton/go/pkg/mod/google.golang.org/[email protected]/server.go:1040 +0x13c
    error: error reading from server: EOF

codecov · 2025-01-27T21:19:15Z

Codecov Report

Attention: Patch coverage is 97.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 67.73%. Comparing base (140a0fb) to head (56bae6e).
Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/logging/logging.go	52.63%	8 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2871      +/-   ##
==========================================
+ Coverage   67.50%   67.73%   +0.22%     
==========================================
  Files         327      328       +1     
  Lines       41743    42088     +345     
==========================================
+ Hits        28180    28508     +328     
- Misses      11977    11992      +15     
- Partials     1586     1588       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

t0yv0 · 2025-01-28T02:44:31Z

panic_recovering_provider.go has actually 100% coverage. I don't know what to make of the codecov report.

VenelinMartinov · 2025-01-28T11:44:46Z

The codecov report is partial, since only some of the tests have run due to the CI failure

VenelinMartinov

LGTM. Do we need to add the interceptor to the muxer in case it panics?

t0yv0 · 2025-01-28T15:49:20Z

That's a good idea. In case we are building a muxed provider it's better to install muxer once at the outer layer instead of installing it in the inner layers.

I went on a few tangents here trying to reconcile the problem of a missing logging sink but unfortunately it doesn't seem like my experiments are converging. Let me try to think it over and maybe relax the code's insistence on a non-nil logging sink..

t0yv0 · 2025-01-28T16:24:26Z

pkg/pf/tests/internal/cross-tests/util.go

 	crosstestsimpl "github.com/pulumi/pulumi-terraform-bridge/v3/pkg/internal/tests/cross-tests/impl"
 )

 type T = crosstestsimpl.T

-type testLogSink struct{ t T }


Moved this code to make it more available.

t0yv0 · 2025-01-28T16:25:26Z

Installed in muxer too, PTAL.

t0yv0 added 14 commits January 27, 2025 13:56

Initial sketch of a panic recovering provider

40571b6

Shorter v log

7731775

Cover more methods

824735c

Fix URN inference

7386cd6

Fixup stack traces

fe148cd

Cover StreamInvoke

a6c64d0

Test URN propagation for resource handling methods

ff22770

Check invoke token propagation

88e817d

Cover providerURN

271d2a9

Fix Configure coverage

02aca87

Cover Construct URN recovery

a4615e7

Lint

b6a6b19

Get cov to 100%

c98d696

Install the panic recovery wrapper

07c05d1

t0yv0 requested a review from VenelinMartinov January 27, 2025 20:11

t0yv0 added 3 commits January 27, 2025 15:49

Ensure non-nil logging sink

4edec09

Another instance of missing log sink

60c191b

Propagate testing sink into one more test context

e34ccff

VenelinMartinov approved these changes Jan 28, 2025

View reviewed changes

t0yv0 added 5 commits January 28, 2025 10:57

Do not insist on logging being set up

48b4604

Account for double-wrapping

b6ae0ea

Lint

eb5fde7

Install in muxer

39cc58c

Lint

5c51da6

t0yv0 commented Jan 28, 2025

View reviewed changes

Fix boolean blindness

56bae6e

t0yv0 enabled auto-merge (squash) January 28, 2025 16:42

t0yv0 merged commit a3a15d2 into master Jan 28, 2025
71 checks passed

t0yv0 deleted the t0yv0/annotate-panics branch January 28, 2025 16:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Annotate panics with provider metadata and origin #2871

Annotate panics with provider metadata and origin #2871

t0yv0 commented Jan 27, 2025 •

edited

Loading

codecov bot commented Jan 27, 2025 •

edited

Loading

t0yv0 commented Jan 28, 2025

VenelinMartinov commented Jan 28, 2025

VenelinMartinov left a comment

t0yv0 commented Jan 28, 2025

t0yv0 Jan 28, 2025

t0yv0 commented Jan 28, 2025

Annotate panics with provider metadata and origin #2871

Annotate panics with provider metadata and origin #2871

Conversation

t0yv0 commented Jan 27, 2025 • edited Loading

codecov bot commented Jan 27, 2025 • edited Loading

Codecov Report

t0yv0 commented Jan 28, 2025

VenelinMartinov commented Jan 28, 2025

VenelinMartinov left a comment

Choose a reason for hiding this comment

t0yv0 commented Jan 28, 2025

t0yv0 Jan 28, 2025

Choose a reason for hiding this comment

t0yv0 commented Jan 28, 2025

t0yv0 commented Jan 27, 2025 •

edited

Loading

codecov bot commented Jan 27, 2025 •

edited

Loading