Feature proposal: new subcommand for computing obfuscation map

[fbbdev]

Rationale

At present, garble provides a reverse subcommand that enables consumers to map obfuscated identifiers back to their plain counterparts as needed.

There are cases where this is not enough for devtools that might want to offer support for garbled builds. This is especially true for tools that provide interoperation between go and other languages (more on a concrete use case below).

What such tools might need is actually the ability to map plain identifiers to their obfuscated counterparts – i.e. perform obfuscation – matching exactly the behaviour of the installed garble binary. This is not an easy problem especially considering that garble has some sophisticated logic for deciding which objects get obfuscated, and which do not.

Proposal

I propose therefore the addition of a new subcommand, tentatively named garble map, with the following usage:

garble [garble flags] map [build flags] packages...

The command should output a JSON object with one key per obfuscated package – including all packages given on the command line and their transitive dependencies, kinda like go list -deps – with the following shape (but other approaches will do as long as they provide equivalent information):

{
    "PLAIN_PACKAGE_PATH_1": {
        "path": "OBFUSCATED_PACKAGE_PATH_1",
        "objects": {
            "TYPES_OBJECT_PATH_1": "OBFUSCATED_NAME_1",
            // ...
            "TYPES_OBJECT_PATH_n": "OBFUSCATED_NAME_n"
        }
    },
    // ...
    "PLAIN_PACKAGE_PATH_n": { /* ... */ },
}

Here, "TYPES_OBJECT_PATH_n": "OBFUSCATED_NAME_n" represents one entry for each obfuscated, externally visible go/types.Object in the package. Keys are object paths as computed by the facilities at golang.org/x/tools/go/types/objectpath. Objects that are not obfuscated, or don't have a path (i.e. they are not part of the package's public API) should not appear in the map.

Security considerations

Accidentally leaking the output of garble map would defeat obfuscation for all exported APIs for all public functions and public/private type names, so it carries a higher risk than garble reverse. Consumers should take extra care when integrating garble map with their development workflows.

That risk can be mitigated by feeding the output of garble map directly into other tools through shell pipelines, without writing it to the filesystem.

Concrete use case

Recently I have been working as an independent contributor on a full rewrite of the binding generator for Wails v3.

Wails is a project that enables developers to write desktop apps using Go and web technologies. Like many similar projects, it aims to support seamless interoperation between the Go backend and the JS frontend.

To this end, they provide an RPC system based upon encoding/json and a binding generator that parses Go code and outputs JS glue code. Specifically, developers register a list of named types as RPC "Services"; a hash is then computed for each method based on package path, type name and method name, and used to identify methods in remote call requests.

This obviously breaks down for garbled builds. Project owner @leaanthony sought advice some time ago at #826 and @lu4p suggested using the reflect.ValueOf trick; that however is not enough in this case, because it is only going to stop obfuscation of fields, not of the type name or package path.

There are workarounds, but they all involve increased boilerplate and upkeep efforts for application developers, and/or degraded obfuscation.

Not to speak of the fact that the reflect.ValueOf trick cannot cross package boundaries, and therefore hurts composability a lot. People who may need or want to use complex type hierarchies in their RPC API are going to experience severe limitations.

From Wails' perspective, the best option is clearly to provide transparent support for garbled builds, which requires some cooperation on garble's part. I devised and implemented the garble map approach described above as a minimal means for them to achieve this end.

It's true that the generated JS glue code could be used (although not without some effort) to partly reverse obfuscation, but Wails users would be made well aware of that and advised to obfuscate JS code as well (which they probably already do).

[fbbdev]

I opened a PR with a draft implementation for this proposal at #850
Technical details are discussed over there.

[fbbdev]

After thinking further about it, I believe the impact of leaking garble map output would be somewhat smaller than I thought initially. The path encoding by x/tools/go/types/objectpath uses names for top-level objects only (i.e. types and functions) hence it shouldn't be usable to deobfuscate field/method names.

[leaanthony]

Any news on this? It would be a huge win.

[mvdan]

Thank you for opening this issue and raising a PR. I agree this would be a useful feature and indeed I planned for it some time ago as well. I will look into this soon, once @lu4p's #889 is finished, given that it should resolve a number of existing bugs and it is touching very related code. In fact I expect the "map" subcommand will be able to reuse much of the same code.

[leaanthony]

Amazing @mvdan ! Thanks so much!

[fbbdev]

Thank you @mvdan for taking the time to answer this.
I have just rebased and tested the PR against the current tip.

In fact I expect the "map" subcommand will be able to reuse much of the same code.

I'd like to point out that in fact the PR is not introducing any new code at all, it just moves around existing code.

Part of the transformGoFile method is being factored into a new obfuscateObjectName method to make it available to the map subcommand. The implementation of the subcommand itself is pretty much a clone of reverse, where the syntax visitor computes and stores additional information about each object, which is then printed to stdout.

[mvdan]

Thanks for the update. Perhaps hold off on spending too much time on the PR, because I'm not yet convinced about how to design this feature. The way you designed the CLI interface and the JSON format with https://pkg.go.dev/golang.org/x/tools/go/types/objectpath#Path is perfectly fine, but I need to think about it a bit more to convince myself whether it's the right approach or if there's a better way.

For example, my initial instinct is that this could live as part of garble reverse, because the two commands would serve very similar purposes. They both need to work out pretty much the same mapping of obfuscated names, anyway.

[fbbdev]

Perhaps hold off on spending too much time on the PR, because I'm not yet convinced about how to design this feature.

Sure, the PR was always meant as a PoC and I do not plan on ever pushing substantial updates.

I just solved the misleadingly large conflicts, as they made the changeset hard to assess. The diffing engine does not deal very well with code factorization.

For example, my initial instinct is that this could live as part of garble reverse, because the two commands would serve very similar purposes.

On a more philosophical note, although they look similar, the purpose of "map"-like functionality is not to reverse obfuscation but to perform it, matching exactly the way the build command does it.

As for the actual interface, personally I have no preference as long as the necessary functionality is there. Should it be an interactive command like reverse instead of printing a full mapping? That would be fine too. Perhaps an importable package exposing a Go API? Fine as well.

They both need to work out pretty much the same mapping of obfuscated names, anyway.

That's right. The main difference is that "map"-like functionality requires knowing in addition which objects (packages/types/methods) are actually obfuscated at build time, and which ones aren't.

Hence why I factored out all related logic from obfuscateGoFile, to ensure alignment between build and map.

garble reverse on the other hand deals just with names, it does not need to reason about "objects".

[fbbdev]

The main difference is that "map"-like functionality requires knowing in addition which objects (packages/types/methods) are actually obfuscated at build time, and which ones aren't.

In light of the work that's being done in #889 I feel like I need to be a bit more specific here.

What wails needs is the ability to predict at compile time, for any given package, public type, method or field, which strings will be reported by Go's reflection methods.

Because wails allows developers to generate bindings for arbitrarily complex and possibly cross-package type hierarchies, requiring them to manually mark all those types as reflected is a hassle and does not compose well (what if they want to depend on an external package that is oblivious to garble?).

Instead, wails can leverage its own code generator to deal transparently with obfuscated names, as long as the information discussed above is made available

[mvdan]

What wails needs is the ability to predict at compile time, for any given package, public type, method or field, which strings will be reported by Go's reflection methods.

Because wails allows developers to generate bindings for arbitrarily complex and possibly cross-package type hierarchies, requiring them to manually mark all those types as reflected is a hassle and does not compose well (what if they want to depend on an external package that is oblivious to garble?).

If that is all you need, in theory with #889 merged, you can simply rely on Go's reflect package to report the original strings for any methods or fields correctly. That was the entire point of the refactor by @lu4p, to allow reflection to work out of the box without authors or users needing to annotate the types they use reflection with. If that is not already the case with garble at master, then please share specific details or a reproducer, because it needs to be looked into.

As you describe in your original post, the fact that the original strings are reachable via reflection can be considered as less powerful obfuscation, but for Go programs to work in general, reflection cannot provide obfuscated names when using garble. I don't think garble as a general tool can work in any other way.

On a more philosophical note, although they look similar, the purpose of "map"-like functionality is not to reverse obfuscation but to perform it, matching exactly the way the build command does it.

To be clear, you absolutely cannot match the obfuscation of garble build with the mechanism that you describe. Obfuscating names is just a tiny part of it; garble does much more to remove other information, and is also able to obfuscate literals and function bodies. Perhaps you're only interested in the names here, but I want to make it clear that we should not provide an alternative command that misleads users into thinking they can do the obfuscation themselves.