Very slow decoding

[novitae]

Hello, I am using blackboxprotobuf to decode a YouTube response, and it's very slow. The response file weighs 1.3mb, and it took me 19 minutes to decode it, as shown below.

Even with the typedef it's extremely slow. Here's the file:
slow_proto.txt

Is there something I can do to improve this ?

[rwinkelmaier-ncc]

Hi! Thanks for the report and for providing the example file. I was able to replicate and run the profiler.

It looks like most of the time is being spent within copy.deepcopy calls. Not sure if there's something special about this protobuf that triggered it, or just the large size, but I'll look into improving the time. Removing these three deepcopy calls (https://github.com/search?q=repo%3Anccgroup%2Fblackboxprotobuf%20copy.deepcopy&type=code) should drastically speed up the parsing, but there might be some edge cases that don't work correctly. I'll have to think through those cases and figure out the best way to handle without as many copies. Might be a good chance to rewrite the clunky alt_typedef system as well.

[rwinkelmaier-ncc]

Hi! I just pushed a fix to the repo which should fix this and improve performance generally. I think it could have been fixed with a few strategic "copy.copy" calls, but I wanted to be sure I wasn't missing any edge cases so ended up rewriting quite a bit. There's definitely more room for performance improvements as well.

I haven't pushed to pypi yet, but will next week after I double check a few more things.

[novitae]

Nice ! I think you might want to give a try to dict.copy or list.copy, that is builtin those classes. Other values (str, int, float, bool and None) are copied by default, so they don't need it.

[rwinkelmaier-ncc]

Ok. The changes should be pushed to pypi now as version 1.4.0. Let me know if you have any issues.

I tried a few different tweaks to improve performance further, such as doing copy-on-write and protocols instead of full classes, but didn't see any performance gains for this payload so left it as-is for now.