Tests for compressed communication with the proxy

[aismail]

Why

Proxy offers support for compression, both with the clients connecting to it, and to the backends. This support is independent: you can have a client with compression disabled communicating with the proxy without compression, but the proxy will communicate in a compressed way with the backends.

What

TODO: write list of issues in here. This is a mini-project

Notes

Pay attention to the fact that package fragmentation in the MySQL protocol is completely different for compressed and non-compressed cases.

[m00dawg]

I've been testing our ProxySQL in our environment and it's been pretty glorious! But one issue I ran into was it looks like client compression is doing some weird things. Using Ruby Sequel, every 5th connection would result in "Lost connection to mysql during query" but only when compression was enabled. It was every 5th request like clockwork even if I tried changing various parameters around. Client compression was enabled by setting Mysql2::Client::COMPRESS. Disabling that, causes the lost connection errors to go away.

No errors I could see from proxysql (or sometimes even the app) so I don't have more to go on just yet, but wanted to toss this out there just in case it sparked any thoughts.

[renecannao]

Hi Tom,
Thanks for the feedback.

Do you have a reproducible test case?

Or can you send me (privately maybe) some tcpdump traffic collected with:

tcpdump -i any -s 0 -w 6033.tcp port 6033

A test case makes debugging easier than tcpdump.

Thanks

[m00dawg]

I had trouble isolating the test case since it was hitting an API test node but I can see what I can do to put one together yup. I'll need to bug our API devs to see if we can restrict it down to an easier to use test case (or I get to learn how to Ruby hehe). May take me a few so I'll see about a tcpdump in the interim but will try to work on a test case for you.

[renecannao]

Indeed, I am not sure if I will be faster in learning Ruby to try the test case, or reverse engineer the tcpdump with compressed traffic 😄
Please feel free to send a tcpdump , and if that isn't enough we try a test case.
Thanks.

[m00dawg]

Whelp I was able to at least grab the queries that Ruby is generating but so far when I run them via the mysql CLI I've only be able to get it to loose connection once. Realized I should probably get approval before busting out a tcpdump unless I can obfuscate the data so I'm waiting on that (and while doing so trying to see if I can run into using other means).

[renecannao]

It is unlikely you to be able to obfuscate the data, as any obfuscation will probably remove also any trace of what caused the problem (I suspect some encoding issue).
So yes, please get the approval.
When running it via CLI, did you get any specific error? For example "malformed format", or "command out of sync"?

[m00dawg]

No, it was the infamous "Lost connection to MySQL during query" error. You're likely onto something though - a couple of the tables use utf8mb4, although I don't think the rows have any mb4 data in them (I may need to verify). What is peculiar is I can cut/paste the queries into the CLI and they run fine but if I toss them into a SQL file and redirect that file to the cli it will fail. I also noticed I need to wrap this around a transaction as well.

That said, I was able to reproduce this directly at least (no need for Ruby) even if it requires piping a file to it rather than a straight cut and paste (not sure what could be different there).

[renecannao]

Closing all "test issues" now obsolete.
Thanks