Overcome python debug adapter's string truncation limit

[dabeschte]

Hey!

Thanks for the great extension! I love how simple it is to add your own custom plots!

I am trying to add more plots for numpy/pytorch arrays, but I quickly ran into the issue of the maximum string length in the debug adapter according to this post.

From some quick tests, I was able to narrow down the maximum supported string length and it seems to be around 64k characters which is not much when you work with large arrays.

Fortunately, it seems like they were able to overcome this issue in this extension. Without really understanding how your and their extensions work, I was hoping that you would be able to find out why they can use larger arrays.
I guess they might be using different data types for communication which are not truncated. But an even better solution would be if they figured out a way to disable the truncation for this extension all together.

[hediet]

Thanks for pointing out https://github.com/elazarcoh/simply-view-image-for-python-debugging!
Looks like a nice alternative to this extension, but focused just on Python (unfortunately I don't know any Python).
I think the adventage of this extension is that it has more visualizations available and a webview UI for watching expressions.

@elazarcoh are you up to collaborate? ;)

Unfortunately I was not able to find out at a first glimpse how they overcame the truncation limit.

Actually, it looks like they are saving the image on disk and then just showing that image.

[elazarcoh]

@elazarcoh are you up to collaborate? ;)

Sure, what do you have in mind?

Actually, it looks like they are saving the image on disk and then just showing that image.

That's right...
But I'm working on an improved viewer these days, which I'm still debating how to serialize from python, so I might come to the same problem. What is the characters limit?

[hediet]

What is the characters limit?

Seems to be 64k characters.

Sure, what do you have in mind?

Up to become a co-maintainer of the python visualization capability of this extension? We could schedule a virtual meeting to discuss how we could do that. I'm up to help you with/walk you through all the TypeScript part, it's just that I don't know much about python.
The debug visualizer already has an image viewer, it's just a matter of how to extract the data.

[dabeschte]

Would be awesome if the best of both projects could be combined.

Saving a temporary json file to disk sounds like a simple solution to overcome the limitations of the python debug adapter.
Do you think that this would even work for remote debugging / using vscode over ssh? Even if it doesn't, it would still improve the situation for the majority of vscode users.

[hediet]

This might work for remote debugging, but it feels like hacky solution. Maybe we can use this as fallback, if the output would be too long.

[elazarcoh]

Up to become a co-maintainer of the python visualization capability of this extension? We could schedule a virtual meeting to discuss how we could do that. I'm up to help you with/walk you through all the TypeScript part, it's just that I don't know much about python. The debug visualizer already has an image viewer, it's just a matter of how to extract the data.

We can discuss, sure. Currently I'm busy with the other thing I mentioned, but feel free to contact me via mail.

What is the characters limit?

Seems to be 64k characters.

Oh that's a shame. I intended to use base64 for serialization. My first alternative would be to use sockets for communication, but I'm not sure how feasible it is. I'm going to need this anyway, because I'm working on a webview, so I'll keep you posted.

[elazarcoh]

Alright. I managed to put something that seems to be working for the JSON case. The numpy case is still a WIP (mostly due to complexity of parsing arrays: datatype, byte-order, alignment, etc.).
https://gist.github.com/elazarcoh/adbbf8761817c8319b7e496d3fd6f888

you can see in main.py that I'm sending 6MB worth of JSON text. It received in the message in its entirety in the extension.

I'm currently working on integrating it in my extension, but feel free to start playing with it and ask questions.

PS usage example in the extension:

const socketServer = Container.get(SocketServer);
socketServer.start();

const requestId = 1; // identify my request, will parse in the response
const varName = "foo";
session.customRequest("evaluate", {
        `open_send_and_close(${socketServer.portNumber}, ${requestId}, ${varName}, ${0x02})`,  // 0x02 is Json
        frameId,
        context,
    } as DebugProtocol.EvaluateArguments)

[elazarcoh]

@hediet I finished the implementation in my extension. Do you want me to add it to your extension? Can you point me to where this can go?
To implement it I need 2 things:
On the extension side, I need to implement a socket server. I used nodejs Net module for it.
On the python side, I need to write a couple of functions that send data over the socket to localhost (no additional dependencies, only python builtins).
If you want me to start, please point me to where you want it to go. I will start with something initial and we can continue discussing it in PR.

[hediet]

The problem with a server however is, that it doesn't run while being debugged. How did you solve that problem?

[elazarcoh]

I'm not what you mean by that.
The server is starting with the extension without any problems.