Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory usage by using a streaming JSON parser and encoder #61

Open
dunglas opened this issue Sep 19, 2020 · 1 comment
Open
Labels
enhancement New feature or request hacktoberfest

Comments

@dunglas
Copy link
Owner

dunglas commented Sep 19, 2020

Currently, we have to wait for the response from the upstream server to be entirely sent, the to load it entirely in memory to analyze and modify the received JSON document.
Analyzing and modifying the JSON stream on the go could improve the memory usage (but probably not the overall performance because we need to compute and to send the Content-Length HTTP header before the JSON body anyway).

This isn't an easy pick because gjson doesn't support JSON streams for now (tidwall/gjson#25, tidwall/gjson#78). We'll have to use json.Decoder directly or to create a new library for this.

@dunglas dunglas added enhancement New feature or request hacktoberfest labels Sep 19, 2020
@mitar
Copy link

mitar commented Aug 2, 2024

Maybe https://github.com/minio/simdjson-go or https://github.com/ohler55/ojg could be used instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request hacktoberfest
Projects
None yet
Development

No branches or pull requests

2 participants