Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drain insert HTTP response #203

Merged
merged 2 commits into from
Oct 11, 2023
Merged

Conversation

varrocs
Copy link
Contributor

@varrocs varrocs commented Oct 11, 2023

In case of an insert call drain the HTTP response instead of destroying it.

Summary

Calling destroy eventually closed the socket even if keep-alive was set. This was a performance limitation.
#202

Checklist

Delete items not relevant to your PR:

Note: I don't know how to test this

  • Unit and integration tests covering the common scenarios were added

Note: I don't know if it applies to my change

  • A human-readable description of the changes was provided to include in CHANGELOG

@CLAassistant
Copy link

CLAassistant commented Oct 11, 2023

CLA assistant check
All committers have signed the CLA.

…ng it. Calling destroy eventually closed the socket even if keep-alive was set.
@varrocs varrocs force-pushed the drain_http_response branch from 8a3cc87 to 6e3efbd Compare October 11, 2023 09:48
async drainHttpResponse(stream: Stream.Readable): Promise<void> {
return new Promise((resolve, reject) => {
function dropdata() {
// We don't care about the data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add resolve() here? Also, minor nit: dropData

Copy link
Contributor Author

@varrocs varrocs Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add resolve() here?

No. This function will be called when data is available but we just want to throw away the data, we don't care.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit: dropData

Fixed

@slvrtrn
Copy link
Contributor

slvrtrn commented Oct 11, 2023

I don't know how to test this

If we want to test it, we need to spy on sockets somehow (with max connections set to 1); I am afraid there is no other way.

(please don't mind failing Cloud tests for now; for some reason, Actions cannot fetch the secret properly for the community PRs).

@varrocs
Copy link
Contributor Author

varrocs commented Oct 11, 2023

If we want to test it, we need to spy on sockets somehow

Well, let's say I don't insist on testing this.

@varrocs
Copy link
Contributor Author

varrocs commented Oct 11, 2023

If we want to test it, we need to spy on sockets somehow

So it's me asking if you accept the pull request without writing tests to test if the consecutive requests are going on the same socket

@slvrtrn
Copy link
Contributor

slvrtrn commented Oct 11, 2023

@varrocs, I think we can live without such a test for now cause nothing else is broken, and we have a specific test for consecutive inserts with one socket, IIRC (and they pass).

I'd only ask you to confirm that with this change, the TCP dump looks like the fixed version mentioned here: #202 (comment)

@varrocs
Copy link
Contributor Author

varrocs commented Oct 11, 2023

Yes you have such test (with a comment that saying that how strange it is that the socket mostly reopened even with keep alive)
And yes, the tcpdump looks good.

@slvrtrn slvrtrn merged commit a777eec into ClickHouse:main Oct 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants