chore: add extra indexes to calls and unread events tables [WPB-11808] #3084

vitorhugods · 2024-10-30T15:00:53Z

WPB-11808 [Android] app stalls when fetching a page of conversations from the local DB

PR Submission Checklist for internal contributors

The PR Title
- conforms to the style of semantic commits messages¹ supported in Wire's Github Workflow²
- contains a reference JIRA issue number like SQPIT-764
- answers the question: If merged, this PR will: ... ³
The PR Description
- is free of optional paragraphs and you have filled the relevant parts to the best of your ability

What's new in this PR?

Issues

We are executing queries on UnreadEvents and Calls that result in full table scans.

Causes

Missing indexes.

Solutions

Add indexes :)

Dependencies

Needs releases with:

fix: slow conversation list queries [WPB-11808] 🍒 #3083

I've created 89.sqm, as the PR #3083 creates 88.sqm

Testing

N/A

PR Post Merge Checklist for internal contributors

If any soft of configuration variable was introduced by this PR, it has been added to the relevant documents and the CI jobs have been updated.

References

https://sparkbox.com/foundry/semantic_commit_messages
https://github.com/wireapp/.github#usage
E.g. feat(conversation-list): Sort conversations by most emojis in the title #SQPIT-764.

sonarqubecloud · 2024-10-30T15:01:25Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-10-30T15:10:13Z

Test Results

2 820 tests - 435 2 812 ✅ - 337 4m 31s ⏱️ +22s
2 suites - 554 8 💤 - 98
2 files - 554 0 ❌ ± 0

Results for commit 63e704e. ± Comparison against base commit 1f3c946.

This pull request removes 3255 and adds 2820 tests. Note that renamed tests count towards both.

PocIntegrationTest ‑ givenApiWhenGettingACMEDirectoriesThenReturnAsExpectedBasedOnNetworkState
PocIntegrationTest ‑ givenEmailAndPasswordWhenLoggingInThenRegisterClientAndLogout
PocIntegrationTest ‑ givenUserWhenHandlingTextMessagesThenProcessShouldSucceed
com.wire.kalium.HttpClientConnectionSpecsTest ‑ givenOkHttpSingleton_whenBuildingClearTextTrafficOkhttpClient_thenEnsureConnectionSpecClearText[jvm]
com.wire.kalium.HttpClientConnectionSpecsTest ‑ givenTheHttpClientIsCreated_ThenEnsureOnlySupportedSpecsArePresent[jvm]
com.wire.kalium.api.base.authenticated.notification.AccessUpdateTest ‑ givenPayloadWithAccessRoleAndDeprecatedAccessRoleField_whenDecoding_thenDeprecatedFieldIsPreferred[jvm]
com.wire.kalium.api.base.authenticated.notification.AccessUpdateTest ‑ givenPayloadWithDeprecatedAccessRoleField_whenDecoding_thenSuccess[jvm]
com.wire.kalium.api.base.authenticated.notification.AccessUpdateTest ‑ givenPayload_whenDecoding_thenSuccess[jvm]
com.wire.kalium.api.common.ACMEApiTest ‑ givenCRLWithHttpsProtocol_whenGettingClientDomainCRL_thenItShouldNotBeChanged[jvm]
com.wire.kalium.api.common.ACMEApiTest ‑ givenNoLocationInHeader_whenCallingSendAcmeRequestApi_theResponseShouldBeConfigureCorrectly[jvm]
…

com.wire.kalium.logic.cache.SelfConversationIdProviderTest ‑ givenFailure_thenErrorIsPropagated
com.wire.kalium.logic.cache.SelfConversationIdProviderTest ‑ givenMLSClientHasBeenRegistered_thenMLSAndProteusSelfConversationAreReturned
com.wire.kalium.logic.cache.SelfConversationIdProviderTest ‑ givenMLSClientHasNotBeenRegistered_thenProteusSelfConversationIsReturned
com.wire.kalium.logic.client.E2EIClientProviderTest ‑ givenIsNewClientTrue_whenGettingE2EIClient_newAcmeEnrollmentCalled
com.wire.kalium.logic.client.E2EIClientProviderTest ‑ givenMLSClientWithE2EI_whenGettingE2EIClient_callsNewActivationEnrollment
com.wire.kalium.logic.client.E2EIClientProviderTest ‑ givenMLSClientWithoutE2EI_whenGettingE2EIClient_callsNewRotateEnrollment
com.wire.kalium.logic.client.E2EIClientProviderTest ‑ givenSelfUserNotFound_whenGettingE2EIClient_ReturnsError
com.wire.kalium.logic.configuration.CustomServerConfigRepositoryTest ‑ givenStoredConfigLinksAndVersionInfoData_whenAddingNewOne_thenCommonApiShouldBeCalculatedAndConfigShouldBeStored
com.wire.kalium.logic.configuration.CustomServerConfigRepositoryTest ‑ givenStoredConfig_whenAddingNewOne_thenNewOneShouldBeInsertedAndReturned
com.wire.kalium.logic.configuration.CustomServerConfigRepositoryTest ‑ givenStoredConfig_whenAddingTheSameOneWithNewApiVersionParams_thenStoredOneShouldBeUpdatedAndReturned
…

This pull request removes 106 skipped tests and adds 8 skipped tests. Note that renamed tests count towards both.

PocIntegrationTest ‑ givenApiWhenGettingACMEDirectoriesThenReturnAsExpectedBasedOnNetworkState
com.wire.kalium.api.common.ACMEApiTest ‑ whenCallingGeTrustAnchorsApi_theResponseShouldBeConfigureCorrectly[jvm]
com.wire.kalium.api.common.ACMEApiTest ‑ whenCallingSendChallengeRequestApi_theResponseShouldBeConfigureCorrectly[jvm]
com.wire.kalium.api.v0.user.register.RegisterApiV0Test ‑ givenAValidEmail_whenActivationEmailWIthCode_theRequestShouldBeConfiguredCorrectly[jvm]
com.wire.kalium.api.v0.user.register.RegisterApiV0Test ‑ givenAValidEmail_whenRegisteringAccountWithEMail_theRequestShouldBeConfiguredCorrectly[jvm]
com.wire.kalium.api.v0.user.register.RegisterApiV0Test ‑ givenAValidEmail_whenSendingActivationEmail_theRequestShouldBeConfiguredCorrectly[jvm]
com.wire.kalium.api.v0.user.register.RegisterApiV0Test ‑ givenActivationCodeFail_thenErrorIsPropagated[jvm]
com.wire.kalium.api.v0.user.register.RegisterApiV0Test ‑ givenRegistrationFail_whenRegisteringAccountWithEMMail_thenErrorIsPropagated[jvm]
com.wire.kalium.api.v0.user.register.RegisterApiV0Test ‑ givenSendActivationCodeFail_thenErrorIsPropagated[jvm]
com.wire.kalium.cryptography.CryptoUtilsTest ‑ givenDummyText_whenEncryptedAndDecryptedWithAES256_returnsOriginalText[js, browser]
…

com.wire.kalium.logic.feature.backup.RestoreBackupUseCaseTest ‑ givenACorrectlyEncryptedBackup_whenRestoringWithWrongPassword_thenTheRightErrorIsThrown
com.wire.kalium.logic.feature.backup.RestoreBackupUseCaseTest ‑ givenAnEncryptedBackupFileFromDifferentUserID_whenRestoring_thenTheRightErrorIsThrown
com.wire.kalium.logic.feature.call.CallManagerTest ‑ givenCallManager_whenCallingMessageIsReceived_then_wcall_recv_msg_IsCalled
com.wire.kalium.logic.feature.conversation.ObserveConversationListDetailsUseCaseTest ‑ null
com.wire.kalium.logic.feature.session.DeleteSessionUseCaseTest ‑ givenSuccess_WhenDeletingSessionLocally_thenSuccessAndResourcesAreFreed
com.wire.kalium.logic.sync.incremental.EventProcessingHistoryTest ‑ measureContainsTimeOverLargeAmountOfEvents
com.wire.kalium.persistence.dao.message.MessageDAOBenchmarkTest ‑ null
com.wire.kalium.persistence.kmmSettings.EncryptedSettingsBuilderTest ‑ givenShouldEncryptDataIsTrue_whenEncryptingData_thenShouldEncryptWithoutFailing

♻️ This comment has been updated with latest results.

github-actions · 2024-10-30T15:16:32Z

Bencher Report

Branch	chore/add-db-indexes
Testbed	ubuntu-latest

⚠️ WARNING: The following Measure does not have a Threshold. Without a Threshold, no Alerts will ever be generated!
Latency
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the --ci-only-thresholds CLI flag.

Click to view all benchmark results

Benchmark	Latency	nanoseconds (ns)
com.wire.kalium.benchmarks.logic.CoreLogicBenchmark.createObjectInFiles	📈 view plot ⚠️ NO THRESHOLD	657,687.10
com.wire.kalium.benchmarks.logic.CoreLogicBenchmark.createObjectInMemory	📈 view plot ⚠️ NO THRESHOLD	341,266,742.43
com.wire.kalium.benchmarks.persistence.MessagesNoPragmaTuneBenchmark.messageInsertionBenchmark	📈 view plot ⚠️ NO THRESHOLD	928,990,035.67
com.wire.kalium.benchmarks.persistence.MessagesNoPragmaTuneBenchmark.queryMessagesBenchmark	📈 view plot ⚠️ NO THRESHOLD	21,653,563.48

🐰 View full continuous benchmarking report in Bencher

datadog-wireapp · 2024-10-30T15:21:58Z

Datadog Report

Branch report: chore/add-db-indexes
Commit report: 34169b6
Test service: kalium-jvm

✅ 0 Failed, 1 Passed, 0 Skipped, 0s Total Time

MohamadJaara

suggestion about removing a redundant index

MohamadJaara · 2024-10-31T08:19:38Z

persistence/src/commonMain/db_user/com/wire/kalium/persistence/Calls.sq

@@ -15,6 +15,7 @@ CREATE TABLE Call (
 CREATE INDEX call_date_index ON Call(created_at);
 CREATE INDEX call_conversation_index ON Call(conversation_id);


the conversation ID index is redundant since the PK starts with conversation_id

saleniuk · 2024-11-07T16:24:27Z

I've created 89.sqm, as the PR #3083 creates 88.sqm

That PR #3083 actually adds 89.sqm, not 88.sqm, so here it should be 90.sqm

Could we also add these indexes to RC?

saleniuk · 2024-11-14T10:28:44Z

I already added these indexes in the PR that provides fixes to make paginated queries faster: #3102 so I'm closing this one

chore: add extra indexes to calls and unread events tables

63e704e

vitorhugods requested review from m-zagorski, MohamadJaara, Garzas and saleniuk October 30, 2024 15:00

github-actions bot assigned vitorhugods Oct 30, 2024

github-actions bot added type: chore 🧹 👕 size: XS labels Oct 30, 2024

vitorhugods requested review from ohassine and damian-kaczmarek October 30, 2024 15:01

yamilmedina approved these changes Oct 30, 2024

View reviewed changes

MohamadJaara reviewed Oct 31, 2024

View reviewed changes

damian-kaczmarek approved these changes Nov 6, 2024

View reviewed changes

echoes-hq bot added the echoes: unplanned Any work item that isn’t part of the product or technical roadmap. label Nov 7, 2024

saleniuk closed this Nov 14, 2024

saleniuk deleted the chore/add-db-indexes branch November 14, 2024 10:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add extra indexes to calls and unread events tables [WPB-11808] #3084

chore: add extra indexes to calls and unread events tables [WPB-11808] #3084

vitorhugods commented Oct 30, 2024 •

edited by github-actions bot

Loading

sonarqubecloud bot commented Oct 30, 2024

github-actions bot commented Oct 30, 2024 •

edited

Loading

github-actions bot commented Oct 30, 2024

datadog-wireapp bot commented Oct 30, 2024

MohamadJaara left a comment

MohamadJaara Oct 31, 2024

saleniuk commented Nov 7, 2024

saleniuk commented Nov 14, 2024

		@@ -15,6 +15,7 @@ CREATE TABLE Call (
		CREATE INDEX call_date_index ON Call(created_at);
		CREATE INDEX call_conversation_index ON Call(conversation_id);

chore: add extra indexes to calls and unread events tables [WPB-11808] #3084

chore: add extra indexes to calls and unread events tables [WPB-11808] #3084

Conversation

vitorhugods commented Oct 30, 2024 • edited by github-actions bot Loading

PR Submission Checklist for internal contributors

What's new in this PR?

Issues

Causes

Solutions

Dependencies

Testing

PR Post Merge Checklist for internal contributors

References

sonarqubecloud bot commented Oct 30, 2024

Quality Gate passed

github-actions bot commented Oct 30, 2024 • edited Loading

Test Results

github-actions bot commented Oct 30, 2024

Bencher Report

datadog-wireapp bot commented Oct 30, 2024

Datadog Report

MohamadJaara left a comment

Choose a reason for hiding this comment

MohamadJaara Oct 31, 2024

Choose a reason for hiding this comment

saleniuk commented Nov 7, 2024

saleniuk commented Nov 14, 2024

vitorhugods commented Oct 30, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Oct 30, 2024 •

edited

Loading