Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ios): multitap consistency after new-lines #10728

Merged
merged 6 commits into from
Feb 19, 2024

Conversation

jahorton
Copy link
Contributor

@jahorton jahorton commented Feb 15, 2024

Fixes #10696.

Continues from #10584 and #10633.

Now that we're synchronizing app context with the internal web engine's context more proactively - which gives us more consistent backspace behavior - we need to ensure that this doesn't break preservation of context going forward, as new text is added.

Commit 1

Simplified:

  • Ensures nothing "goes wrong" when you type a lot of text, even when starting new paragraphs.
    • The lone exception: multitaps at the start of a new paragraph (and rarely, within the middle of a long one) may have unexpected side effects.

Internal details:

  • Fixes context synchronization for all cases except a multitap immediately following a textDocumentProxy context-window slide.

  • Ensures that all context manipulations for each tap of a multitap requires only a single call when updating host-app context.

    • Before, each tap of a multitap triggered two calls: one to restore the old context, then one to apply the new tap's key.

Commit 2

Simplified

  • Fixes the paragraph-initial multitap issues mentioned for "commit 1".
  • Prevents the engine from forgetting deadkeys too quickly when new paragraphs are started (which was an unexpected side effect of recent blocked-backspace fixes - fix(ios): long-held backspace handling #10633).

Internal details:

  • Ensures textDocumentProxy context window jumps that result as new text is added (without other changes) do not trigger a context reset within the internal web engine.
    • Leverages fix(web): corrects Android over-deletion of selected text, other context diffs involving selected text #10662 to accomplish this goal. I did not expect to find another use for the method so soon, but here we are.
    • It is imperative that a context window jump does not change the internal engine's context during an ongoing multitap.
    • Toward the former point: we were getting excess newlines for context-window-initial multitaps - one per tap of the ongoing multitap gesture - because the context window was being reset between each tap due to textDocumentProxy quirks.
  • This should also prevent context-resets during backspaces in which the textDocumentProxy context window returns to text that was once "in window" but had since slid out of it.

There is a chance that an extreme edge case might still have issues:

  • Suppose a keyboard rule that changes the last character before the newline preceding the caret. (Say, a\n + b => b\n.)
  • Now suppose that rule is triggered by a multitap key, thus cycling in and out during multitap rota alongside context rewind operations.
  • As the tap with the rule would trigger an edit (within the host app) past context-window boundaries, this would naturally trigger context-synchronization after the related tap should the textDocumentProxy perform a context-window jump as a result... and also for the tap after it, which would need to make counter-edits to revert such a rule's change, as such a tap would likely retrigger a textDocumentProxy update.

Upon request, I suppose I could create a highly-focused test keyboard for validating such a case. That said, it does seem pretty niche - I don't think we currently have a keyboard with such rules. (As a result, I haven't prioritized testing this scenario.) Part of me wonders if the best course of action would be to spin this concern off into a separate issue so that it doesn't hold this PR up from fixing a lot of cases that are far more commonly seen.

User Testing

TEST_LONG_DELETION: Using the iOS Keyman app on an iPhone 13 / iPhone 13 Pro, delete a lot of pre-existing text from within a third-party application. (Simulator is fine, but stick to iPhone 13 targets.)

  1. Ensure that Keyman is enabled for use as a system keyboard.
  2. Open a third party application and swap to the default iOS system keyboard.
  3. Visit this PR's page on the test device and copy the full text from the code box above. ("Testing testing...")
  4. Open the Notes app and create a new note.
  5. Paste the text copied from this PR's page as the content of the new note.
  6. Use the globe key keyboard-selection menu to activate Keyman as your system keyboard.
  7. Hold down BKSP until no further deletions occur.
  8. If any visible text remains that refuses to be deleted, FAIL this test.

This test can be done on Simulator by using the Messages app instead and selecting one of the sample contacts. The Notes app is typically not available for Simulator-based iOS devices. (Notes doesn't come with the risk of butt-texting, though.)

TEST_SHIFTED_CONTEXT_MULTITAP_IOS: Using the iOS Keyman app on an iPhone 13 / iPhone 13 Pro, verify that multitaps at the start of new paragraphs do not have unwanted side effects. (Simulator is fine, but stick to iPhone 13 targets.)

  1. Download this KMP: https://jahorton.github.io/diacritic_rota.kmp
  2. Open / install it for use within the Keyman app.
  3. Double-tap the 8 key.
    • Expected total result: a
  4. Triple-tap the '◌̀' key - a circumflex diacritic should appear over the a.
  5. Triple-tap the 9 key.
    • Expected total result:
  6. Tap the '◌̀' key. A diacritic should appear over the ê.
  7. Open the in-app Settings menu.
    • Note, for test maintenance: the point of this step and the next one is to trigger a context-reset.
  8. Close the in-app Settings menu.
  9. Double-tap the 8 key.
    • Expected final result: aềa
    • For clarity:
      • With a copy of the à's accent-mark on top the original ê.
        • It looks different in raw-text form here due to GitHub's font / rendering.
  10. Double-tap the ENTER key.
  11. Repeat steps 3-9. The result for the first and last lines should be identical, with no extra lines produced after step 10.

Note: this PR does not include the fix from #10589; things will break here if a diacritic is the first entry on a line. This is known and expected.

TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID: Using the Keyman for Android app, verify that multitaps at the start of new paragraphs do not have unwanted side effects.

  1. Download this KMP: https://jahorton.github.io/diacritic_rota.kmp
  2. Open / install it for use within the Keyman app.
  3. Double-tap the 8 key.
    • Expected total result: a
  4. Triple-tap the '◌̀' key - a circumflex diacritic should appear over the a.
  5. Triple-tap the 9 key.
    • Expected total result:
  6. Tap the '◌̀' key. A diacritic should appear over the ê.
  7. Open the in-app Settings menu.
    • Note, for test maintenance: the point of this step and the next one is to trigger a context-reset.
  8. Close the in-app Settings menu.
  9. Double-tap the 8 key.
    • Expected final result: aềa
    • For clarity:
      • With a copy of the à's accent-mark on top the original ê.
        • It looks different in raw-text form here due to GitHub's font / rendering.
  10. Double-tap the ENTER key.
  11. Repeat steps 3-9. The result for the first and last lines should be identical, with no extra lines produced after step 10.

TEST_NEW_PARAGRAPH_PREDICTIONS: Using the iOS Keyman app on an iPhone 13 / iPhone 13 Pro, verify that predictive text works properly at the start of new paragraphs.

  • Type a few paragraphs, making sure to use predictive text to apply a suggestion at the start of each.
  • Type a few letters for the first word for some, but for others, immediately use a suggestion before typing anything (if possible).
  • On occasion, add extra new lines between new paragraphs.
  • If no suggestions are available after double-tapping ENTER, do not fail this test as a result.
    • It might be worth an issue, though... and I do see this occurring in my local tests.
  • If anything else unexpected occurs, or feels wrong, FAIL this test and take note of the issues.

Note: this PR does not include the fix from #10589; things will break here if a diacritic is the first entry on a line. This is known and expected.

@keymanapp-test-bot keymanapp-test-bot bot added has-user-test user-test-required User tests have not been completed labels Feb 15, 2024
@keymanapp-test-bot
Copy link

keymanapp-test-bot bot commented Feb 15, 2024

User Test Results

Test specification and instructions

  • TEST_LONG_DELETION (PASSED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Enabled Keyman as the system keyboard. 2. Opened Safari browser. Switched to the default iOS system keyboard. 3. Copied the text "Testing testing" from another PR's page on the test device. 4. Opened the Notes app and created a new note. Pasted the text copied from the PR's page. 5. Switched to Keyman Keyboard. 6. I confirmed that all the texts were deleted after pressing and holding down the Backspace key.
  • TEST_SHIFTED_CONTEXT_MULTITAP_IOS (PASSED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for two times. Verified that the results for the first and last lines were identical with no extra lines produced after step 4. (notes)
  • TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID (PASSED): Tested with the attached PR build (17.0.268-beta-test-10728) in the Android Mobile device (ver 13) and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for the second time. Verified that the results for the first and last lines were identical with no extra lines produced after step 4. However, there was a delay or slowness experienced (so that the number appeared on the screen) when double-tapping the 8 or 9 key on the keyboard. I have attached the Video file for reference. Thanks. (notes)
  • TEST_NEW_PARAGRAPH_PREDICTIONS (PASSED): Retested with the attached PR build (17.0.268-beta-test-10728) on an iPhone 13 Mobile device and here are my observations: Tried a lot to replicate the issue that I have already posted, but this time I did not see any predictive text stuck during the process. Also, I tried to save this in a video file (3 minute .mp4 video) but its size more than 60MB. I tried to compress the video file but it only compressed upto 30MB. I was not able to reproduce this issue. Anyhow, I will keep an eye on this issue and will post as a separate one if it is reproducible.

Test Artifacts

@keymanapp-test-bot keymanapp-test-bot bot added this to the B17S1 milestone Feb 15, 2024
Base automatically changed from fix/web/transform-selection-interactions to beta February 15, 2024 08:24
@bharanidharanj
Copy link

bharanidharanj commented Feb 15, 2024

Test Results

  • TEST_LONG_DELETION (PASSED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Enabled Keyman as the system keyboard. 2. Opened Safari browser. Switched to the default iOS system keyboard. 3. Copied the text "Testing testing" from another PR's page on the test device. 4. Opened the Notes app and created a new note. Pasted the text copied from the PR's page. 5. Switched to Keyman Keyboard. 6. I confirmed that all the texts were deleted after pressing and holding down the Backspace key.

@bharanidharanj
Copy link

Test Results

  • TEST_SHIFTED_CONTEXT_MULTITAP_IOS (PASSED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for two times. Verified that the results for the first and last lines were identical with no extra lines produced after step 4.

@bharanidharanj
Copy link

Test Results

  • TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID (FAILED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for the second time. Noticed that the Double-tap the 8 key showed the number 8 on the text screen. Seems to be an issue.

@bharanidharanj
Copy link

Test Results

  • TEST_NEW_PARAGRAPH_PREDICTIONS (FAILED): Tested with the attached PR build (17.0.268-beta-test-10728) on an iPhone 13 Mobile device and here are my observation: 1. Typed a few paragraphs and applied suggestions while typing words. 2. I noticed that tapping on the word "many" in the banner incorrectly outputs the word "managed" on the text screen. 3. Additionally, long-pressing the period key did not bring up the subkey menu. 4. Sometimes, a key on the keyboard would get stuck and appear in blue color.

@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Feb 15, 2024
@jahorton
Copy link
Contributor Author

Test Results

  • TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID (FAILED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for the second time. Noticed that the Double-tap the 8 key showed the number 8 on the text screen. Seems to be an issue.

on iPhone 13 Mobile

@bharanidharanj

I'm pretty sure that's not an iPhone 13 in the screenshot. Alas, I'm having trouble getting a local repro of this at the moment, so I actually might need details for the device you were using in order to reproduce the issue properly.

Alternatively, it might help to have a video of the repro - largely to confirm that both intended taps did land. If only one did... yeah, you'd be stuck with that 8.

@jahorton
Copy link
Contributor Author

jahorton commented Feb 16, 2024

Test Results

  • TEST_NEW_PARAGRAPH_PREDICTIONS (FAILED): Tested with the attached PR build (17.0.268-beta-test-10728) on an iPhone 13 Mobile device and here are my observation: 1. Typed a few paragraphs and applied suggestions while typing words. 2. I noticed that tapping on the word "many" in the banner incorrectly outputs the word "managed" on the text screen. 3. Additionally, long-pressing the period key did not bring up the subkey menu. 4. Sometimes, a key on the keyboard would get stuck and appear in blue color.

     <img alt="" width="300" src="https://private-user-images.githubusercontent.com/19683143/305104770-39f87fa7-d96f-46ad-bc30-9773094e5f7b.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDgwNDgwOTksIm5iZiI6MTcwODA0Nzc5OSwicGF0aCI6Ii8xOTY4MzE0My8zMDUxMDQ3NzAtMzlmODdmYTctZDk2Zi00NmFkLWJjMzAtOTc3MzA5NGU1ZjdiLmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAyMTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMjE2VDAxNDMxOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIzYWQ2NTY2Y2IxM2Y2YTY0ZWM3ZDlkOTVmMzUwNDcyOThhYzc2ZjAzOGY2ZTIzM2ZiMzIxZTNjMTBjNDk4MjgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.gsWDcJ_cX0bc4BpCc1AdA-LQizr7QBQwfi9OiBrt4VA">
    

@bharanidharanj I'm going to have to ask for video on this one. I'm not currently able to reproduce it, and the applied suggestion is highlighted in your screenshot above. There's an admittedly small chance of this, but the possibility exists that I'd need to replicate what you did quite precisely to get the same effects, and video would be, by far, the most convenient way to provide that.

As for the other bits (sticky highlighting, period subkey menu), I've gone ahead and merged in recent changes that may be related, so... on that basis...

@keymanapp-test-bot retest TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID TEST_NEW_PARAGRAPH_PREDICTIONS

See my previous comment re: the Android-oriented redo request.

Also note that the sticky-highlighting issue is being handled by a separate PR that is not yet merged - #10728. Do not report sticky-highlighting errors for keys or fail these tests on that basis. Do report sticky highlighting for suggestions if that occurs, though.

@keymanapp-test-bot keymanapp-test-bot bot added user-test-required User tests have not been completed and removed user-test-failed labels Feb 16, 2024
@bharanidharanj
Copy link

Test Results

  • TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID (FAILED): Tested with the attached PR build (17.0.268-beta-test-10728) on iPhone 13 Mobile device and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for the second time. Noticed that the Double-tap the 8 key showed the number 8 on the text screen. Seems to be an issue.

on iPhone 13 Mobile

@bharanidharanj

I'm pretty sure that's not an iPhone 13 in the screenshot. Alas, I'm having trouble getting a local repro of this at the moment, so I actually might need details for the device you were using in order to reproduce the issue properly.

Alternatively, it might help to have a video of the repro - largely to confirm that both intended taps did land. If only one did... yeah, you'd be stuck with that 8.

@jahorton Oops! Sorry for the confusion. Yes. I have tested this in my Android Mobile (ver 13) device. Sure. I will add a video file for reference. Thanks.

@bharanidharanj
Copy link

Test Results

  • TEST_SHIFTED_CONTEXT_MULTITAP_ANDROID (PASSED): Tested with the attached PR build (17.0.268-beta-test-10728) in the Android Mobile device (ver 13) and here is my observation: 1. Downloaded and installed diacritic_rota.kmp file in the Keyman In-App. 2. Switched to English - Diacritic 10-key Rota keyboard. 3. Followed the steps from 3 - 9 and verified that it showed the expected final result aềa. 4. Double-tapped the ENTER key. 5. Repeated Steps 3 - 9 for the second time. Verified that the results for the first and last lines were identical with no extra lines produced after step 4. However, there was a delay or slowness experienced (so that the number appeared on the screen) when double-tapping the 8 or 9 key on the keyboard. I have attached the Video file for reference. Thanks.
Multitapfn.mp4

@bharanidharanj
Copy link

Test Results

  • TEST_NEW_PARAGRAPH_PREDICTIONS (PASSED): Retested with the attached PR build (17.0.268-beta-test-10728) on an iPhone 13 Mobile device and here are my observations: Tried a lot to replicate the issue that I have already posted, but this time I did not see any predictive text stuck during the process. Also, I tried to save this in a video file (3 minute .mp4 video) but its size more than 60MB. I tried to compress the video file but it only compressed upto 30MB. I was not able to reproduce this issue. Anyhow, I will keep an eye on this issue and will post as a separate one if it is reproducible.

@keymanapp-test-bot keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Feb 16, 2024
@mcdurdin mcdurdin modified the milestones: B17S1, B17S2 Feb 17, 2024
@keymanapp keymanapp deleted a comment from gitguardian bot Feb 17, 2024
Copy link
Member

@mcdurdin mcdurdin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think LGTM. I don't really understand all the tendrils here -- there's a whole lot of complexity both in the original code and in the changes, and I don't think I have a full handle on all the interactions.

Upon request, I suppose I could create a highly-focused test keyboard for validating such a case. That said, it does seem pretty niche - I don't think we currently have a keyboard with such rules. (As a result, I haven't prioritized testing this scenario.) Part of me wonders if the best course of action would be to spin this concern off into a separate issue so that it doesn't hold this PR up from fixing a lot of cases that are far more commonly seen.

Yes, sounds sensible, probably turn it into an issue.

Comment on lines 286 to 288
var end = pos + length;
// undefined + <number> => NaN.
end = isNaN(end) ? undefined : end;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
var end = pos + length;
// undefined + <number> => NaN.
end = isNaN(end) ? undefined : end;
const end = start === undefined ? undefined : start + length;

We probably shouldn't be using var any longer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fix the var bit.

The comment isn't explicit about it, but both pos and length can be undefined. The same property holds if both are undefined at the same time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, that's pretty ugly and side-effecty. Let's make sure we add a comment to that effect.

keyman.resetContext();
}

return keyman.context.getText();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this never used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Here's the only thing calling it:

webView!.evaluateJavaScript("setKeymanVal(\"\(jsonText)\");", completionHandler: nil)

Note how nothing is checking for a result. To be extra clear here, that would be done by the second argument if one were provided.

https://developer.apple.com/documentation/webkit/wkwebview/1415017-evaluatejavascript

No callback means no need for a return value.

Copy link

gitguardian bot commented Feb 19, 2024

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!

@jahorton
Copy link
Contributor Author

Upon request, I suppose I could create a highly-focused test keyboard for validating such a case. That said, it does seem pretty niche - I don't think we currently have a keyboard with such rules. (As a result, I haven't prioritized testing this scenario.) Part of me wonders if the best course of action would be to spin this concern off into a separate issue so that it doesn't hold this PR up from fixing a lot of cases that are far more commonly seen.

Yes, sounds sensible, probably turn it into an issue.

I've made a basic issue for this at #10761.

@jahorton jahorton merged commit 453d24d into beta Feb 19, 2024
15 checks passed
@jahorton jahorton deleted the fix/ios/context-slide-into-multitap branch February 19, 2024 03:17
@keyman-server
Copy link
Collaborator

Changes in this pull request will be available for download in Keyman version 17.0.271-beta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants