change(web): initializes OSK with keyboard if available during init 🪠 #11174

jahorton · 2024-04-05T03:42:55Z

By delaying the end of KeymanWeb's init method in order to check if any keyboards are pre-registered, we can initialize the OSK correctly "the first time". This matters by far the most on touch devices, where we have been building a stand-in default touch-layout the first time first due to not waiting long enough for pre-registered keyboards to fully load. (On desktop devices, we just build a very simple, empty stand-in.) Preparing the stand-in is surprisingly expensive/slow on some older touch devices.

By preventing unnecessary construction of a stand-in touch layout (and thus, related DOM initialization + layout reflow) on touch devices when fully leveraged, we can eliminate a notable source of Web engine startup lag that has long been affecting our mobile apps.

I have verified that this allows us to fully bypass the empty touch-layout construction step within the Android Engine, notably cutting the host-page startup time. It currently appears that the iOS app is unable to leverage this as-is, though.

With a little rework of Web's test pages, it can be demonstrated to work for standalone KeymanWeb as well. Their current structures generally do not register keyboards until engine-initialization is complete, which is a necessary prerequisite.

User Testing

TEST_WINDOWS_DESKTOP: Test general use of KeymanWeb on a Windows desktop or laptop. Report any unexpected behavior.

Refresh the page and start using KeymanWeb least 5 times. (Test page: "Test unminified Web")

TEST_ANDROID_TABLET: Test general use of KeymanWeb on an Android tablet. Report any unexpected behavior.

Force-quit and restart the app at least 5 times.
Swap in and out of the system keyboard at least 5 times.

TEST_IOS_PHONE: Test general use of KeymanWeb on an iPhone. Report any unexpected behavior.

Force-quit and restart the app at least 5 times.
Swap in and out of the system keyboard at least 5 times.

keymanapp-test-bot · 2024-04-05T03:43:00Z

User Test Results

Test specification and instructions

✅ TEST_WINDOWS_DESKTOP (PASSED): Tested using "Test unminified Web" page on a Windows desktop and here is my observation: 1. Refresh the test page. 2. Tested text selection and CJK keyboard functions on a Windows desktop. 3. Repeated Steps 1 and 2 more than 5 times and verified that everything works fine.
✅ TEST_ANDROID_TABLET (PASSED): Retested with the attached PR build (Keyman 17.0.303-beta-test-11174) on an Android 12 / API 30 emulator and here is my observation: 1. Opened the Keyman In-app. 2. Force-quit and restarted the app more than five times, verifying that no issues occurred during this process. 3. Set Keyman as the System keyboard. 4. Opened Chrome browser. 5. Switched to Keyman Keyboard. 6. Swapped to Gboard using the globe key. Repeat step 5 and 6 for more than five times and verified that no problems occurred during the process.
✅ TEST_IOS_PHONE (PASSED): Retested with the attached PR build (Keyman 17.0.303-beta-test-11174) on an iPhone 13 Mobile device (iOS 17.4) and here is my observation: 1. Opened the Keyman In-app. 2. Force-quit and restarted the app more than five times, verifying that no issues occurred during this process. 3. Set Keyman as the System keyboard. 4. Opened Safari browser. 5. Switched to Keyman Keyboard. 6. Swapped to default keyboard using the globe key. Repeat step 5 and 6 for more than five times and verified that no problems occurred during the process.

Test Artifacts

Android
Developer
iOS
- Keyman for iOS (simulator image)
- FirstVoices Keyboards for iOS (simulator image)
- TestFlight internal PR build version - 17.0.310 (0.11174.10776)
Keyboards
- Test Keyboards
Web
- KeymanWeb Test Home
Windows

This prevents unnecessary construction of a stand-in touch layout and related DOM initialization on touch devices when fully leveraged.

jahorton · 2024-04-08T01:46:32Z

Just noticed an issue with when the OSK is in desktop-keyboard mode; needs a bit more refinement first.

bharanidharanj · 2024-04-08T10:13:18Z

Test Results

TEST_WINDOWS_DESKTOP (PASSED): Tested using "Test unminified Web" page on a Windows desktop and here is my observation: 1. Refresh the test page. 2. Tested text selection and CJK keyboard functions on a Windows desktop. 3. Repeated Steps 1 and 2 more than 5 times and verified that everything works fine.

bharanidharanj · 2024-04-08T10:45:18Z

Test Results

TEST_ANDROID_TABLET (FAILED): Tested with the attached PR build (Keyman 17.0.303-beta-test-11174) on an Android 12 Tablet / API 30 emulator and here is my observation: 1. Force-quit and restart the app for 5 times. 2. Swapped in and out of the system keyboard at least 5 times and noticed that the Keyman Keyboard banner appears instead of the predictive text after keyboard swapping. Seems to be an issue.

bharanidharanj · 2024-04-08T11:02:03Z

Test Results

TEST_IOS_PHONE (FAILED): Tested with the attached PR build (Keyman 17.0.303-beta-test-11174) on an iPhone 13 mobile device and here is my observation: 1. Noticed that a blank keyboard appeared on the screen after opening the keyman In-App. Seems to be an issue.

sentry-io · 2024-04-09T01:16:31Z

Sentry Issue: KEYMAN-WEB-JH

From the iOS user-test failure

jahorton · 2024-04-10T04:06:45Z

@keymanapp-test-bot retest TEST_IOS_PHONE TEST_ANDROID_TABLET

I've added code that should ensure the banner displays consistently now; the OSK now proactively checks what mode the banner should be in when it's initialized. (Previously, it always waited to be told.)

bharanidharanj · 2024-04-10T09:34:46Z

Test Results

TEST_ANDROID_TABLET (PASSED): Retested with the attached PR build (Keyman 17.0.303-beta-test-11174) on an Android 12 / API 30 emulator and here is my observation: 1. Opened the Keyman In-app. 2. Force-quit and restarted the app more than five times, verifying that no issues occurred during this process. 3. Set Keyman as the System keyboard. 4. Opened Chrome browser. 5. Switched to Keyman Keyboard. 6. Swapped to Gboard using the globe key. Repeat step 5 and 6 for more than five times and verified that no problems occurred during the process.

bharanidharanj · 2024-04-10T09:44:13Z

Test Results

TEST_IOS_PHONE (PASSED): Retested with the attached PR build (Keyman 17.0.303-beta-test-11174) on an iPhone 13 Mobile device (iOS 17.4) and here is my observation: 1. Opened the Keyman In-app. 2. Force-quit and restarted the app more than five times, verifying that no issues occurred during this process. 3. Set Keyman as the System keyboard. 4. Opened Safari browser. 5. Switched to Keyman Keyboard. 6. Swapped to default keyboard using the globe key. Repeat step 5 and 6 for more than five times and verified that no problems occurred during the process.

mcdurdin · 2024-04-13T04:32:28Z

android/KMEA/app/src/main/assets/android-host.js

+  try {
+    KeymanWeb.registerStub(JSON.parse(jsInterface.initialKeyboard()));
+  } catch {};


We should not be silently throwing away exceptions. If there is a specific exception that we need to handle and ignore, then that needs to be documented -- at the very least with a comment. But better to avoid the exception altogether if possible.

Note: this is location that, if an error triggers, is guaranteed to cause a blank keyboard. It's before the keyman.init call.

Here's a Sentry event this is meant to silence: https://keyman.sentry.io/share/issue/d9d9bd46c7404254891b606b87570bec/

I'm... not sure how that error looks on the JS side. I can attempt to reproduce it and see if there's a simple-enough way to filter it.

Inspecting that Sentry event, though... it hasn't really shown up in several builds, so perhaps it is safe to drop. (Last case was from 17.0.293-beta.)

Perhaps the best solution for now is log all exceptions to sentry, and then we can filter out known 'good' exceptions as we encounter them. So we should never have an empty catch {} -- it should at the very least have a tellSentryAboutMyPain() even if we then want to continue on as 'normal'.

I went ahead and dropped the try-catch entirely.

... and restored, now with a console.error(<error>) in it.

…ding

keyman-server · 2024-04-19T18:07:54Z

Changes in this pull request will be available for download in Keyman version 17.0.311-beta

change(web): revert #11174, which loads keyboards before initializing the OSK

…-revert-11174 change(web): revert #11174, which loads keyboards before initializing the OSK 🍒 🏠

keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Apr 5, 2024

github-actions bot added web/ android/ android/engine/ labels Apr 5, 2024

keymanapp-test-bot bot added this to the B17S5 milestone Apr 5, 2024

jahorton force-pushed the change/web/anti-layout-reflow-thrashing branch from da7ea66 to c595303 Compare April 5, 2024 06:04

jahorton added 2 commits April 5, 2024 13:07

change(web): initializes OSK with keyboard if available during init

4b1394a

This prevents unnecessary construction of a stand-in touch layout and related DOM initialization on touch devices when fully leveraged.

fix(web): OSK set activeKeyboard should not bypass during init

eebf3e5

jahorton force-pushed the change/web/init-osk-with-keyboard branch from 56f28bb to eebf3e5 Compare April 5, 2024 06:07

keymanapp-test-bot bot added has-user-test user-test-required User tests have not been completed and removed user-test-missing User tests have not yet been defined for the PR labels Apr 5, 2024

jahorton marked this pull request as ready for review April 8, 2024 01:30

jahorton requested review from ermshiperete, darcywong00 and sgschantz as code owners April 8, 2024 01:30

jahorton marked this pull request as draft April 8, 2024 01:46

fix(web): adds newly-needed null guards due to new pattern

786c480

jahorton marked this pull request as ready for review April 8, 2024 01:56

keymanapp-test-bot bot added the user-test-failed label Apr 8, 2024

keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Apr 8, 2024

fix(ios): ios-host adjustments

829d40f

github-actions bot added ios/ ios/engine/ labels Apr 9, 2024

github-actions bot added the common/web/ label Apr 10, 2024

keymanapp-test-bot bot added user-test-required User tests have not been completed and removed user-test-failed labels Apr 10, 2024

keymanapp-test-bot bot removed the user-test-required User tests have not been completed label Apr 10, 2024

ermshiperete approved these changes Apr 10, 2024

View reviewed changes

Base automatically changed from change/web/anti-layout-reflow-thrashing to beta April 12, 2024 00:49

darcywong00 modified the milestones: B17S5, B17S6 Apr 12, 2024

chore(web): Merge branch 'beta' into change/web/init-osk-with-keyboard

b107daa

mcdurdin reviewed Apr 18, 2024

View reviewed changes

chore(android): reverts error-silencing try-catch

db4a7bf

github-actions bot added android/ android/engine/ and removed android/ android/engine/ labels Apr 18, 2024

chore(android): restores try-catch, now with console.error err forwar…

d705589

…ding

jahorton merged commit 09a65a4 into beta Apr 19, 2024
19 checks passed

jahorton deleted the change/web/init-osk-with-keyboard branch April 19, 2024 05:41

jahorton mentioned this pull request Jun 14, 2024

bug(web): desktop OSK fails to load, with error, if keyboards are added before init() is called #11785

Closed

jahorton mentioned this pull request Jul 15, 2024

bug(web): Console error reading modifierCodes of undefined after installing custom Greek keyboard package #11962

Closed

This was referenced Jul 25, 2024

fix(web): fix engine initialization when initial keyboard is debug-compiled #11968

Closed

change(web): revert #11174, which loads keyboards before initializing the OSK #12015

Merged

mcdurdin mentioned this pull request Jul 26, 2024

change(web): revert #11174, which loads keyboards before initializing the OSK 🍒 🏠 #12040

Merged

jahorton added a commit that referenced this pull request Jul 29, 2024

Merge pull request #12015 from keymanapp/change/web/revert-11174

7e5903b

change(web): revert #11174, which loads keyboards before initializing the OSK

jahorton mentioned this pull request Jul 29, 2024

fix(web): unrevert #11258, leaving OSK hidden before instructed to display #12049

Merged

mcdurdin added a commit that referenced this pull request Jul 30, 2024

Merge pull request #12040 from keymanapp/change/web/cherry-pick/11962…

faaf90a

…-revert-11174 change(web): revert #11174, which loads keyboards before initializing the OSK 🍒 🏠

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change(web): initializes OSK with keyboard if available during init 🪠 #11174

change(web): initializes OSK with keyboard if available during init 🪠 #11174

jahorton commented Apr 5, 2024 •

edited

Loading

keymanapp-test-bot bot commented Apr 5, 2024 •

edited

Loading

jahorton commented Apr 8, 2024

bharanidharanj commented Apr 8, 2024

bharanidharanj commented Apr 8, 2024

bharanidharanj commented Apr 8, 2024

sentry-io bot commented Apr 9, 2024

jahorton commented Apr 10, 2024

bharanidharanj commented Apr 10, 2024

bharanidharanj commented Apr 10, 2024

mcdurdin Apr 13, 2024

jahorton Apr 18, 2024

jahorton Apr 18, 2024

mcdurdin Apr 19, 2024

jahorton Apr 19, 2024

jahorton Apr 19, 2024

keyman-server commented Apr 19, 2024

change(web): initializes OSK with keyboard if available during init 🪠 #11174

change(web): initializes OSK with keyboard if available during init 🪠 #11174

Conversation

jahorton commented Apr 5, 2024 • edited Loading

User Testing

keymanapp-test-bot bot commented Apr 5, 2024 • edited Loading

User Test Results

Test Artifacts

jahorton commented Apr 8, 2024

bharanidharanj commented Apr 8, 2024

Test Results

bharanidharanj commented Apr 8, 2024

Test Results

bharanidharanj commented Apr 8, 2024

Test Results

sentry-io bot commented Apr 9, 2024

jahorton commented Apr 10, 2024

bharanidharanj commented Apr 10, 2024

Test Results

bharanidharanj commented Apr 10, 2024

Test Results

mcdurdin Apr 13, 2024

Choose a reason for hiding this comment

jahorton Apr 18, 2024

Choose a reason for hiding this comment

jahorton Apr 18, 2024

Choose a reason for hiding this comment

mcdurdin Apr 19, 2024

Choose a reason for hiding this comment

jahorton Apr 19, 2024

Choose a reason for hiding this comment

jahorton Apr 19, 2024

Choose a reason for hiding this comment

keyman-server commented Apr 19, 2024

jahorton commented Apr 5, 2024 •

edited

Loading

keymanapp-test-bot bot commented Apr 5, 2024 •

edited

Loading