Compliance with iOS Permissions Guidelines #1094

JGreenlee · 2024-10-03T16:05:41Z

Our current permissions flow is deemed unacceptable. I think this is due to:

On iOS 13+ for location settings, we send users to the settings page without prompting any of the system popups for location requests
- I think we are doing this to avoid the "provisional" always authorization flow, which does not work well for OpenPATH's use case. We need users to allow "always" upfront, otherwise there is no point in using the app.
- On iOS 13.4+, we can use a 2-step strategy, where we first obtain 'when in use' permission, then obtain 'always' permission, to avoid the "provisional" always flow https://developer.apple.com/videos/play/wwdc2020/10660/?time=1274. If we do this, then it is only iOS >=13,<13.4 that we have to send users to the app settings page.
Notifications permissions have to be optional but are currently required
- I believe that this only applies to user-facing notifications, not silent notifications. If the users refuses this permission, tracking should still work and we can allow them to proceed. All users will be missing is labeling reminders.

And perhaps a third point:

The verbiage we use to describe the permissions (e.g. "Fix", "Please select 'Always allow'", etc) could be perceived as "displaying a screen or window that looks like a request" or "influencing" the user's choices https://developer.apple.com/design/human-interface-guidelines/privacy#Pre-alert-screens-windows-or-views

I am working on the changes to resolve these issues, and hopefully provide a better onboarding UX in the process.

JGreenlee · 2024-10-03T16:14:07Z

Given that latest versions of iOS will use a two-step strategy, and latest versions of Android are also using a two-step strategy (#990, implemented in e-mission/e-mission-data-collection#210), I think the best long-term solution may be to split the "Location Permissions" item into two separate items

However, for now I will just try to meet the guidelines as quickly as possible so we are unblocked

shankari · 2024-10-03T17:07:23Z

I think we are doing this to avoid the "provisional" always authorization flow, which does not work well for OpenPATH's use case. We need users to allow "always" upfront, otherwise there is no point in using the app.

Yes. We used to use the standard popups, but then people would not give us the correct permissions and then complain that "the app does not work"

shankari · 2024-10-03T17:40:22Z

@catarial is going to help verify

I believe that this only applies to user-facing notifications, not silent notifications.

To do that, we need to turn off the notification permission and see if we still get the silent push notifications once an hour.
However, we currently include the notification checks as part of the regular "is my app working" checks that we do in the background, and I think that if the checks fail, we go into the START state. My recollection is that we should keep getting the push notifications even in the start state, but we should verify that.

So I would suggest two potential ways to check this:

Option 1:

turn off location from the app settings
wait for 4 hours; check to see that we went into the START state, but we do see the T_RECEIVED_SILENT_PUSH logs
then turn location on, but turn notifications off
wait for 4 hours; see if we received T_RECEIVED_SILENT_PUSH in the same way

Option 2:
We can also just disable the notification checks so that the FSM will stay in the WAITING_FOR_TRIP_STATE. In that case, we would need to change code, but we don't need to turn the location off first to set the baseline because we will still be in a valid state.

The set of checks for validity are:
https://github.com/e-mission/e-mission-data-collection/blob/dcc4853919089527427c444260f63a49dca66af3/src/ios/Verification/TripDiarySensorControlChecks.m

So to hack, we can just change checkNotificationsEnabled to always return TRUE.

With either option, we should be able to see whether turning off notifications affects silent push notifications. So we know/can see that we get silent push when the notification permission is present; do we get them when the permission is off.

catarial · 2024-10-03T21:13:06Z

I left the simulator running with the notification check disabled for about 2 hours and the only thing I saw in the debug logs was one instance of "Ignoring silent push notification".

catarial · 2024-10-03T22:33:31Z

Im getting the same thing when I turn notifications back on. I need test this more. What makes this most challenging is that I have to disable push notifications to be able to install the test app on a real phone.

catarial · 2024-10-03T22:34:47Z

I'll try option 1 with the app store version on a real phone and see what happens tomorrow.

shankari · 2024-10-03T22:48:00Z

@catarial push notifications are not delivered to the simulator (you may see the notification about that when you start up the app). You have to try it on a physical phone.

catarial · 2024-10-04T16:23:41Z

So I didn't see any silent push notifications, even when notifications were turned on. It seems like the app stopped logging when I turned location off. Do I have to keep the phone unlocked or the app open for it to log?

catarial · 2024-10-04T16:27:54Z

This is when I turned notifications off

1106,1728011701.52389,2024-10-03T23:15:01.523890-04:00,Application went to the background
1107,1728011701.66687,2024-10-03T23:15:01.666870-04:00,new AppState: background
1108,1728052761.58053,2024-10-04T10:39:21.580530-04:00,Application will enter the foreground

Either it doesn't log in the background or it doesn't log if it doesn't have all the required permissions

JGreenlee · 2024-10-04T17:12:09Z

I have the majority of these changes done now.

I implemented the two-step approach for location permissions on iOS versions >=13.4. iOS 13.0 to 13.3 is still using the old flow – hopefully that is not a problem.

I am still running into one issue where after granting location permission via popups, the UI does not update with a green checkmark until clicked again or "refresh" is clicked.
I think this is because the native code is missing a return value.

Making the notifications optional was mainly just a UI change in e-mission-phone. Added an optional flag to the checks and allow overallStatus to be true even if an optional permission has been denied.

I adjusted text in several places, a few examples:

"Please allow." -> "Allow motion and fitness tracking to enable mode detection"
"Location services should be enabled. This allows us to access location data and generate the trip log" -> "Turn on location services to enable location tracking"

I also found that the location permissions popups were displaying text related to iBeacons (e.g. "This app would like to scan for iBeacons while it is in use.") Those strings aren't present in any e-mission codebase.

We actually have our own strings for those descriptions: https://github.com/e-mission/e-mission-data-collection/blob/dcc4853919089527427c444260f63a49dca66af3/plugin.xml#L214-L224

But they were being overridden by the descriptions from cordova-plugin-ibeacon: https://github.com/petermetz/cordova-plugin-ibeacon/blob/270ffbbc12159861a16e5e81481103c1e09139cb/plugin.xml#L61-L72

Hopefully adjusting the order that the plugins are listed in package.cordovabuild.json will prevent this override.

JGreenlee · 2024-10-04T18:01:50Z

Hopefully adjusting the order that the plugins are listed in package.cordovabuild.json will prevent this override.

This worked, but I can't find internationalized versions of these strings. Are we missing translations??

shankari · 2024-10-04T19:08:26Z

@catarial wrt #1094 (comment), do you see push notifications in the logs before you made the permission changes?

@JGreenlee wrt #1094 (comment), it is possible. We were really scrambling to get the beta version out ASAP.

catarial · 2024-10-04T20:53:46Z

@catarial wrt #1094 (comment), do you see push notifications in the logs before you made the permission changes?

no, I'm retesting with the phone on and the app open

e-mission/e-mission-docs#1094 (comment) If on iOS 13.4 or higher, we will first ask for 'whenInUse' authorization. If the user accepts this, we will receive the authorization status change in didChangeAuthorizationStatus; at which point we will attempt to trigger another request, this time for 'always', which should show the user a second prompt. If that fails, we will navigate to the app settings. This approach involved re-registering the foreground delegate from within that foreground delegate's handler, so I adjusted the way the delegate list is cleared, so as to not clear out the new additions. Also adjusted the strings in plugin.xml to be more descriptive and transparent about what permissions are needed for tracking, as well as when and why.

shankari · 2024-10-04T21:48:16Z

Wait, are you testing with the emission app that you built and deployed on the phone? If so, that would explain it, since the public app does not have the correct credentials/keys to register for OpenPATH push notifications. This is as we would expect, since we don't want anybody on the internet to get notifications from our instance - they should get their own credentials.
If this is the case, you can test using the version from TestFlight. Our test gmail account should already be enrolled in TestFlight, and you should find the invite to install the app in one of the emails, if not in the app already

Note (for your learning) that this again highlights why we need the baseline. If we had tested only with notification permission off, we may have concluded that no permission = no notifications. But since we never saw notifications, we know that there is something more fundamentally incorrect.

shankari · 2024-10-05T07:01:20Z

I tried to test this out on my own physical phone, and I am not seeing silent push notifications since Sept 12, even for an app installed using TestFlight

$ grep "Received notification T_RECEIVED_SILENT_PUSH" /tmp/loggerDB.check_silent_push.withdate.log | tail -n 10
9521,1725397158.60911,2024-09-03T13:59:18.609110-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
9550,1725397159.16517,2024-09-03T13:59:19.165170-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
24191,1726103615.3623,2024-09-11T18:13:35.362300-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
24250,1726103800.83029,2024-09-11T18:16:40.830290-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
24295,1726103803.83894,2024-09-11T18:16:43.838940-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
24340,1726105053.73581,2024-09-11T18:37:33.735810-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
24385,1726105081.42415,2024-09-11T18:38:01.424150-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
24430,1726105084.5227,2024-09-11T18:38:04.522700-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
26528,1726113423.37986,2024-09-11T20:57:03.379860-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification
26902,1726158219.42557,2024-09-12T09:23:39.425570-07:00,Received notification T_RECEIVED_SILENT_PUSH while processing silent push notification

shankari · 2024-10-05T07:09:53Z

Checking the logs, that is because the previous fix to migrate to the new version of the FCM HTTP API (e-mission/e-mission-server#980) works for visible notifications, but not for silent push. Will fix that and retry this weekend.

Traceback (most recent call last):
  File "/usr/src/app/bin/push/silent_ios_push.py", line 24, in <module>
    response = pnu.send_silent_notification_to_ios_with_interval(args.interval, dev=args.dev)
  File "/usr/src/app/emission/net/ext_service/push/notify_usage.py", line 41, in send_silent_notification_to_ios_with_interval
    return __get_default_interface__().send_silent_notification(token_map, 
{}
, dev)
  File "/usr/src/app/emission/net/ext_service/push/notify_interface_impl/firebase.py", line 194, in send_silent_notification
    fcm_token_map = self.convert_to_fcm_if_necessary(token_map, dev)
  File "/usr/src/app/emission/net/ext_service/push/notify_interface_impl/firebase.py", line 137, in convert_to_fcm_if_necessary
    importedResultList = self.retrieve_fcm_tokens(unmapped_token_list, dev)
  File "/usr/src/app/emission/net/ext_service/push/notify_interface_impl/firebase.py", line 98, in retrieve_fcm_tokens
    importHeaders = {"Authorization": "key=%s" % self.server_auth_token,
AttributeError: 'FirebasePush' object has no attribute 'server_auth_token'

JGreenlee · 2024-10-06T06:21:24Z

Testing done for e-mission/e-mission-phone#1182 and e-mission/e-mission-data-collection#237, screen recordings showing some of the scenarios tested

iOS 13.3

Granted location + fitness, denied notifications:

Untitled.mp4

iOS 15.6

Granted location ("Allow While Using App" on first popup, then "Change to Always Allow" on second popup). Granted fitness. Denied notifications.

Untitled2.mp4

Initially chose "Just Once" for location, then fixed in app settings. Granted fitness. Denied notification.

Untitled1.mp4

Initially denied location, then fixed in app settings. Initially denied fitness, then fixed in app settings. Granted notifications.

Untitled3.mp4

After taking these I realized that the location permissions popups do not actually show up in the screen recordings. But I hope you can tell what's happening by how I have labeled the scenarios

JGreenlee · 2024-10-06T06:27:10Z

The one thing I would flag as problematic is in the last scenario, where fitness is denied and then fixed in app settings. When returning to the app, it appeared to restart so I had to go back through the privacy policy. I was ultimately able to get through onboarding, but it was a bit annoying and might signify an issue in the native code that caused a crash. I don't think it was a result of my changes.

shankari · 2024-10-06T21:58:10Z

I fixed the silent push issue in e-mission/e-mission-server#987 (edit: validation in the staging environment is at e-mission/e-mission-server#987 (comment)) although it needs some additional investigation on why it only occurs on certain deployments.

Now we can return to this verification:

verify that the silent push notifications show up on the baseline
turn off the location permission
check if they continue showing up
turn off the notification permission
check if they continue showing up

shankari · 2024-10-07T15:17:48Z

Time	notes
3:59	silent push sent from the server
3:59	received silent push on phone (`51390,1728255561.7504,2024-10-06T15:59:21.750400-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"`)
4:23	turned off location permissions from the settings screen (Always -> Never)
4:59	silent push sent from the server
4:59	silent push received on phone (`51832,1728259160.58975,2024-10-06T16:59:20.589750-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"`)
4:59	phone detects that settings are wrong and requests a fix `allChecksPass = 0, but location checks fail, generating error and notification, allChecks: [loc settings, loc permissions, motion settings, motion permissions, notification] = (1, 0, 1, 1, 1)"`
4:59	We stay in `WAITING_FOR_TRIP_START` even when location is turned off. We should fix this [1]
5:15	Launch app and give location permission; app stays stuck in `WAITING_FOR_TRIP_START`, but does not actually create a geofence again (at least as far as I can make out)
5:15	Launch app and give location permission; app stays stuck in `WAITING_FOR_TRIP_START`, but does not actually create a geofence again (at least as far as I can make out)
5:37	Turn off notification permissions from the settings
5:59	silent push sent from the server
6:05	launched app, received notification to turn on permissions. Note that there was no notification to do this, since we have notifications turned off, but we did receive the push notification [2]
6:40	Turned off notification permissions again to ensure that this is reliable; will keep it off overnight
8:00 (next day)	Received silent push notifications consistently overnight [3]

[1] FSM gets stuck in `WAITING_FOR_TRIP_STATE if settings are wrong

51851,1728259161.09846,2024-10-06T16:59:21.098460-07:00,"allChecksPass = 0, but location checks fail, generating error and notification, allChecks: [loc settings, loc permissions, motion settings, motion permissions, notification] = (
    1,
    0,
    1,
    1,
    1
)"
51852,1728259161.12093,2024-10-06T16:59:21.120930-07:00,"In TripDiaryStateMachine, received transition T_GEOFENCE_CREATION_ERROR in state STATE_WAITING_FOR_TRIP_START"
51853,1728259161.14963,2024-10-06T16:59:21.149630-07:00,"Found existing field {
    redirectParams =     {
        launchAppStatusModal = 1;
    };
    redirectTo = ""root.main.control"";
}, retaining existing"
51854,1728259161.15339,2024-10-06T16:59:21.153390-07:00,"Did not find existing field (null), filling in default {
    type = calendar;
}"
51855,1728259161.15973,2024-10-06T16:59:21.159730-07:00,"Did not find existing field (null), filling in default {
    enabled = 0;
}"
51856,1728259161.17056,2024-10-06T16:59:21.170560-07:00,"In valid state STATE_WAITING_FOR_TRIP_START, nothing to do..."

[2] Receiving push notification when notification permissions are turned off

52618,1728260263.7866,2024-10-06T17:17:43.786600-07:00,"addStatReading: adding CLIENT_TIME event: {""name"":""app_state_change"",""ts"":1728260263.764,""reading"":""background"",""client_app_version"":""1.8.7"",""client_os_version"":""15.7.5""}"
52619,1728262818.87896,2024-10-06T18:00:18.878960-07:00,"additionalData is already an object, no need to parse it"
52620,1728262818.89577,2024-10-06T18:00:18.895770-07:00,"publishing cloud:push:notification with data {""additionalData"": {""coldstart"":false, ""foreground"":false,""content-available"":1}}"
52622,1728262818.91084,2024-10-06T18:00:18.910840-07:00,"Found silent push notification, for platform ios"
52623,1728262818.91707,2024-10-06T18:00:18.917070-07:00,"Platform is ios, calling handleSilentPush on DataCollection"
52626,1728262818.97652,2024-10-06T18:00:18.976520-07:00,handleSilentPush outside the try block = DataCollection1101510912
52627,1728262818.98613,2024-10-06T18:00:18.986130-07:00,"Before invoking launchTripEndCheckAndRemoteSync, id = DataCollection1101510912"
52628,1728262818.99385,2024-10-06T18:00:18.993850-07:00,"Received background sync call when useRemotePush = 1, about to check whether a trip has ended"
52629,1728262819.00184,2024-10-06T18:00:19.001840-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
...
52648,1728262819.7086,2024-10-06T18:00:19.708600-07:00,"allChecksPass = 0, but location permimssions pass, so one of the non-location checks must be false: [loc settings, loc permissions, motion settings, motion permissions, notification] = (
    1,
    1,
    1,
    1,
    0
)"
52649,1728262819.73244,2024-10-06T18:00:19.732440-07:00,"Found existing field {
    redirectParams =     {
        launchAppStatusModal = 1;
    };
    redirectTo = ""root.main.control"";
}, retaining existing"
...
52652,1728262819.77804,2024-10-06T18:00:19.778040-07:00,"In valid state STATE_WAITING_FOR_TRIP_START, nothing to do..."

[3] Not sure where the 19:06 is from, but otherwise, we seem to get values fairly consistently on the hour

$ grep "SILENT_PUSH in state" /tmp/loggerDB.check_silent_push.with_notification_off_overnight.withdate.log
51881,1728259162.64144,2024-10-06T16:59:22.641440-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
52629,1728262819.00184,2024-10-06T18:00:19.001840-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53096,1728266784.87332,2024-10-06T19:06:24.873320-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53151,1728269962.33908,2024-10-06T19:59:22.339080-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53199,1728269964.86586,2024-10-06T19:59:24.865860-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53259,1728273647.41162,2024-10-06T21:00:47.411620-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53314,1728277763.95109,2024-10-06T22:09:23.951090-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53369,1728281360.24779,2024-10-06T23:09:20.247790-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53424,1728284880.31137,2024-10-07T00:08:00.311370-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53479,1728288060.80447,2024-10-07T01:01:00.804470-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53534,1728291766.25226,2024-10-07T02:02:46.252260-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53589,1728295471.08587,2024-10-07T03:04:31.085870-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53644,1728299267.84832,2024-10-07T04:07:47.848320-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53699,1728302857.9972,2024-10-07T05:07:37.997200-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53754,1728306469.28447,2024-10-07T06:07:49.284470-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"
53809,1728309632.35278,2024-10-07T07:00:32.352780-07:00,"In TripDiaryStateMachine, received transition T_RECEIVED_SILENT_PUSH in state STATE_WAITING_FOR_TRIP_START"

I hearby declare this investigation done. We do continue to get silent push notifications even when the notification permission is turned off; that seems to mainly affect the visibility of notifications

@JGreenlee has already made the notification permission optional in the UI.

The next steps are to:

make it optional in the background checks as well (e.g. remove it from the code around allChecks: [loc settings, loc permissions, motion settings, motion permissions, notification] ) so that we don't annoy users by prompting them to "fix" an optional permission
- make sure to make it optional on both android and iOS
test that this continues to work on a fresh install in which the user does not give us the optional permission during onboarding (as opposed to removing permissions on an existing install).

@catarial can you get these changes done ASAP? We should then investigate and fix [1], but that is not a showstopper.

JGreenlee · 2024-10-07T15:30:30Z

make it optional in the background checks as well (e.g. remove it from the code around allChecks: [loc settings, loc permissions, motion settings, motion permissions, notification] ) so that we don't annoy users by prompting them to "fix" an optional permission

Is this necessary to unblock the release? If the user denies the notification permission, they won't see any "fix" prompts, right? How are users being "annoyed" in this scenario?

shankari · 2024-10-07T17:57:42Z

Good point. They won't see any "fix" prompts since notifications are turned off.

Couple of things to check;

we will still generate a redirection to the control page

52649,1728262819.73244,2024-10-06T18:00:19.732440-07:00,"Found existing field {
    redirectParams =     {
        launchAppStatusModal = 1;
    };
    redirectTo = ""root.main.control"";
}, retaining existing"

Given that they are not launching the app through the notification, I assume the redirection will not happen, but we should test.

I believe we will still generate the toast at the bottom of the screen; or is that from the UI as well

The permission checks are also modular, so the change is super simple. If @catarial can't get to it in the next couple of hours, I can just make the change, and push a release out to staging for testing.

JGreenlee · 2024-10-07T18:40:09Z

I believe we will still generate the toast at the bottom of the screen; or is that from the UI as well

The toast is from the UI (it's a SnackBar)

…hecker As part of e-mission/e-mission-docs#1094, we made the notification permission optional. Since it is not required for silent push notifications, notifications are nice-to-have. They are still helpful to ensure that people know if there are issues, and hopefully help them label their trips, but the app will still work if the permission is turned off. Since this is no longer required for normal operation, we don't need to check it in the background every hour, and nag the user to fix it if it is turned off. This is a super-easy change (two lines), so rolling this in while fixing e-mission/e-mission-docs#1094 instead of coming in as a future fix.

…hecker As part of e-mission/e-mission-docs#1094, we made the notification permission optional. Since it is not required for silent push notifications, notifications are nice-to-have. They are still helpful to ensure that people know if there are issues, and hopefully help them label their trips, but the app will still work if the permission is turned off. Since this is no longer required for normal operation, we don't need to check it in the background every hour, and nag the user to fix it if it is turned off. This is a super-easy change (two lines), so rolling this in while fixing e-mission/e-mission-docs#1094 instead of coming in as a future fix. Testing done: - Replaced the data collection plugin with the most recent version - Code compiles - Wasn't able to test in the emulator since with the permission turned off, without e-mission/e-mission-phone#1182 in place, we got a prompt to fix the notification, so we couldn't simulate a remote push Pushing this and testing all the changes together

shankari · 2024-10-07T23:03:56Z

Pulled and verified that:

on remote push

All settings valid, nothing to prompt
DEBUG: All settings valid, nothing to prompt

the lack of notification permissions is not a blocker to accepting permissions.
- I do think that there is a corner case here that is a bit awkward.
  - If the user allows notifications during onboarding, but then turns off the permission later from the setting screen, then the first time they launch the app after turning off, they will still see the permission screen. I think this is because the notifications_blocked is not set in the NSDefaults.
  - Both the buttons will be enabled, and if they choose to keep the notification permissions off, they can still hit "Accept". If they do that, then the permission screen will not show up in subsequent launches

I do think this is something that needs to be fixed, but it is a corner case, so we can handle it in the point release.
I will add a recording here of what that looks like once I have had a chance to push out this release to staging.

shankari · 2024-10-08T02:42:53Z

Here's the weird behavior with the popup. I think it only happens in the upgrade scenario - where the user had originally installed the app with the old onboarding flow, and then the app upgraded and they went and turned off the notification permission, and then launched the app.
https://github.com/user-attachments/assets/cc98a728-872d-493a-9e1c-c56dd2dd106f

I have also not tested similar scenarios on android.

shankari · 2024-10-09T06:54:22Z

Android seems to be similar to iOS; shows the notification if the app was upgraded but then the permission turned off, but not on subsequent launches

Screen.Recording.2024-10-08.at.11.51.18.PM.mov

shankari · 2024-10-09T18:42:14Z

Transferring discussion from Teams

@JGreenlee Even in the new onboarding flow, we would have requested the notification permission from the user, right?
when would we have consented = true but hasrequested = false?

If the user has been through onboarding, we can guarantee that notifications have been requested, but not necessarily granted

we already have a flag for "gone through onboarding" though, which is "isConsented"
why is that not good enough?

The original reason I added hasrequested was because iOS only lets you prompt via the system popup once - if you try to prompt again, nothing happens. They don't offer a way to check if we have already prompted, so I was keeping track of that state with hasrequested

(if we upgraded to the newer UNUserNotificationCenter API we would be able to check if we already prompted)

I think maybe the iOS native code needs the flag stored in UserDefaults like I had before, the UI needs to use isConsented in place of hasRequested, and the Android native code needs nothing

I think the mistake yesterday was in thinking that the UI needed to access hasRequested, when all we really needed was to know if onboarding has been done before (which the consent state will tell us)

the advantage of this is that when we migrate to UNUserNotificationCenter in the glorious future, the UI won't need to change. it will all be handled in the native code

@JGreenlee implemented this in e-mission/e-mission-data-collection#241

shankari · 2024-10-09T18:43:16Z

I think we will still need the change for setting HasRequestedNotificationPermission if isConsentedbecause otherwise the following case will break:

user installed 1.8.3 or earlier
user upgraded to 1.9.0+
user turned off notifications (this is OK and we won't prompt them again)
user changes their mind and wants to turn on notifications
user launches the "app status" from the profile screen
user clicks on the arrow next to notifications. HasRequestedNotificationPermission is not set so we will try to open the popup. But because the user turned it off from the settings, the popup will not show. It will look to the user like nothing is happening.

This is a two line change, so I am going to make that change.

If it is not set and the user has consented This fixes e-mission/e-mission-docs#1094 (comment)

shankari · 2024-10-09T22:29:36Z

That seems to have fixed it. I tried a couple of upgrade scenarios and they all worked.
Pushing version 1.9.1 with this.
@JGreenlee and @catarial can both of you poke at this a bit more today or tomorrow morning?
I will plan on submitting for review at around 9am PT tomorrow

catarial · 2024-10-10T15:52:40Z

Just tested on a short walk with notifications off. It was able to go from ONGOING_TRIP to WAITING_FOR_TRIP_START, but I'm not seeing the trip in the UI after force pushing.

catarial · 2024-10-10T16:03:35Z

Just tested on a short walk with notifications off. It was able to go from ONGOING_TRIP to WAITING_FOR_TRIP_START, but I'm not seeing the trip in the UI after force pushing.

It shows up now, I think it's just a delay with the server processing the trip.

shankari · 2024-10-15T23:35:57Z

This has been accepted by app store review now, so closing this issue.

FCM is doubling down on the "I'm going to change my API and break everything" approach. We made one round of fixes in: e-mission/e-mission-docs#1094 (comment) at which time the mapping to convert APNS tokens to FCM was working However, in the ~ 2 months since, that has also regressed, and we are now getting a 401 error with the old code. The new requirements include: - using an OAuth2 token instead of the server API key - passing in `"access_token_auth": "true"` as a header We already use an OAuth2 token to log in and actually send the messages ``` DEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443 DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): fcm.googleapis.com:443 ``` So it seems like it would be best to just reuse it for this call as well. However, that token is retrieved from within the pyfcm library and is not easily exposed outside the library. Instead of retrieving the token, this change retrieves the entire authorization header. This header includes the token, but is also formatted correctly with the `Bearer` prefix and is accessible through the `requests_session` property. With this change, the mapping is successful and both silent and visible push notification are sent to iOS phones. Before the change: ``` DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iid.googleapis.com:443 DEBUG:urllib3.connectionpool:https://iid.googleapis.com:443 "POST /iid/v1:batchImport HTTP/1.1" 401 None DEBUG:root:Response = <Response [401]> Received invalid result for batch starting at = 0 after mapping iOS tokens, imported 0 -> processed 0 ``` After the change ``` DEBUG:root:Reading existing headers from current session {'User-Agent': 'python-requests/2.28.2', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Authorization': 'Bearer ...'} DEBUG:root:About to send message {'application': 'gov.nrel.cims.openpath', 'sandbox': False, 'apns_tokens': [.... DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iid.googleapis.com:443 DEBUG:urllib3.connectionpool:https://iid.googleapis.com:443 "POST /iid/v1:batchImport HTTP/1.1" 200 None DEBUG:root:Response = <Response [200]> DEBUG:root:Found firebase mapping from ... at index 0 DEBUG:root:Found firebase mapping from ... at index 1 DEBUG:root:Found firebase mapping from ... at index 2 ... ``` Visible push ``` ... s see if the fix actually worked" -e nrelop_open-access_default_1hITb1CUmGT4iNqUgnifhDreySbQUrtP WARNING:root:Push configured for app gov.nrel.cims.openpath using platform firebase with token AAAAsojuOg... of length 152 after mapping iOS tokens, imported 0 -> processed 0 combo token map has 1 ios entries and 0 android entries {'success': 0, 'failure': 0, 'results': {}} Successfully sent to cK0jHHKUjS... {'success': 1, 'failure': 0, 'results': {'cK0jHHKUjS': 'projects/nrel-openpath/messages/1734384976007500'}} ```

JGreenlee mentioned this issue Oct 6, 2024

🍎 iOS Permissions Changes to Comply with iOS Guidelines e-mission/e-mission-data-collection#237

Merged

shankari mentioned this issue Oct 7, 2024

🔥 Single-line changes to remove notification checks in the background checker e-mission/e-mission-data-collection#238

Merged

shankari added a commit to shankari/e-mission-data-collection that referenced this issue Oct 9, 2024

Hack to set the HasRequestedNotificationPermission

3a7b6b4

If it is not set and the user has consented This fixes e-mission/e-mission-docs#1094 (comment)

shankari mentioned this issue Oct 9, 2024

Hack to set the HasRequestedNotificationPermission e-mission/e-mission-data-collection#242

Merged

shankari closed this as completed in e-mission/e-mission-data-collection@7b0120f Oct 9, 2024

shankari reopened this Oct 9, 2024

shankari closed this as completed Oct 15, 2024

shankari mentioned this issue Dec 16, 2024

🛂 Fix authorization while mapping APNS tokens to FCM e-mission/e-mission-server#1002

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compliance with iOS Permissions Guidelines #1094

Compliance with iOS Permissions Guidelines #1094

JGreenlee commented Oct 3, 2024 •

edited

Loading

JGreenlee commented Oct 3, 2024 •

edited

Loading

shankari commented Oct 3, 2024

shankari commented Oct 3, 2024 •

edited

Loading

catarial commented Oct 3, 2024

catarial commented Oct 3, 2024

catarial commented Oct 3, 2024

shankari commented Oct 3, 2024

catarial commented Oct 4, 2024

catarial commented Oct 4, 2024

JGreenlee commented Oct 4, 2024

JGreenlee commented Oct 4, 2024

shankari commented Oct 4, 2024 •

edited

Loading

catarial commented Oct 4, 2024

shankari commented Oct 4, 2024

shankari commented Oct 5, 2024

shankari commented Oct 5, 2024 •

edited

Loading

JGreenlee commented Oct 6, 2024

JGreenlee commented Oct 6, 2024 •

edited

Loading

shankari commented Oct 6, 2024 •

edited

Loading

shankari commented Oct 7, 2024 •

edited

Loading

JGreenlee commented Oct 7, 2024 •

edited

Loading

shankari commented Oct 7, 2024

JGreenlee commented Oct 7, 2024

shankari commented Oct 7, 2024

shankari commented Oct 8, 2024

shankari commented Oct 9, 2024

shankari commented Oct 9, 2024

shankari commented Oct 9, 2024

shankari commented Oct 9, 2024

catarial commented Oct 10, 2024

catarial commented Oct 10, 2024

shankari commented Oct 15, 2024

Compliance with iOS Permissions Guidelines #1094

Compliance with iOS Permissions Guidelines #1094

Comments

JGreenlee commented Oct 3, 2024 • edited Loading

JGreenlee commented Oct 3, 2024 • edited Loading

shankari commented Oct 3, 2024

shankari commented Oct 3, 2024 • edited Loading

catarial commented Oct 3, 2024

catarial commented Oct 3, 2024

catarial commented Oct 3, 2024

shankari commented Oct 3, 2024

catarial commented Oct 4, 2024

catarial commented Oct 4, 2024

JGreenlee commented Oct 4, 2024

JGreenlee commented Oct 4, 2024

shankari commented Oct 4, 2024 • edited Loading

catarial commented Oct 4, 2024

shankari commented Oct 4, 2024

shankari commented Oct 5, 2024

shankari commented Oct 5, 2024 • edited Loading

JGreenlee commented Oct 6, 2024

iOS 13.3

iOS 15.6

JGreenlee commented Oct 6, 2024 • edited Loading

shankari commented Oct 6, 2024 • edited Loading

shankari commented Oct 7, 2024 • edited Loading

[1] FSM gets stuck in `WAITING_FOR_TRIP_STATE if settings are wrong

[2] Receiving push notification when notification permissions are turned off

[3] Not sure where the 19:06 is from, but otherwise, we seem to get values fairly consistently on the hour

I hearby declare this investigation done. We do continue to get silent push notifications even when the notification permission is turned off; that seems to mainly affect the visibility of notifications

JGreenlee commented Oct 7, 2024 • edited Loading

shankari commented Oct 7, 2024

JGreenlee commented Oct 7, 2024

shankari commented Oct 7, 2024

shankari commented Oct 8, 2024

shankari commented Oct 9, 2024

shankari commented Oct 9, 2024

shankari commented Oct 9, 2024

shankari commented Oct 9, 2024

catarial commented Oct 10, 2024

catarial commented Oct 10, 2024

shankari commented Oct 15, 2024

JGreenlee commented Oct 3, 2024 •

edited

Loading

JGreenlee commented Oct 3, 2024 •

edited

Loading

shankari commented Oct 3, 2024 •

edited

Loading

shankari commented Oct 4, 2024 •

edited

Loading

shankari commented Oct 5, 2024 •

edited

Loading

JGreenlee commented Oct 6, 2024 •

edited

Loading

shankari commented Oct 6, 2024 •

edited

Loading

shankari commented Oct 7, 2024 •

edited

Loading

JGreenlee commented Oct 7, 2024 •

edited

Loading