Status 400 and reason BadDeviceToken

[halavins]

I've updated the parse-server to the latest 2.5.3 version and parse-server-push-adapter v 2.0.0 and now my error logs are full of these messages:

3|parse-wr | node-pre-gyp
3|parse-wr |  
3|parse-wr | ERR!
3|parse-wr |  
3|parse-wr | parse-server-push-adapter APNS
3|parse-wr |  APNS error transmitting to device 28c89bdc6a0e7a6454fb0cc446229a0841eecdea955533d3c549a658bb15fc98 with status 400 and reason BadDeviceToken

There are so many of them that I can't see anything else.

The questions are:

1. How can I ignore these errors in my pm2 config file?
2. Is there a way to remove bad device tokens automatically from Installation Collection?

[flovilmart]

How can I ignore these errors in my pm2 config file?

There is no particular way to ignore those, they are produced when running with VERBOSE

Is there a way to remove bad device tokens automatically from Installation Collection?

For now there isn't

[halavins]

Hey @flovilmart, my VERBOSE setting is disabled, but my logs are filled with these errors by 90%.

How do I turn them off? The fact that servers show and write these errors damage the performance.

[flovilmart]

After looking into the repo, the error is reported by the logger as an error log, you won’t be able to disable it at that time.

[miguel-s]

Hi @flovilmart, we are running our parse server with logLevel: 'none', but we are still getting these errors logged since we upgraded to 2.5.3, is this intended?

[halavins]

Looks like they just forgot to set the right setting for these errors.

I've downgraded to an earlier version.

[halavins]

Any news on this issue? Is there a way to switch the Push errors off?

Or is there a way to automatically erase all Installations with Bad Device Tokens?

[funkenstrahlen]

As far as I know in the latest version there is a way to automatically delete bad tokens.

[halavins]

Do you know any adequate way to find the bad device tokens and erase them?

We have a bunch of them, and they only increase in numbers, especially the ones for Androids. They never self-delete even if the app is reinstalled.

[smiley-yoyo]

Hi @funkenstrahlen can you please tell the way to automatically delete bad tokens? thanks a lot.

I have figured it out. After release 2.6.3, you can automatically delete bad tokens by setting the PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS environment variable = 1.
I have tested it on 2.6.5 and pass.

Still thanks a lot to @funkenstrahlen for giving the hint.

[halavins]

@wicked-wei , our tests for this PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS option showed that:

1. even the VALID iOS deviceTokens are erased for some reason
2. the old, invalid, unused android deviceTokens are not erased at all

[smiley-yoyo]

@halvini thanks for your comment. I haven't seen the problem you mentioned yet but I will keep eyes on it.
And, the code that PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS option works is at parse-server/src/StatusHandler.js, you can check it out for more detail.

[flovilmart]

@halvini can you please point out what part of the logic there is not working as expected?

[halavins]

@flovilmart , the issue is described in this ticket:
parse-community/Parse-SDK-iOS-OSX#1224

We are 100% sure that PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS flag cleans up even VALID ios installations. Why? I don't know. I've checked the StatusHandler, I've checked the APNS.js. I don't see anything unusual out there, but I believe that when you send many many push notifications to different installations, probably APNS returns BadDeviceToken error even for valid installations. It doesn't happen immediately. For instance, my own valid installation was unset by this flag in a week after I turned it on. So, probably, if there is a timeout, or any other connection or delivery error, the APNS thinks that device is either Unregistered, or Bad.

[smiley-yoyo]

@halvini you can also check that if there is an unexpected mixed using of both apple sandbox and production push on a same environment(or database). This may cause unexpected deletion of deviceToken.

[halavins]

@wicked-wei the sandbox was used by 20 beta-testers only.

When we set PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS flag, nothing bad happened at first, but then in 3 days we started getting complaints from both sandbox users and production users. Why were the production users affected by this flag?

[smiley-yoyo]

@halvini because sandbox and production use different deviceToken. If any chance you send a production deviceToken to a sandbox apns server, it will report that token is BadToken. Reverse is the same.

[halavins]

@wicked-wei ok, so why did production users lose their deviceToken data, if we send push notifications from cloud code only? The parse-server config has both certificates installed:

    "PARSE_SERVER_PUSH": {
      "android": {
        "senderId": "*****",
        "apiKey": "*****"
      },
      "ios": [
        {
          "pfx": "*****/developmentPush.p12",
          "topic": "*****",
          "production": false
        },
        {
          "pfx": "*****/productionPush.p12",
          "topic": "****",
          "production": true
        }
     ]
    },

[smiley-yoyo]

@halvini the config looks ok, but I am not sure if parse server can tell a installation deviceToken is production or sandbox. For the js documents does not mention that, maybe it will send a token to both production or sandbox? I am not sure because I never do that.
Oh, I forgot that. The installation has 'appIdentifier' field to tell deviceToken belongs app. So If you config two pfx with different topic, that will be Ok.

[halavins]

@wicked-wei , this is all quite informative, but it doesn't resolve the issue of undefined deviceTokens for valid ios installations.

Your idea is to change the development certificate to a new bundle id, am I right? How can I be sure that that's the reason? I can't risk my production environment again. I've lost too much data when I set the PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS flag. It worked for a few days only but resulted in a huge mess.

[halavins]

Did anyone find a solution to this issue?

I'd like to set the PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS to 1 without losing valid iOS installations.

[alexblack]

Any update here? is PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS safe to use? Are there docs on it?

[jeffreyjackson]

im interested in this as well. just for the sake of keeping logs clean

[flovilmart]

@alexblack @jeffreyjackson the only way to know if it's safe to use is for you guys to use it and report your findings.

[alexblack]

Well, there are some findings above, sounds risky, we'll find another way.

[flovilmart]

we'll find another way.

That's not really helpful as the gate behind the feature flag is for the community to experiment. I'd rather revert the PR / feature if no one is willing to experiment with us.

[alexblack]

Fair point. I should share my other feedback - also not too excited about the feature in general, if device tokens are erased then it seems to make it harder to distinguish installations with no device token vs ones that were erased.

This combined with the lack of real errors/transparency from parse push means we'll eventually just integrate with apple and google, then we'll have real errors back, and we can just mark installations as unreachable when we learn that from the source.

Also as we've reported parse-dashboard doesn't display past pushes for us. So we're not really having a great experience here.

[flovilmart]

also not too excited about the feature in general, if device tokens are erased then it seems to make it harder to distinguish installations with no device token vs ones that were erased.

This is what you were expecting originally, that this feature is working correctly, not sure where you're coming from with that.

This combined with the lack of real errors/transparency from parse push means we'll eventually just integrate with apple and google, then we'll have real errors back, and we can just mark installations as unreachable when we learn that from the source.

I'm open for improving the project / provide a better feature but without community feedback this is unfortunately impossible.

Opensource doesn't mean we the maintainers have to take all the risks and responsibility.

The community pressed to have a cleanup invalid token implemented, IIRC you were part of it, I spent a shit load of hours on it, a tentative implementation has been available for a while and now you're pressing again today to have something else implemented. Do you understand how unfair, unethical and unwelcome those commentaries are?

[alexblack]

Sorry. Just sharing my feedback. :(

[alexblack]

Could we not discuss the requirements up front? If that new field (as I described it) is not what you'd like to see, then best to figure that out now I think.

[flovilmart]

in all those discussions you would have had 10 times the time to implement this and run it, and see if it works for your use case. My gut feeling tells me that it will take you multiple iterations and days running in production to see if this works properly.

[alexblack]

For me, I think it'd make sense for us to have some tentative agreement on what the new field in _PushStatus looks like before getting started.

(Also, we can't get started right away regardless...)

[flovilmart]

We have an agreement that we want to track errors coming back from the push adapter into the _PushStatus object, so a user can at a later time identify which installation has an invalid deviceToken.

I believe this is clear enough as a description, and his gives you the freedom to implement it the way you feel is best. Also if uou’re Not able to start it, either immediately or in a foreseeable future, you don’t really need that feature, nor to discuss it this much is such short amount of time. Again I could be mistaken, but it feels nothing has much moved forward and we’ve both wasted our time on words rather than code, given that what you expect would pretty much take only a few minutes to write and about an hour of adding tests.

[alexblack]

I hear you.

For me, if we had tentative agreement on what the field looked like in _PushStatus that would really help. Without that, I worry that a PR would get rejected or require significant rework if it missed the mark on what this field should look like.

[flovilmart]

And so what if the PR is rejected or require rework? Does it never happen that at code review a better solution arises? Or that when implementing you realize things don’t fit the pre agreed mold? What if the solution is just to write a properly structured log because it works better for you? Wouldn’t it be a waste? Do you believe the time we spend here is not more wasteful than the time it would take to implement, and rework the said PR?

[alexblack]

I thought if we chatted briefly for a few minutes we'd land on what we wanted to see in _PushStatus (even knowing it might change when talk changes to code...) Seemed worth it to me. But I can appreciate if that doesn't seem worth it to you.

[flovilmart]

@halvini please update to the latest version of parse-server that now uses the HTTP/2 interface to APNS. It should yield significantly less logs and open a new issue if it persists.

@alexblack, I stated my point loud and clear, and you should probably take what I say and work from it. I am not against your proposal, and waiting for an implementation as well as suffisant testing on your side that would ensure we’re not doing all of that for nothing.

[alexblack]

@flovilmart if we had some tentative agreement on what we'd like to see in _PushStatus it would give me the confidence I needed to proceed.

If we can't get agreement there, I'd still hope we can contribute to this great project somehow. Its definitely a huge help for us.

[flovilmart]

Please stop it. What are you expecting? That i’ll Sign off right now on your prospective PR because we agreed on the naming? Ok the name is approved! Now if you want to implement it go ahead!

[alexblack]

What are you expecting?

{
  "resultsByInstallationId": {
    "VvNFL12st4": {
      "succeeded": true,
      "result": {
        "message_id": "0:1526665330066787%b4783826f9fd7ecd"
      }
    },
    "LBsWhDeAMn": {
      "succeeded": false,
      "result": {
        "error": "NotRegistered"
      }
    }
  }
}

I was hoping for some feedback that this looked like what you wanted, or, it didn't.

[flovilmart]

Go ahead with that, and run in on your infra for a week

[halavins]

@flovilmart thanks for the response!

Guys, please don't shoot each other :)

[pcg92]

same error here
backend_1 | node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device 7bc3dc72efe2538299e8c775ds58dd0a9a8e8b50794d2 with status 410 and reason Unregistered backend_1 | node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device 5b7c2580e1d0bd89d5399338b2c470c0c37c68b3862fc7 with status 400 and reason BadDeviceToken

[flovilmart]

@pcegarra ensure you are using the latest parse server, and the device token is actually valid for your certificate pair.

[pcg92]

@flovilmart
Im sending notifications to all the users but when this error is raided the notification job is stopped.
This is actually my code trying to fix this error:

try{
			Parse.Push.send({
				where: installationsQuery, 
				data: {
					alert: message,
					badge: 1,
					sound: 'default'
				}
			},{useMasterKey:true})
			.then(function() {
					promise.resolve();
			}), function(error) {
					promise.resolve();
			};
		}
		catch(ex){
			promise.resolve();
		}
		return promise;

Any idea what can I do?
I have an android app and two differents ios apps with diferents bundle id example:
com.myapp.admin
com.myapp.client

so my server config looks like:

ios: [ { passphrase: process.env.PUSH_IOS_PASSPHRASE, pfx: process.env.PUSH_IOS_CERTFILE_DEV, topic: process.env.PUSH_IOS_BUNDLE_ID, production: false }, { passphrase: process.env.PUSH_IOS_PASSPHRASE, pfx: process.env.PUSH_IOS_CERTFILE, topic: process.env.PUSH_IOS_BUNDLE_ID, production: true }, { passphrase: process.env.PUSH_IOS_PASSPHRASE, pfx: process.env.PUSH_IOS_CERTFILE_DEV_2, topic: process.env.PUSH_IOS_BUNDLE_ID_2, production: false }, { passphrase: process.env.PUSH_IOS_PASSPHRASE, pfx: process.env.PUSH_IOS_CERTFILE_2, topic: process.env.PUSH_IOS_BUNDLE_ID_2, production: true },

Could be this the problem?

[flovilmart]

Can you provide the logs when running with VERBOSE=1? Also the state of the associated _PushStatus object for this push?

[pcg92]

Ok, im going to look with verbose=1, but i dont know how to see the state of the associated _PushStatus object..
Can you explain me it with other words please? thanks!

[flovilmart]

Make a query to the _PushStatus table and get the most recent object. Or with the verbose logs, you should have the objectId associated with it.

[pcg92]

Parse verbose log:

backend_1 | verbose: _PushStatus a47E2wHCoa: sending push to installations with 1 batches backend_1 | verbose: Sending push to 2 backend_1 | verbose: Sending push to 1 backend_1 | verbose: Sending push to 1 backend_1 | verbose: Sending push to 2 backend_1 | verbose: Sending push to 1 backend_1 | verbose: _PushStatus pdEUFObZ0g: sent push! 0 success, 1 failures backend_1 | verbose: _PushStatus pdEUFObZ0g: needs cleanup devicesToRemove=[APA91bHrIxaqZ2mVv33U89Hsk9mzPQk1d2s_rN2Fw59EhmSU9hAJqgHcw3M2u1MFc3s-IgiYyhmk0Cw-ZWNj6n9g5ljzZGmHrTwBipYLtnNgnRAswc_a7IcC1_7aj0GFRk9cGVod2Sq9] backend_1 | verbose: _PushStatus ztbkBDkM1z: sent push! 0 success, 1 failures backend_1 | verbose: _PushStatus ztbkBDkM1z: needs cleanup devicesToRemove=[dG6GAAOXF_c:APA91bG7vKuW58ni0p3uWnSCEyZn_ZxqRrmG6FSZE3SZrRr9vNnIRwBAff9tZLwvgXbaKwTjoo__qdSdV1xQeajzY3TvVqtsB-NpyV7rhTeccMqAcg8cyw5UJynw_Dw7YqcseBITKGkY] backend_1 | verbose: _PushStatus 44XCEpCDpc: sent push! 1 success, 1 failures backend_1 | verbose: _PushStatus 44XCEpCDpc: needs cleanup devicesToRemove=[cFcF8b-oBEw:APA91bF3aftS13tRQduyF1lUvCr9pcRIV4h1E0J_RrQOKpKOBB9JlxI70MLaH2-EMP0Oh6Oh5iw-swxzF0v80KzfGFH7cUe0X-8wyOaiV1Mrz0sy-izI3HYEQvDRGJ0RNfvdoN6eU1hW] backend_1 | verbose: _PushStatus MHGZ2UPH2F: sent push! 2 success, 0 failures backend_1 | verbose: _PushStatus MHGZ2UPH2F: needs cleanup devicesToRemove=[] backend_1 | verbose: _PushStatus a47E2wHCoa: sent push! 1 success, 0 failures backend_1 | verbose: _PushStatus a47E2wHCoa: needs cleanup devicesToRemove=[] backend_1 | verbose: _PushStatus KKcNQReTQJ: sent push! 1 success, 1 failures backend_1 | verbose: _PushStatus KKcNQReTQJ: needs cleanup devicesToRemove=[finju-C12RI:APA91bGtTbGBXfoDegjganBKoBf_Z8mNo_A-dO18JdCudyQTOKk7qKmtuxpRvbRYG5qa_6cw7IvOGP2AYL1NI6eVvywjDgaUgM9MDsd0wfdxY85zveZ--FkphO1vlcI2xeXg0-J78jTW] backend_1 | node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device 7bc3dc72efe2538299e8c7755fdf1a20ca01073eea32b58dd0a9a8e8b50794d2 with status 410 and reason Unregistered backend_1 | verbose: _PushStatus tGmYrDZiaY: sent push! 0 success, 1 failures backend_1 | verbose: _PushStatus tGmYrDZiaY: needs cleanup devicesToRemove=[7bc3dc72efe2538299e8c7755fdf1a20ca01073eea32b58dd0a9a8e8b50794d2] backend_1 | node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device dc5dd60549ab890a42f3afb1d47b99d766d2fd8b84ba67152a55b1afcc839b69 with status 410 and reason Unregistered backend_1 | verbose: _PushStatus KiPDDNApW3: sent push! 2 success, 1 failures backend_1 | verbose: _PushStatus KiPDDNApW3: needs cleanup devicesToRemove=[dc5dd60549ab890a42f3afb1d47b99d766d2fd8b84ba67152a55b1afcc839b69]

Query id for KiPDDNApW3
{ "results": [ { "objectId": "KiPDDNApW3", "pushTime": "2018-05-26T12:23:40.842Z", "query": "{\"restaurant\":{\"__type\":\"Pointer\",\"className\":\"Restaurant\",\"objectId\":\"6FgXTZOtI0\"}}", "payload": "{\"alert\":\"¿Que hay de comer en \\\"Tayo’s\\\"?\",\"badge\":1,\"sound\":\"default\"}", "source": "rest", "status": "succeeded", "numSent": 2, "pushHash": "ed5d0f1208f198fc8d337b2105a53b28", "createdAt": "2018-05-26T12:23:40.842Z", "updatedAt": "2018-05-26T12:23:43.183Z", "numFailed": 1, "sentPerType": { "ios": 2 }, "failedPerType": { "ios": 1 }, "ACL": {} } ] }

[flovilmart]

From what I can see, everything is fine for that push. In your logs we see at least 3 different pushs intertwined, but, the one with the push status look good, as the expected number of pushs is 3 and the total send / failed is 3 as well

[mtrezza]

@flovilmart I've tested PARSE_SERVER_CLEANUP_INVALID_INSTALLATIONS with development certificates in APNS sandbox and it worked just fine.

It always takes me about 5 remote push notifications within 5 minutes until APNS reports a failed push and the bad device token gets removed. It worked several times, the good device tokens untouched, the bad ones removed.

Parse server push configuration:

push: {
        ios: [
            {
                pfx: 'apns_dev.p12',
                topic: 'com.myDomain.myApp',
                production: false
            }
        ]
    },

Tested with:
parse-server v2.7.4 and v2.8.1

@halvini's case sounds like some kind of mix-up of production and development APNS certificates / device tokens. Note that my parse server configs only includes one APNS certificate per server.

Looking at the code I suspect the following cause:

Each push certificate creates a provider. The list of providers is sorted by production certificates before development certificates. When sending a push notification, the production provider is tried first, if it fails the development provider is used. Now if a push using an APNS production device token that is sent through the production provider fails for any reason, the push is sent through the development provider, which will identify it as a bad device token. This leads to the removal of the token in parse server because it has been sent to the wrong environment. So in @halvini's case the APNS production server could simply have been unreachable for a brief moment, 3 days after activating the clean-up flag, causing the removal of the good device tokens.

If that is the cause of the problem, a solution could be to add a production: Bool field to the Installation document to flag the device token as either production or development. The push sending logic then has to be changed to send production tokens only to production providers and development tokens only to development providers.

Another solution would be to simply remove the possibility of adding development certificates to parse server in a production environment, i.e. remove the production key from the parse push configuration. A development / testing environment should be sandboxed from any production environment anyway. This would be my preferred solution.

[mtrezza]

@flovilmart, this issue is closed, but can you confirm or dismiss my analysis above?

[flovilmart]

Yeah it’s possible that the errors reported when using multiple certificates in production may be problematic. I’ll reopen while we find a proper workaround

[allanberrocal]

Hi, I'm having the issue when sending push notification to iOS.
Error

node-pre-gyp ERR! parse-server-push-adapter APNS APNS error transmitting to device ...f6r4... with status 400 and reason BadDeviceToken

We have the latest version of parse. One production certificate, no dev.
Push notifications worked well for iOS devices for a day, but not afterwards. Not even new installations of iOS are not receiving the notifications.

We were using the token-based authentication (w/ .p8 file) when Push stopped working. Them moved to signing certificate (w/ .p12 file), but with no better luck. Anyone is having or has had this issue? Ideas?

Thank you

[davimacedo]

Can you please open a new issue and fill out the template?