Ensure that data is transmitted as UTF-8 #333
Comments
|
I followed the FAQ https://github.com/e-mission/e-mission-docs/blob/master/docs/e-mission-server/troubleshooting_tips_FAQ.md#confirm-that-the-entries-are-really-in-the-database and find 0 for len(data_list), which confirms what I've seen in mongo. |
|
I've run the intake pipeline, there are no apparent errors except at the beginning : |
|
another clue perhaps, there is only one intake_single.log file and an empty intake_single_error.log file, no other intake_.log files in /var/tmp (we use the default sample conf for intake logs) and there is not any data after February 7. I've run the intake pipeline several times manually since Feb 7, I can't find where the corresponding logs are... |
|
couple of quick checks:
|
|
Are you sure you didn't change the settings in
Does the |
yes I am sure, the reason is not a good one, the was modified by hand in intake-multiprocess.py so as to redirect the logs to another file. We will correct this.
Yes /usercache/put list entries but for other users, for instance: |
|
I looked at the scripts in the bin directory, there is a debug/fix_usercache_processing.py script, do you think it is worthwhile to execute it? Also I wanted to execute the delete_user.py script, but there is |
|
I think you should first see if your data is getting to the server correctly because as I said, I saw connection errors in the phone logs that you sent earlier. Was the earlier |
|
The |
|
I cheked again and confirm, there is no other occurrence of usercache/put for my UUID, except when the server receives stats/server_api_time data ; no timeserie data is transmitted when I look at the log file. |
|
so then your data is not making it from the phone to the server. You can mail your phone logs to yourself and look at them, or browse the log directly on the phone (Profile -> Check log). If you "Force sync" and then look at the logs on the phone, you should get a quick sense of whether the sync is succeeding or not. |
|
Ok, thanks ! |
|
Quick check: have you pulled the geofence changes and switched back to your own apk? If you are still on emTripLog with the interscity channel, then your data is coming to my server. Check the host that you are connected to (Profile -> Developer Zone -> Sync -> host). Also, just to clarify, you can browse logs also in Profile -> Developer Zone -> Check Logs. I forgot to put "Developer Zone" in #333 (comment) |
|
the host is http://145.239.101.212:8080 , so I suppose it is the good one. |
|
in the phone log, there is an ERROR serverSyncAdapter IO Error ... connection refused while posting converted trips to JSON |
|
so you are having some issues with connectivity?! are you able to access that webserver from (say) the browser on the phone? |
|
yes i can see the web pages when entering the URL in firefox. and as i said
at least 2 other users could send their trip data to mongodb
Le ven. 8 mars 2019 18:25, shankari <[email protected]> a écrit :
… so you are having some issues with connectivity?! are you able to access
that webserver from (say) the browser?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#333 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEzlV8-m9xpqEQC0TRG0yLAPll3mqRaMks5vUpz3gaJpZM4bjN7C>
.
|
|
I would just follow standard debugging steps:
There's a lot of potential causes for the server being inaccessible on stackoverflow (e.g. no |
|
Ok, thank you! I will try to understand what is wrong. |
That is just for the javascript local storage and the native database plugin
Yeah this seems to indicate that you are connecting to the server but getting an error. As we already discussed, some aggregate calls go through |
|
thanks! I will try with Yann on Monday, he knows Android Studio and so the debugger too, I guess. |
|
Thanks to some interactive debugging and looking at the webserver logs, we now know the issue has to do with encoding février at some point along the collection and sync path. |
|
let's try to debug where we are not using UTF-8 along the data collection and sync path. First, is the phone saving the data correctly as UTF-8? If you look at the |
|
hi, in the sqlite usercachedb, I have for instance I don't know how to check the encoding but the "é" string is shown (correctly) as "é" not as "\xe9" |
|
so this seems overall consistent with the other stuff you have noticed. basically, the python implementation is not UTF-8 consistent, at least on your server. I am going to manually set the date/time and locale in the emulator and see if this is an overall issue. |
|
ok, so I tried to reproduce this in the emulator and it is not working because the location points are sent to the emulator by the laptop and that is still set to March 26.
|
|
I really don't want to reset my laptop datetime to Feb. Let's try on a physical device instead... |
|
If that still doesn't work, I'll have to try some other python scripts because it looks like we won't be able to reproduce this until August (Août) |
|
so in python3, JSON is supposed to be represented in UTF-* https://docs.python.org/3.6/library/json.html#character-encodings
other encodings are supported when Note that the body is in fact in bytes (this is the informal string representation starting with bottle does support python3 |
|
ok so because JSON requests are supposed to be in UTF-8 Note that even the standard #333 (comment) says that we should use latin-1 only for So the real problem appears to be that the app is encoding the data in There are two possible fixes:
The principled fix is better because ISO-8859 doesn't even support the full latin alphabet (https://en.wikipedia.org/wiki/ISO/IEC_8859-1#Languages_with_incomplete_coverage), let alone other languages. |
|
I will do a quick verification that the cheap fix will in fact work, though, since we may run out of time for the principled fix. |
|
Hm, I am a little confused. If I try to generate the iso-8859 string file by encoding the utf-8 one The file still shows the |
|
ok, that's just vim being smart and figuring out the encoding automatically. If I use |
|
ok, so the previous When I try to load it directly, I still get an error but if I try to load it with the proper encoding it works fine and the bottle trick of decoding and encoding also works So the following one line fix is a cheap fix that we can fall back to... |
|
Tried it with force sync.
and in fact, the data is in the database |
|
ok, so the server communication is in drum roll (https://github.com/e-mission/cordova-server-communication). Looking more closely at the code, the only method that is really called externally (from
And yes, the default encoding is 8859-1 https://hc.apache.org/httpclient-legacy/charencodings.html
We can set the charset in the content-type header
But we currently don't. So it sounds like the simple fix is to just set it there :) |
|
We should also seriously consider upgrading this plugin. Even the apache HTTP components are currently at 4.5, and the legacy version that we are using is EOL. And there are new android-specific options such as volley. However, this is not the biggest priority right now (as it wasn't 3 years ago e-mission/cordova-server-communication@0ff1f05#diff-b45f027217ab8fcefc52d843dfc2a781) so will punt again. But if it turns out that we touch this code more frequently (e.g. every 6 mos or so), we should revisit that decision. |
|
Retrying with the fix; again, we have a mix of entries, but we do have some with We revert the workaround and force sync aaaannnndddd... the fix didn't work. on the phone |
|
so it turns out that that in addition to the HTTP headers, we also need to set the charset of the string entity that we use to generate the string body of the HTTP request. Changing all the instances of seems to fix the problem. On the phone on the server |
|
@PatGendre since you are building your own app, do you want to try out the fix first? Alternatively, I can just check this in and bump up the version number, so when you rebuild, you will get the fix. I'm going to wait a bit to push a new emTripLog release with this fix because the Indian team is starting their survey; they just finished the pilot testing and they are going to be paranoid about any change to the app. And this fix doesn't really affect them. Let me know if you have any concerns. |
|
@PatGendre Great catch BTW. This was such a corner case if you consider English and Spanish-speaking countries, but such an important component if you consider the rest of the world, specially countries with non-latin alphabets. |
shankari
changed the title
shankari
changed the title
|
Let's briefly check to see whether there is a similar issue with iOS. Here's where we communicate with the server in iOS. and it's true that we don't set UTF-8 as the encoding. But to fill in the data, we use and that will return UTF-8
So I should put in the @PatGendre are any of you testing with iPhone just to confirm? |
|
Hi @shankari , great debug job :-) Internationalisation is definitely an art! |
|
Testing this assumption (#333 (comment)) on iOS before I finalize this fix:
|
|
I have to test this on a physical phone, since the iOS emulator doesn't appear to allow you to set the date and time. After setting this, deploying the phone and manually starting a trip, I got locations whose dates were displayed in French while logging. However, when the trip ended and the data was synced, the fmt_time sent to the server did not have any words, so no accents would apply. So I will set this to UTF-8 anyway, but I don't think it is an issue on iOS. |
|
@PatGendre were you able to test with the patch? I actually feel pretty comfortable releasing these changes based on my testing, and then you can just release the plugin. But once you have built with them, it would be good if you could reset your phone and test before August. If there is something subtly broken, I don't want to find out once it is already broken and then scramble to fix it. |
|
on iOS, changed to content header to be UTF-8 anyway and tested to ensure that there was no regression. There was not. on the phone on the server |
On android, unless we explicitly serialize the data as UTF-8, it is serialized as ISO-8851. On iOS, it is serialized as UTF-8 by default. In both, it is probably best to explicitly mark the content as UTF-8 in the HTTP header Bump up the version number to match This fixes e-mission/e-mission-docs#333 Testing done: - on android: set the locale to French, set the date to Fevr and synced data - before this change, it did not work - after this change, it works - on iOS, set the locale to French, set the date to Fevr and synced data - the formatted data did not have any words so even before this fix, it worked - this did not cause a regression
I could not find a similar issue so I open this one:
I have recorded tracks for the last few days, but no data is visible in mongo, except stats/server_api_time. There is no background/location nor background/filtered_location. And therefore no analysis data either.
I checked the server log, for 2 other users there is some location data in mongo.
In the server log, I see 2 kinds of error :
I looked at the phone logs but I don't what to look for, I send it to you (12MB) in case it is useful and you can have a look...
Thanks if you see what could explain this problem !
dblogs-7mars.tar.gz
The text was updated successfully, but these errors were encountered: