Importing dictionaries stuck on validation step

[Nyuutrino]

Description
When importing/downloading dictionaries, they will get stuck on the validation step, stopping at a random progress point (sometimes 0%, sometimes 17%, etc). This happens on both my Windows & Arch Linux installations, which are both on the same computer.

Browser version
Firefox (134.0.1 (64-bit)

Yomitan version
25.1.5.0

[djahandarie]

@Nyuutrino We've had some reports of this but so far none of the devs have been able to replicate it, so it's hard for us to fix. Would you happen to have programming background?

If you could go to about:debugging#/runtime/this-firefox and hit "Inspect" on the Yomitan extension, as well as all the "Shared Workers" as well as "Other Workers" which start with moz-extension, then try to do the import and report if there are any errors etc in any of the windows?

[nmw]

I'm having the same problem. I'm running on Arch Linux with the same browser and Yomitan versions as Nyuutrino.

I've tried the "Inspect" on the extension and workers, and it doesn't throw any errors on their "Console" tabs when it stalls. Is there anywhere else I should be looking?

I can confirm that it is intermittent, and gets stuck at different points, though it does seem to be somewhat consistent about which points. I believe I've had it stick at 17% as well, and frequently 37%.

While writing this I did some further attempts, the next was successful, so I deleted it to try again and ran a bunch more attempts that stopped as follows:

37%, 37%, success, 56%, 56%, 56%, 54%, 37%, 52%, 92%, 92%, 50%, 37%, success.

Each time it got stuck, I would stop it by disabling Yomitan in the Add-ons Manager, which would give me a warning pop-up about leaving the page, and then turn it back on and make another attempt.

It is a particularly large dictionary (84MB zip), which I was having the most trouble with. I did have one of the smaller ones I installed earlier stall as well, though, so I don't think it's specifically size related, but huge dictionaries might be the way to reproduce it. I also added a 64G swap file, in addition to 32G of ram thinking it might be memory related, but that didn't seem to make a difference.

EDIT:
I did more testing on my other PC, Also Arch Linux, Yomitan 25.1.5, but Firefox 133.0.3
Installed the same dictionary successfully, 4 times in a row.

Upgraded the firefox package to 134.0.1

Stalled at 19%,
Stalled at 19%,
Failed a 3rd time, with an error message! "Error: JSON.parse: unterminated string at line 1 column 15951730 of the JSON data in 'term_bank_10.json'
Stalled at 50%
Stalled at 76%
Stalled at 29%
Stalled at 52%

Guessing something's busted between firefox 133.0.3, and 134.0.1

I noticed a possibly tangentially related bug on Firefox's tracker, which was someone getting a bunch of junk at the end of their JSON data, which jives pretty well with the one "unterminated string" error I got. That issue looks like it's in limbo, but my bet is that firefox has broken something in their JSON parsing, but I'm guessing the Yomitan devs here might be able to pass a more useful bug report up than what I could come up with if they can give it a go in firefox 134 and reproduce it.

-Nick

[chrngb]

Same problem for me too. First time installing the extension. I was able to download all recommended dictionaries but for Jitendex it always gets stuck at random number. Upon running the inspection, I was able to obtain following errors

Extension has encountered a problem.
Originating URL: moz-extension://c4a8b302-2ea6-47a4-9709-d269292d6e55/js/comm/shared-worker-bridge.js
SharedWorkerBridge: backend port is not registered

Issues can be reported at https://github.com/yomidevs/yomitan/issues 7 log.js:138:35
downloadable font: Glyph bbox was incorrect (glyph ids 50 55 111 124 155 231 232 263 290 303 323 368 387 414 415 430 435 439 453 473 482 515 540 556 591 625 634 638 639 641 645 652 656 658 659 660 664 665 666 667 668 669 672 673 675 679 681 682 684 686 687 692 695 708 710 712 721 760 767 768 781 800 808 809 810 851 907 909 1179 1182 1183 1184 1185 1188 1189 1190 1191 1193 1194 1195 1197 1206 1207 1244 1255 1257 1258 1261 1268 1303 1308 1312 1314 1316 1382 1542 1544 1547 1548 1549 1560 1576 1577 1578 1620 1651 1679 1683 1689 1706 1717 1721 1756 1760 1763 1778 1819 1820 1821 1835 1836 1858 1884 1892 1901 1946 1951 1959 1996 2001 2013 2024 2049 2063 2095 2097 2102 2111 2136 2141 2155 2159 2172 2178 2179 2219 2220 2233 2239 2254 2265 2295 2309 2327 2338 2349 2370 2371 2381 2384 2386 2419 2434 2441 2450 2459 2488 2501 2519 2557 2558 2559 2570 2575 2600 2634 2637 2662 2679 2685 2700 2702 2719 2764 2769 2778 2800 2801 2803 2804 2814 2824 2827 2831 2837 2871 2909 2933 2934 2937 2938 2939 2986 3079 3086 3094 3113 3134 3138 3139 3148 3161 3170 3225 3246 3254 3263 3270 3276 3299 3306 3319 3339 3344 3353 3355 3359 3387 3399 3417 3436 3438 3503 3561 3605 3631 3640 3649 3670 3678 3686 3689 3692 3735 3746 3757 3775 3778 3791 3827 3836 3843 3867 3869 3890 3912 3931 3934 3936 3949 3994 4009 4016 4038 4059 4060 4072 4077 4079 4098 4117 4122 4150 4168 4180 4185 4189 4204 4205 4208 4209 4211 4212 4229 4230 4234 4237 4241 4244 4248 4251 4255 4257 4259 4276 4306 4308 4325 4337 4364 4396 4413 4458 4459 4461 4498 4501 4534 4549 4557 4563 4570 4594 4596 4608 4619 4622 4632 4660 4663 4677 4680 4685 4714 4722 4737 4742 4750 4763 4770 4776 4792 4806 4817 4819 4831 4858 4862 4885 4887 4893 4895 4902 4905 4915 4934 4937 4939 4944 4947 4979 4986 5001 5029 5035 5043 5054 5055 5060 5066 5104 5108 5117 5121 5124 5129 5139 5149 5155 5160 5173 5177 5197 5218 5222 5232 5233 5267 5285 5287 5296 5347 5377 5378 5388 5399 5434 5442 5467 5476 5479 5486 5491 5525 5526 5528 5551 5575 5616 5632 5645 5655 5666 5667 5676 5685 5707 5712 5736 5741 5744 5757 5770 5793 5797 5804 5810 5815 5844 5863 5865 5880 5926 5928 5935 5942 6051 6058 6073 6075 6119 6150 6168 6171 6197 6198 6215 6220 6225 6228 6242 6260 6291 6310 6326 6347 6366 6376 6390 6424 6450 6478 6486 6509 6510 6511 6529 6532 6534 6549 6559 6560 6565 6566 6572 6579 6638 6674 6679 6685 6698 6699 6704 6705 6709 6711 6718 6722 6745 6758 6765 6771 6773 6778 6781 6784 6787 6788 6803 6806 6833 6849 6852 6862 6866 6867 6875 6894 6899 6900 6905 6913 6941 6959 6967 6974 6978 6985 6991 6997 6998 7028 7038 7058 7119 7131 7137 7142 7143 7146 7156 7167 7176 7178 7186 7192 7199 7203 7210 7214 7223 7224 7225 7235 7236 7239 7243 7250 7262 7266 7274 7283 7299 7301 7303 7321 7327 7329 7385 7410 7431 7457 7463 7469 7474 7479 7480 7492 7513 7514 7541 7553 7556 7573 7582 7584 7587 7606 7610 7632 7635 7643 7646 7647 7650 7652 7656 7658 7659 7675 7676 7679 7697 7699 7700 7702 7706 7715 7716 7719 7730 7748 7754 7768 7773 7789 7791 7793 7795 7796 7798 7800 7801 7806 7807 7808 7811 7815 7817 7819 7823 7828 7833 7835 7837 7840 7844 7848 7850 7854 7859 7886 7896 7903 7918 7922 7923 7924 7925 7933 7935 7942 7944 7948 7954 7962 7982 7983 7986 7988 8073 8085 8088 8095 8096 8097 8102 8107 8108 8202 8212 8213 8214 8215 8216 8236 8246 8324 8325) (font-family: "kanji-stroke-orders" style:normal weight:400 stretch:100 src index:0) source: moz-extension://c4a8b302-2ea6-47a4-9709-d269292d6e55/data/fonts/kanji-stroke-orders.ttf
Background event page was not terminated on idle because a DevTools toolbox is attached to the extension. background.html

​

[Nyuutrino]

@djahandarie Yes, I do have a background in programming, including JS.
I did not find any errors when downloading the dictionaries on Windows. To make things even more confusing, when checking out the process via breakpoints & stepping, Yomitan imported the dictionary without any issues. The only error I found was when I closed and re-opened Firefox to see if that changed anything, and the message I got was this:

Extension has encountered a problem.
Originating URL: moz-extension://4f0a41c1-1a07-41f7-a0f5-5221450e6e86/js/comm/shared-worker-bridge.js
SharedWorkerBridge: backend port is not registered

Issues can be reported at https://github.com/yomidevs/yomitan/issues

I tried this on Arch with the same dictionary (BCCWJ) and the same thing happened (completed when pausing & stepping through the process). However, it froze up when importing another dictionary which I had from a zip file. Note that the same error occurred when I opened up Firefox and checked the console (backend port is not registered).
If this helps any, I can pause the JavaScript through the debugger when the validation seems to be working, but once the validation stops pausing the debugger does nothing. This likely means that no JS is running at that moment, which means the validation code has exited for some reason. I made sure to have the debugger pause on both caught and uncaught exceptions but none showed up during the validation process.

[nmw]

I did see the "backend port is not registered" as well, at some point, but I think it's probably not at the core of this issue.

I'm pretty sure it's a firefox bug in JSON handling. I don't know enough how to actually characterize it, but the fact that it consistently worked on 133.0.3 and consistently broke after upgrading only firefox to 134.0.1, and finally getting a related error message the one time makes me fairly confident they're mangling json on the way into the browser.

I also remembered that I got an odd JSON error after installing ankiconnect, and browsing to localhost:8765. "SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data"
Meanwhile, no errors when connecting from chromium, and ankiconnect seems to be working, regardless.

If I can figure out how to connect and see the raw response, I'll see if it's mangled, and likely that's where our problem is.

Ehh, I guess I need to see it from firefox's view point, since the actual data being imported is fine, and it's getting broken on the way in when firefox parses it. I might have to learn to use a debugger.

EDIT: I think I might have incorrectly correlated those two error messages. Browsing to ankiconnect on firefox 133 still gives the error.

I did test some more, though, and after downgrading to 133.0.2, the dictionaries will import. They do seem to randomly stop for a long time, but unlike in 134.0.1, they do eventually continue, and I've had a 100% success rate importing the same dictionary in 133.0.2. I ran it sucessfully 4 more times in a row on 133.0.2, and then upgraded to 134.0.1 again, and stalled at 58% on the first attempt.

[manolol1]

I had the same issue in Firefox 134 on Fedora.
Downgrading to the last available version fixed it.

[Nyuutrino]

It seems like Firefox came out with a new patch (134.0.2), which fixes this bug:

Fixed an issue where data consumption from service workers may unexpectedly halt (Bug 1941210).

I traced some processes with the debugger and it seemed that the process of sending data between threads just randomly stopped, so I wouldn't be surprised if this was the problem and is now fixed.

I was able to import the dictionary that wasn't importing before now. Does anyone else have any luck with the new patch?

[nmw]

Yes, seems to work again for me as well. 134.0 works, 134.0.1 broken, 134.0.2 works.