chore(common): Update standards data

[ermshiperete]

@keymanapp-test-bot skip

[keymanapp-test-bot]

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

• Android
  • Keyman for Android apk
  • FirstVoices for Android 16.0 apk
  • FirstVoices Keyboards for Android apk
  • KeyboardHarness apk
  • Keyman for Android 16.0 apk
  • KMSample1 apk
  • KMSample2 apk
• Developer
  • Keyman Developer
  • Compiler Regression Tests
  • kmcomp.zip
• iOS
  • Keyman for iOS (simulator image)
  • FirstVoices Keyboards for iOS (simulator image)
  • TestFlight internal PR build version - 18.0.41 (0.11003.11044)
• Keyboards
  • Test Keyboards
• Linux
  • Keyman for Linux: focal
  • Keyman for Linux: jammy
  • Keyman for Linux: mantic
  • Keyman for Linux: noble
• macOS
  • Keyman for macOS
• Web
  • KeymanWeb Test Home
• Windows
  • Keyman for Windows
  • FirstVoices Keyboards for Windows
  • Text Editor (32 bit)
  • Text Editor (64 bit)

[mcdurdin]

We just have to be a little careful with testing in case metadata formats have changed.

[ermshiperete]

One thing that changed is in iso639-3.tab the Language_Type column now has H (Historical) instead of A (Ancient) for several languages. But I'm not sure that information is used anywhere in our code.

Other than that it's only updated data AFAICT.

[rc-swag]

Here are some observations using the Delphi debugger. However, I really don't understand the whole code.
There is an error about inserting dulplicates

class procedure TISO6393ToBCP47Map.Fill(dict: TDictionary<string, string>);
begin
  dict.Add('aar','aa');
  dict.Add('abk','ab');

For the failing test of sqt-YE the following if statement in CanonicalLanguageCodeUtils.FindBestTag evaluates to true and exits. For the master branch this if statement evaluates to false.

if not TLangTagsMap.LangTags.TryGetValue(t.Tag, LangTag) then
   begin
     // Not a known tag but perhaps it's a custom language
     // We'll make no further assumptions
     Exit(t.Tag);
   end;

[rc-swag]

The duplicate insertion area was the key here, due to duplicates in the langtag.json source, this meant the dictionary was not completely loaded resulting in these other tests failing. In other words those failed tests where a side effect of the duplicate data error.

[mcdurdin]

Failing tests due to data issues in langtags.json, reported at:

• bug: "ar-aeb-Arab-TN" is listed twice in tags property of "aeb-Arab-TN" silnrsi/langtags#15
• bug: "thc-Zzzz-LA-x-laipao", "thc-Zzzz-x-laipao" are listed under both tpo-Zzzz-LA and tpo-Zzzz-LA-x-laipao silnrsi/langtags#16

Note that most of the failing tests are caused by a cascade of errors rather than these root causes.

[mcdurdin]

Suggested change

	"latnnames": [ "Phu Thai", "Phuu Thai" ],
	"latnnames": [ "Phu Thai or Phuu Thai", "Phu Thai or Phuu Thai" ],

Per discussion in silnrsi/langtags#17 (comment), this is valid.

[ermshiperete]

Updated to the latest standards data which has the data fixed that previously required to handcraft langtags.json.

[rc-swag]

lgtm

[keyman-server]

Changes in this pull request will be available for download in Keyman version 18.0.42-alpha