Rewrite the i18n scripts #5177

obulat · 2024-11-22T10:56:19Z

Fixes

Fixes #4979 by @sarayourfriend
Fixes #629 by @obulat
Fixes #5059 by @obulat
Related to #592 by @sarayourfriend

Description

This PR re-writes the i18n locale metadata and translation files generation.

Folder structure

The recent update in the Nuxt i18n has introduced a new top-level i18n folder. Now all i18n-related files are located there:

i18n/data folder contains the source-of-truth original en.json5 file. The locale metadata (valid-locales.json and untranslated-locales.json) are saved there as well.
i18n/locales folder is where all downloaded translations are saved.
i18n/scripts folder contains the scripts that download locale metadata, translations, convert en.json5 to .pot file that is used for translations on https://translate.wordpress.org

Code modernization

Now that it's easy to run modules in Node, the scripts can be converted to .mjs and use import instead of require. This PR contains the conversion, as well as migration from axios to native fetch.

Flattening of the keys

To simplify finding static and dynamic keys in the source code, as well as the conversion from and to .pot files, the keys of the translation strings have been flattened. So now, en.json5 is a flat JSON file, where all keys are top-level. This required the update to the ESLint rule for the json5 keys.

Description of changed files (what to review)

The main entry point for the i18n scripts is now i18n/scripts/setup.mjs. This file reads the CLI options, and creates the necessary i18n files. All modes (English-only, production - downloading all locale metadata and translations, testing - copying of the local files from the test folder) are now combined in a single workflow.

The order of the steps for production was changed because the translated property for the locale metadata is now generated based on the number of keys in the locale's json file, so we need to have the translations before creating the list of available locales.

Previously, the steps were:

get the list of locales and their properties from the GlotPress GitHub repository
parse the translation statistics from the Openverse project page on https://translate.wordpress.org and add the translated property to the locales from step 1.
download all translations in a zip and convert the contained flat json files into nested json.
save the list of locales for which the translations were downloaded to valid-locales.json file, and the locales without translations to invalid-locales.json.

In this PR, step 2 was eliminated, and now the process is:

(previously, step 3) download all translations and save the non-empty json files. We no longer need to convert flat json files, and we don't need to save empty files as we are doing now.
(previously, step 1) get the list of locales and their properties from the GlotPress GitHub repository, and compute the translated percentage (comparing the number of keys in the original English en.json5 file and the translated json file.
(previously, step 4) save the list of locales for which the translations were downloaded to valid-locales.json file, and the locales without translations to invalid-locales.json.

The code for the new step 1 is located in i18n/scripts/translations.mjs, for the new step 2 and 3 - in i18n/scripts/metadata.mjs.

The code for generating the pot file was also simplified now that we don't need to handle nested json, and is now located in i18n/scripts/generate-pot.mjs.

All of the paths for files are now located in a single file, i18n/scripts/paths.mjs, so it will be easier to safely move a file and update its paths in a single place.

The i18n check script was removed from frontend/package.json because it wasn't working, and the library doesn't seem to be maintained.

Testing Instructions

Run the i18n scripts, deleting the locale data between the runs to ensure that existing data does not interfere.

"i18n" - the main script, by default generates all the data needed for production. Can also be used with flags.
"i18n:debug" - the same as i18n, but has more verbose logging, including the logging of keys/placeholders that have been modified to fix one of the linked GitHub issues
"i18n:en" - only generates the en.json file and an empty valid-locales.json. This can be used for quickly running local environment, including with the --watch flag (that watches the en.json5 file and updates en.json when the original file changes)
"i18n:test" - copies the files from test/locales into correct folders
"i18n:no-get" - used to generate valid-locales.json file based on existing translation files, without downloading translations from GlotPress
"i18n:generate-pot" - converts en.json5 file to pot file. The generated file should be the same as the one generated on main.

Checklist

My pull request has a descriptive title (not a vague title likeUpdate index.md).
My pull request targets the default branch of the repository (main) or a parent feature branch.
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no visible errors.
I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
PRs) or the media properties generator (ov just catalog/generate-docs media-props
for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

github-actions · 2024-11-22T18:44:43Z

Latest k6 run output¹

     ✓ status was 200

     checks.........................: 100.00% ✓ 6400      ✗ 0   
     data_received..................: 1.5 GB  8.1 MB/s
     data_sent......................: 838 kB  4.6 kB/s
     http_req_blocked...............: avg=20.4µs   min=2.04µs   med=3.71µs   max=8.68ms  p(90)=5.23µs   p(95)=5.66µs  
     http_req_connecting............: avg=16.23µs  min=0s       med=0s       max=8.62ms  p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=635.3ms  min=37.57ms  med=547.78ms max=2.67s   p(90)=1.15s    p(95)=1.25s   
       { expected_response:true }...: avg=635.3ms  min=37.57ms  med=547.78ms max=2.67s   p(90)=1.15s    p(95)=1.25s   
   ✓ http_req_failed................: 0.00%   ✓ 0         ✗ 6400
     http_req_receiving.............: avg=143.45µs min=41.7µs   med=109.6µs  max=17.79ms p(90)=176.16µs p(95)=212.33µs
     http_req_sending...............: avg=21.53µs  min=6.34µs   med=18.56µs  max=1.24ms  p(90)=24.11µs  p(95)=26.03µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=635.14ms min=37.44ms  med=547.64ms max=2.67s   p(90)=1.15s    p(95)=1.25s   
     http_reqs......................: 6400    35.062444/s
     iteration_duration.............: avg=3.39s    min=299.09ms med=2.87s    max=9.56s   p(90)=6.61s    p(95)=6.92s   
     iterations.....................: 1200    6.574208/s
     vus............................: 9       min=9       max=30
     vus_max........................: 30      min=30      max=30

This comment will automatically update with new output each time k6 runs for this PR ↩

github-actions · 2024-11-23T13:11:00Z

Full-stack documentation: https://docs.openverse.org/_preview/5177

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

Changed files 🔄:

https://docs.openverse.org/_preview/5177/frontend/reference/i18n.html

- Convert CJS to MJS - Use flat keys and simplify json handling - Update ESLint rule to allow for dot in keys

sarayourfriend

This is cool! Thanks for the ping on the issues closing. Did a quick little review and have some simple suggestions. What a nice clean up and reorganisation 🙂

sarayourfriend · 2024-11-26T21:46:00Z

documentation/frontend/reference/i18n.md

@@ -78,7 +75,7 @@ processed and loaded into Nuxt via the Nuxt i18n module.
 ## Test locales

 Three locales are maintained in the code base (as opposed to downloaded from
-Glotpress) for use in testing, alongside the default English locale. These
+GlotPress) for use in testing, alongside the default English locale. These


tip: You can add GlotPress to the Vale TermCasing.yaml so these errors are found automatically in the future.

sarayourfriend · 2024-11-26T21:49:50Z

packages/js/eslint-plugin/src/rules/key-name-casing.ts

+export type CasingKind = "camelCaseWithDot" | "snake_case_with_dot"
+
+export const allowedCaseOptions: CasingKind[] = [
+  "camelCaseWithDot",
+  "snake_case_with_dot",
+]


Just a tip, if you want to avoid duplicating the names in this definition, you can derive CasingKind from allowedCaseOptions:

Suggested change

export type CasingKind = "camelCaseWithDot" | "snake_case_with_dot"

export const allowedCaseOptions: CasingKind[] = [

"camelCaseWithDot",

"snake_case_with_dot",

]

export const allowedCaseOptions = [

"camelCaseWithDot",

"snake_case_with_dot",

] as const

export type CasingKind = typeof allowedCaseOptions[number]

Alternatively, you can derive both from the checkersMap:

const checkersMap = { camelCaseWithDot: isCamelCase, snake_case_with_dot: isSnakeCase, } as const export type CasingKind = keyof typeof checkersMap export const allowedCaseOptions: CasingKind[] = Object.keys(checkersMap)

sarayourfriend · 2024-11-26T21:52:52Z

packages/js/eslint-plugin/src/rules/key-name-casing.ts

+function hasSymbols(str: string) {
+  return /[\u0021-\u0023\u0025-\u002c./\u003a-\u0040\u005b-\u005e`\u007b-\u007d]/u.test(
+    str
+  ) // without " ", "$", "-" and "_"


Are these ranges inclusive of the bounds? When I check https://symbl.cc/en/unicode-table/, 0025 is the space character. Should the the second range be updated to 0026?

Also: I'm not sure whether the inclusion of ., / and ` separate from the unicode ranges is intentional, but if it is, it might be helpful to move them to the start or end of the bracket group so they do not get lost.

sarayourfriend · 2024-11-26T21:56:20Z

packages/js/eslint-plugin/src/rules/key-name-casing.ts

+  return /[\u0021-\u0023\u0025-\u002c./\u003a-\u0040\u005b-\u005e`\u007b-\u007d]/u.test(
+    str
+  ) // without " ", "$", "-" and "_"


Because this rule is only used for locale json files, it might be easier to understand this particular check by inverting it to check the characters that are allowed in a locale key. Would that be, [a-zA-Z._], for example? If this rule is only used for locale key strings, and that would allow simplifying aspects of it by making it more exact, then it's worth doing. This regex is a little obscure as it is 😅

In that case, it would also be good to change the name of the rule to @openverse/locale-key-name-casing.

I copied the code as-is from the key name casing rule, only adding the . :) Great suggestions, thank you!

sarayourfriend · 2024-11-26T21:56:54Z

packages/js/eslint-plugin/src/rules/key-name-casing.ts

+/**
+ * Return case checker ('camelCaseWithDot', 'snake_case_with_dot')
+ */
+export function getChecker(name: "camelCaseWithDot"): (str: string) => boolean {


Suggested change

export function getChecker(name: "camelCaseWithDot"): (str: string) => boolean {

export function getChecker(name: CasingKind): (str: string) => boolean {

obulat requested review from a team as code owners November 22, 2024 10:56

obulat requested review from krysal and dhruvkb and removed request for a team November 22, 2024 10:56

openverse-bot added 🧱 stack: documentation Related to Sphinx documentation 🧱 stack: frontend Related to the Nuxt frontend 🧱 stack: mgmt Related to repo management and automations 🚦 status: awaiting triage Has not been triaged & therefore, not ready for work labels Nov 22, 2024

obulat force-pushed the rewrite-i18n-scripts branch 4 times, most recently from 415f12a to 91e3150 Compare November 22, 2024 18:32

obulat force-pushed the rewrite-i18n-scripts branch 3 times, most recently from 7cde493 to e528812 Compare November 23, 2024 12:57

obulat marked this pull request as draft November 25, 2024 04:37

obulat force-pushed the update-only-nuxt-i18n branch from 2d8ae64 to 362799e Compare November 25, 2024 05:27

obulat force-pushed the rewrite-i18n-scripts branch 2 times, most recently from 1eed35d to 6e19cc4 Compare November 25, 2024 05:31

obulat force-pushed the update-only-nuxt-i18n branch from 362799e to b0b0537 Compare November 25, 2024 05:35

obulat force-pushed the rewrite-i18n-scripts branch from 6e19cc4 to 5afc4b5 Compare November 25, 2024 05:39

obulat force-pushed the update-only-nuxt-i18n branch from b0b0537 to d2c3fea Compare November 25, 2024 05:42

obulat mentioned this pull request Nov 25, 2024

Update Nuxt i18n #5176

Merged

8 tasks

Base automatically changed from update-only-nuxt-i18n to main November 25, 2024 16:10

obulat force-pushed the rewrite-i18n-scripts branch 2 times, most recently from 1c42674 to 4c5854e Compare November 26, 2024 06:45

Rewrite the i18n scripts

86e1b36

- Convert CJS to MJS - Use flat keys and simplify json handling - Update ESLint rule to allow for dot in keys

obulat force-pushed the rewrite-i18n-scripts branch from 4c5854e to 86e1b36 Compare November 26, 2024 08:57

obulat marked this pull request as ready for review November 26, 2024 14:53

sarayourfriend reviewed Nov 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite the i18n scripts #5177

Rewrite the i18n scripts #5177

obulat commented Nov 22, 2024

github-actions bot commented Nov 22, 2024 •

edited

Loading

github-actions bot commented Nov 23, 2024

sarayourfriend left a comment

sarayourfriend Nov 26, 2024

sarayourfriend Nov 26, 2024

sarayourfriend Nov 26, 2024

sarayourfriend Nov 26, 2024

obulat Nov 27, 2024

sarayourfriend Nov 26, 2024

	export function getChecker(name: "camelCaseWithDot"): (str: string) => boolean {
	export function getChecker(name: CasingKind): (str: string) => boolean {

Rewrite the i18n scripts #5177

Are you sure you want to change the base?

Rewrite the i18n scripts #5177

Conversation

obulat commented Nov 22, 2024

Fixes

Description

Folder structure

Code modernization

Flattening of the keys

Description of changed files (what to review)

Testing Instructions

Checklist

Developer Certificate of Origin

github-actions bot commented Nov 22, 2024 • edited Loading

Latest k6 run output1

Footnotes

github-actions bot commented Nov 23, 2024

sarayourfriend left a comment

Choose a reason for hiding this comment

sarayourfriend Nov 26, 2024

Choose a reason for hiding this comment

sarayourfriend Nov 26, 2024

Choose a reason for hiding this comment

sarayourfriend Nov 26, 2024

Choose a reason for hiding this comment

sarayourfriend Nov 26, 2024

Choose a reason for hiding this comment

obulat Nov 27, 2024

Choose a reason for hiding this comment

sarayourfriend Nov 26, 2024

Choose a reason for hiding this comment

github-actions bot commented Nov 22, 2024 •

edited

Loading

Latest k6 run output¹