Add solution to check consistency of translations

[benoit74]

With #69, we have introduced i18n.

However, currently there is no tooling to ensure consistency:

• identify unused strings (strings present in JSON files but not used - anymore - in JS or Python codebase)
• identify missing strings (strings present in JS or Python codebase but not present in JSON files - must be present in at least en for proper operation and in qqq for proper TranslateWiki operation)

[advait-zx]

Flag any keys that are present in the JSON files but not used in the code
how can i contribute to this issue

[benoit74]

I think we first need to check if such a tool does not already exist. Be aware that while all translated strings are stored in JSON files, they are used both in .vue and Jinja2 templates. They could theoretically soon be used in .py files.

If it does not exist, then I would suggest to develop a small Python script doing the job. Be aware that we need both directions, i.e. flag any keys present in the JSON but not used in the code AND flag any keys used in the code but not present in at least en and qqq JSON files.

This Python script should exit with non-zero exit code when problem(s) are detected, and we can add this script to invoke tasks.py and the the QA CI.

I will soon merge #69, so this will be easier to work on.

[benoit74]

@advait-zx do you confirm you will work on it, shall I assign you the issue?

[advait-zx]

yeah but i will need help for this issue

[benoit74]

Isn't what I've described a sufficient start point to make progress and if needed prepare a PR? If not unfortunately I can't help more than that.

[Prakhar-Shankar]

Hi @benoit74, can I work on this issue?

[benoit74]

Hi @Prakhar-Shankar, sure. Note that at first, no code is needed, you need to search for a tool capable of fulfilling this issue requirements.

[Prakhar-Shankar]

Sure, I will first search for the tool, and will try to tell the approach. If the tool does not exist I will try to write a python script. I will try to provide descriptions before committing anything.

[Prakhar-Shankar]

Hi @benoit74, I found a tool called vue-i18n-extract. This tool can flag all the string that are unused or are missing. I ran the tool and here is the output -

Also we can make a JSON file where the outputs can be stored.

[benoit74]

Seems promising!

Is it possible to check all languages at once?

The problem is that :

• some of these keys are in fact used, I don't get why your tool report they are not used, see e.g.

  zimit-frontend/ui/src/stores/main.ts

  Line 137 in e8bac13

  text: this.t('newRequest.fetchingDefinition')

• some of these keys are used in Python backend, we need a tool capable to handle both Vue.JS and Python. Or two tools and we merge their results: it might be better to rely on a tool to detect which path are unused because this is probably complex, and then merge their result with a custom tool of our own
• we should be able to ignore some key(s): for now, language is not used on purpose
• it seems to do only the first point of my initial comment ; we need to do the other direction as well (check which path are used in the code but are not present in at least qqq and en)

Note that we need to integrate this check to the CI, so we need to check all languages and we need a clean return code (different than 0 when there is a problem, so that it will fail the CI).

[Prakhar-Shankar]

Got it, the issue with this tool is, it can check only .vue files and not the python files.
I am still searching for a tool which can analyze the python files as well.

Therefore, we may have to use two different tools and later combine the result as you said. If nothing works out we may have to write a python script to check the .py files.

Coming to your third point, this tool is capable of finding missing keys as well, I don't know why I didn't work, I will surely work on it and verify the process. Also I will keep in mind about the CI part.
Thanks for the instructions, I will try to get the solution soon.