Localization primitive proposal

[victormihalache]

Preamble

The current i18n primitive is good as long as you don't need to take care of the plural/singular distinction languages make, or the masculine/feminine distinction; hence larger project often find themselves using heavier, more complex libraries just to deal with that in addition to things such as date formats.

There are great projects like Mozilla's Project Fluent that do a great job at syntax handling for many languages, at the expense of performance (this is due to the translation syntax needing to be parsed at runtime, with more complex syntax being its own mini programming language).

My draft proposal for the l10n/localization primitive strives to solve both the performance bottlenecks and the limitations a pseudo-programming language that needs to be quickly parsed at runtime brings.

Note on i18n vs l10n

Although I couldn't find much online the general consensus seems to be:

• i18n ("internationalization") is about making a product usable and available in many countries (e.g.: making payment work between many countries);
• l10n ("localisation") is about making a product or service feel like it was made specifically for that region (refered to as "locale") (e.g.: translating the interface).

Main argument

JavaScript has a native template string using `, and ${... JS code here ...}, so why not use that? By asking people I known (devs an non-devs) the impression I get from devs is that "translators can't/don't want to code in JS when all they are here to do is translate" but I would argue that using the ternary if/else statement makes just as much sense as Fluent's syntax, here is a comparison:

/** JS template string */
`${username} ${
  photoCount === 0
    ? "added no new photos"
    : photoCount === 1
    ? "added a new photo"
    : photoCount > 1
    ? "added " + photoCount + " new photos"
    : ""
} to ${
  userGender === "male"
    ? "his stream"
    : userGender === "female"
    ? "her stream"
    : "their stream"
}
`;

# Fluent
shared-photos =
    {$userName} {$photoCount ->
        [one] added a new photo
       *[other] added {$photoCount} new photos
    } to {$userGender ->
        [male] his stream
        [female] her stream
       *[other] their stream
    }.

If someone is taught how the conditional ternary operator looks and works, how a template string can inject variables, and how string concatenation works (all things you indirectly understand with Fluent's syntax too), they will be able to read the templated string and make sense of it.

No string interpolation was needed, just plain JavaScript working. This has a many benefits:

• Type safety (at leas the way I made it).
• No code needed at runtime to interpolate the translation file (that takes time!).
• Very easy implementation (a context and API to handle the locales).
• "Out-of-the-box" reactivity (the template string is reactive by default).
• Flexible translations.
• Flexible project structure (some libraries require a certain structure to work with type safety).

Implementation

I'll just paste the files here since it's less than 40 LOC.

/** index.ts */
import { formalName, ISOName, l10n } from "./store";
import { Phraseology, TranslationStringConstructor } from "./types";
import { setLocale } from "./utils";

export { setLocale, l10n, formalName, ISOName };
export type { Phraseology, TranslationStringConstructor };

Just exporting all the stuff a dev might need from the primitive

/** store.ts */
import { createSignal } from "solid-js";
import { Phraseology } from "./types";

export const [l10n, setL10n] = createSignal<Phraseology>();
export const [ISOName, setISOName] = createSignal<string>();
export const [formalName, setFormalName] = createSignal<string>();

We store some data for the locales to work.

/** types.d.ts */
export interface TranslationStringConstructor<T> {
  (data: T): string;
}

export interface Phraseology {
  [key: string]: (data: { [key: string]: string | number }) => string;
}

These types are used throughout the primitive.

/** utils.ts */
import { setFormalName, setISOName, setL10n } from "./store";
import { Phraseology } from "./types";

export const setLocale = (
  phraseology: Phraseology,
  ISOName: string,
  formalName: string
): void => {
  if (phraseology) {
    setL10n(phraseology);
    setISOName(ISOName);
    setFormalName(formalName);
  }
};

Only real function, makes sure to set everything correctly

Since the "magic" is done by the native templates, the only thing we have to do is make sure the developer can utilize the primitive and that everything stays reactive.

Translating

I'll assume a folder structure here, but it can be really really custom.

.
src/
| l10n/
| | locales.d.ts
| | it_IT.ts
| | en_US.ts
| | ...
| 
| ...

...

In the locales.d.ts we just have the TypeScript definition for what strings we want (that way we can get errors when not all locales have that string defined)

/** locales.d.ts */
import { Phraseology, TranslationStringConstructor } from "../lib/types";

export interface Translation extends Phraseology {
  GREETING: TranslationStringConstructor<{ gender: string; name: string }>;
  NEW_PHOTOS_UPLOADED: TranslationStringConstructor<{ count: number }>;
}

The use of the TranslationStringConstructor type is so we only really focus on the data it receives, not the fact that it's a function.

We then go on to define the language files (it_IT, and en_US):

/** it_IT.ts */
import { Translation } from "./locales";

const it_IT: Translation = {
  GREETING: ({ gender, name }) =>
    `Ciao ${gender === "female" ? "cara" : "caro"} ${name}!`,
  NEW_PHOTOS_UPLOADED: ({ count }) =>
    `${
      count <= 0
        ? "Non sono state caricate nuove foto"
        : count === 1
        ? "È stata caricata una nuova foto"
        : count > 1
        ? "Sono state caricate " + count + " nuove foto"
        : ""
    }`,
};

export default it_IT;

Usage in the app

There is only one function: setLocale()

setLocale(it_IT, "it_IT", "Italian");

and then in the JSX:

<p>{l10n().GREETING({ name: name(), gender: gender() })}</p>

where name() and gender() might be signals. They keep the reactivity.

End notes

This is my first attempt at suggesting something for an open source project, if there are any things I should go over, or give a better reason for some of the choices please tell me!

[thetarnav]

Hmmm
This looks amazing.
I like the simplicity and focusing only on a single feature — managing locales. Because of that, it could be used with any solution for getting the used language (e.g. browser preferred language) or providing the state to the application (e.g. context API). Or it could even be used with the existing i18n package, as they seem to have different aims.

There are a couple of things I'm wondering:

• Could the Phraseology interface also accept string as property value? I don't see a problem with it not being a function sometimes.

• The TranslationStringConstructor could accept an array of arguments really, no need to limit it to only one I think.

• Is it necessary to create and export formalName and ISOName signals? They aren't being used anywhere internally, the user could easily create their own.
  But if we keep them — users should benefit from them being updated simultaneously, so batched, and currently, they aren't.

• Could the l10n be a store/proxy? Since it will always be an object you might as well, right?
  The translation access could be potentially shortened to: (with array arguments idea too)

<p>{ l10n.GREETING(name(), gender()) }</p>

• The three internal signals, should probably be scoped to some constructor function. (at least the l10n one)
  If the package would be used by multiple parties on the same page — e.g. in application and 3-rd party library logic — then they will be conflicts.
  It would also help with typings. You've claimed that this would have good ts support, but from what I'm seeing the l10n signal will always just return the base Phraseology type, which is too broad — you could easily put wrong arguments to some string function without even knowing about it.

Other than that — it's a great idea :)

[victormihalache]

For starters, you're correct: setting the language can be done however the developer likes, be it browser preference or not, you just have to use the setLocale() function, pass it the locale object, the ISO name, and the formalName/displayName, and everything will update.

I initially used the Context API (as that's how i18n did it) to share the two non-l10n signals, but found that exporting the signals was simpler, and pretty much the same (I don't know much about the Context API to know if I was doing something wrong, but the idea of doing const [ISOName] = useLocalization() + its import, instead of a simple import { ISOName } from "lib-path-here" felt odd to me).

As for the points:

Could the Phraseology interface also accept string as property value? I don't see a problem with it not being a function sometimes.

You're absolutely right, I forgot there cases where there is a simple "Accept" string.

The TranslationStringConstructor could accept an array of arguments really, no need to limit it to only one I think.

It does accept any number as long as it's used like this:

GREETING: TranslationStringConstructor<{ gender: string; name: string }>;

Is it necessary to create and export formalName and ISOName signals? They aren't being used anywhere internally, the user could easily create their own.

But if we keep them — users should benefit from them being updated simultaneously, so batched, and currently, they aren't.

You're also right here, there's no need for them, I wrote them in because I would've used them for sure, but there's no need to use them (they wouldn't bee tree-shakeable) if the user doesn't want to.

Could the l10n be a store/proxy? Since it will always be an object you might as well, right?

I don't know much about stores but from a quick chat on discord it seemed to me like a store would make a signal for every entry. This would never change a single entry in the object, either all the object changes (aka a locale change), or nothing changes, there's no need to change only some keys.

The translation access could be potentially shortened to: (with array arguments idea too)

<p>{ l10n.GREETING(name(), gender()) }</p>

I haven't thought about that, should it accept both or be exclusively one or the other (I don't know if both would be doable with typescript)?

The three internal signals, should probably be scoped to some constructor function. (at least the l10n one)

If the package would be used by multiple parties on the same page — e.g. in application and 3-rd party library logic — then they will be conflicts.

It would also help with typings. You've claimed that this would have good ts support, but from what I'm seeing the l10n signal will always just return the base Phraseology type, which is too broad — you could easily put wrong arguments to some string function without even knowing about it.

I know that there is not any intellisense when using the l10n object, but I don't know how to make them aviable without referencing the src folder.

---

There are one thing that I want to clear:

This primitive does conflict with the i18n primitive, as both aim to provide a way to translate strings, and manage locales, just in a different way.

I personally thing that this approach is better as it is closer to the actual JS and doesn't need to interpolate the string to translate, just uses the native JS way of things.

[thetarnav]

As for the types, proxy, and factory function, here is a demo of what I was talking about: https://playground.solidjs.com/?hash=86161458&version=1.4.1

[thetarnav]

Also,
Would love to see some kind of support for lazy loading translations — you only need them when selected, right?
Not shipping all of the translations at once would be a nice thing to solve for users, otherwise, they would have to figure out that themselves.

[victormihalache]

Pretty sure lazy loading can be done the same way as it's done for the router, will have to test it though.

[thetarnav]

@atk I feel like you might have some ideas here.

[atk]

Lazy loading i18n is always a compromise. Sure, you can use a resource and suspense to only show the translated result, but you'll slow down the initial page load.

It would be interesting though to research different loading strategies, i.e. sequential loading for initial translations and lazy loading for subsequent views.

[thetarnav]

Isn't this how the router does it? When using lazy I mean. Handle initial synchronously and append to the main bundle, and fetch lazily subsequent routes. Maybe not, just guessing.

[atk]

I never tried that. In any case, it would probably be cool if you could use a Promise instead of actual translations and then the TFunction would just output lazy(Promise) instead of a string. That would make for an awesome DX, because the whole asynchronity suddenly becomes transparent. Something like (additional features omitted to show the principle):

export interface Translations {
  [key: string]: string | Translations
};
export type TOptions = { /* ... */ }:
export type TFunction<Output = string>(key: string, opts?: TOptions) => Output;

export function i18n(translations: Translations): TFunction;
export function i18n(translations: Promise<Translations>): JSX.Element;

[davedbase]

This is a great suggestion! We recently had a chat in the solid-start Discord channel re i18n/l10n. I like that this proposal remains flexible, considers reactivity and seems to fit well with the future plans of Solid Start. Would love to see this proceed.

[samuelstroschein]

Very interesting. I have not seen the usage of template literal strings for the expression of plurals etc. The benefits from a dev perspective are clear. However, I fear that not all nuances of human languages can be (easily) expressed while leveraging template literal strings. Fluent syntax goes beyond pluralization. Nevertheless, neat idea.

"translators can't/don't want to code in JS when all they are here to do is translate"

That argument is true but will become obsolete.

I am working on an entire localization ecosystem that includes an editor for translators. The editor abstracts away code syntax. Albeit a draft, here is an RFC opral/monorepo#127. One of the questions in the RFC is how different syntaxes, like the one in this proposal, can be supported. I think I found a novel solution.

Summary of the RFC

The editor uses git, no sync whatsoever is required. The only thing that is required to use the editor would be a config file. The neat thing, which we have seen with front-end tooling, is the fact that the config is written in JS and can import external code. To support different syntaxes that express human languages, like the one in this discussion, parsing and serialization can be defined in the JS config. The editor, which runs in the browser, loads and executes the config (I am exploring sandboxing). Using a JS config that is executed in the browser, different syntaxes and libraries could be supported.

I am keen to hear feedback. Ideally, in the RFC discussion.

[victormihalache]

Most of what I wanted to implement has now been implemented.

[samuelstroschein]

We are exploring to build a metaframework-agnostic (Solid, Svelte, NextJS, etc.) i18n library that overlaps with this proposal.

Discussion is ongoing at opral/monorepo#395

[atk]

@samuelstroschein: I'm happy to see multiple solutions with different levels of complexity, so that we may be able to provide the best solution possible for different use cases.

It seems to me that inlang is a great solution for larger-scale projects with more complex i18n requirements that our current solutions wouldn't be easily sufficient for. Maybe we can even partially derive the tooling for our less sophisticated solutions.

[samuelstroschein]

It seems to me that inlang is a great solution for larger-scale projects with more complex i18n requirements that our current solutions wouldn't be easily sufficient for.

Inlang is supposed to be the easiest solution that scales well with applications like Instagram. We need to improve our communication. While yes, Instagram is large, the processes of localizing Instagram or a Todo app are identical.

What gives you the impression that inlang is not suited for simple Todo apps?

[atk]

And it certainly makes managing your translations easy, that much is understood.

It is certainly suited, but the ~5 small strings that you need for a Todo app are easily manually put into a translation object. Installation size should also be a consideration.

That being said, if your project has more than 5 translated strings, inlang is definitely a good choice to build your i18n at scale.

[samuelstroschein]

Yes, we need to improve our communication! :D No difference exists between having 5 strings and 50000 strings.

Installation size should also be a consideration.

The proposal at https://github.com/orgs/inlang/discussions/395 will lead to a compiler-based i18n library. A compiler-based i18n library should outperform runtime solutions regarding KB shipped and simultaneously scale to complex i18n processes.

[atk]

No difference exists between having 5 strings and 50000 strings.

Sometimes, good enough is better than perfect. Having a full-featured i18n tool chain with ast-based transformations is massive over engineering for 5 tokens that will never, ever change.

I actually share the vision of compiler-based i18n and had suggested such a solution in an earlier thread. If you need any help with the solid part of your solution, I'll gladly support you.

Still, our aim as community is to give our users a rich choice of quality solutions to their actual issues. Therefore, I feel we must support a simpler solution as well, the same way that solid supports other templating choices than JSX.

[samuelstroschein]

Having a full-featured i18n tool chain with ast-based transformations is massive over engineering for 5 tokens that will never, ever change.

Now I understand your stance. Yes, please don't do that if the translations never change haha.

If you need any help with the solid part of your solution, I'll gladly support you.

Thanks! I will get back to the offer if we decide to move ahead and build the i18n library.