Compatibility between devices (Apple and Microsoft have incompatible encodings)

[hakavlad]

All password inputs must be converted into Unicode Normalization Form KC, NFKC, and then encoded as UTF-8 bytes. This is to ensure compatibility between devices (Apple and Microsoft have incompatible encodings).

https://github.com/covert-encryption/covert/blob/main/docs/Specification.md#argon2-password-hashing

What do you think about it? Why doesn't Kryptor do Normalization?

[samuel-lucas6]

It sounds like that's a sensible idea (it's recommended by NIST), although it's unclear to me how big of an issue this is. It might be problematic with non-English keyboards, which is something I've never tested.

That just never occurred to me. It doesn't seem that widely discussed, and it looks like age and probably lots of programs don't do normalization either.

[hakavlad]

See also:

https://datatracker.ietf.org/doc/html/rfc8265
Preparation, Enforcement, and Comparison of Internationalized Strings Representing Usernames and Passwords
RFC 8265
RFC - Proposed Standard
"It represents the consensus of the IETF community."

   4.  Normalization Rule: Unicode Normalization Form C (NFC) MUST be
       applied to all strings.

It doesn't seem that widely discussed

However, it has been discussed for over 20 years (at least since RFC 3454 and RFC 4013) and is widely recommended by NIST and IETF.

[hakavlad]

FiloSottile/age#601 Let's expand the number of participants in this discussion.

[samuel-lucas6]

Your "complete" solution is simply to cut off a sore head.

Blame Unicode. You either ignore this problem, do a partial fix, or do a complete fix. And different people/organisations recommend different approaches to doing the partial fix so it's not even like there's one agreed method, whereas there's a clear complete fix.

The DEI department condemns your decision! Think of minorities!

A lot of people speak/write English, ASCII characters are probably accessible on every device, some websites are more restrictive than ASCII characters, and people should be using randomly generated passwords/passphrases anyway, which use ASCII characters unless you go out of your way to use non-ASCII (e.g., a non-English BIP-39 wordlist). The EFF/Diceware/standard BIP-39 wordlists are in English.

Maybe I'll change my mind, but this problem doesn't have a pretty solution.

[hakavlad]

the standards/guidance only ever say SHOULD rather than MUST

rfc8265 says MUST

[hakavlad]

The better approach is probably to restrict the input to ASCII characters, which is I believe what a lot of websites do. Something to think about for the next major version.

How are you going to do this? Reject with an error phrases containing non-ASCII? I would consider such behavior of the tool to be offensive and discriminatory, unreasonably limiting my freedom to choose a password.

[samuel-lucas6]

rfc8265 says MUST

Ok that's my bad; I misremembered. However, that RFC says something different to NIST and NIST only says it's optional, so who do you listen to?

How are you going to do this? Reject with an error phrases containing non-ASCII? I would consider such behavior of the tool to be offensive and discriminatory, unreasonably limiting my freedom to choose a password.

Yes, that's how it would be done, although this issue is also problematic for file names, which probably shouldn't only be ASCII.

It's not discriminatory given this is a problem with Unicode that it sounds like literally cannot be fixed. You can use normalization and still have users fail to log in to a service/decrypt a file, which in my mind isn't a fix at all but a band-aid.

[cipriancraciun]

@hakavlad thanks for pointing this out on the Age project (which I silently follow).

Following @samuel-lucas6's comments, I see that his main counterargument is the unreliability of non-ASCII passwords, which is in fact @hakavlad main purpose for Unicode normalization, and unfortunately on this subject (namely "passwords") I think I side with Samuel's argument.

For example, I've developed a password generator tool (and a few related topics), and a few weeks ago I was pondering of adding Unicode-exclusive password patterns (i.e. just emojis or non ASCII-based passwords). But I've decided against that because I think they'll generate more issues that they'll solve (for once due to possible poor support for Unicode inputs, and second for the portability issue). (The main feature would have been increased entropy for the same "visual length", but as Samuel said, entropy can easily be increased with ASCII-only passwords.)

---

However, and the reason I've opened my own ticket on the subject, is that sometimes there might be cases where Unicode inputs (that do contribute cryptographically) could be encountered.

For example in the case of Associated Data (as part of AEAD) encryption schemes. (Similar to Kryptor and Covert, I too have implemented as part of z-tokens an Age-like encryption tool.)

Now, I don't think this is the case for Kryptor (based on what I've read from the specification), but imagine that one adds a CLI argument like --additional-data [some-human-readable-text] where the user could specify the intended usage of the file (like for example, is this the shutdown.bash or the startup.bash script?), and would like this to be part of the authentication. (This could protect against an attacker that swaps the two files.)

In such a case, the user is more likely to texts native to his own tongue, and thus Unicode. Thus perhaps, especially in such situations Unicode normalization would be a requirement.

Getting back to Kryptor, it allows the usage of both a password and a secret key file (shared secret); thus a user might just use the secret key file as the intended password, and the "password" as the "additional data" in a scheme similar to the above.

---

In the end, each developer makes a decision based on his priorities (which might be completely different from another's). However I find this discussion quite interesting, at least from a theoretical point of view.

Perhaps, if a tool doesn't support Unicode normalization for a particular input, and that normalization is essential for portability (or cryptography), perhaps that tool should issue a warning or error on the usage of non ASCII inputs?

---

How about CLI tools (in Linux/BSD environments) that receive C-strings as inputs (i.e. byte arrays terminated by \0) via main (either as arguments or as environment variables); how should these tools treat these inputs? As UTF-8 encoded strings? As byte encoded Unicode strings according to LC_* environment variables?

[hakavlad]

Additionally, you should almost never be memorising passwords; you should be using a password manager

"My house burned down, my devices burned down, I couldn't access my cloud storage and my password manager..."
https://shkspr.mobi/blog/2022/06/ive-locked-myself-out-of-my-digital-life
This story also shows that remembering passwords can be useful.

In fact, everywhere I say "passwords", I mean "passphrases".

[hakavlad]

you'll probably have access to your password manager

I want to be able to download backups from a public storage and successfully decrypt them after I lose all my devices.

[samuel-lucas6]

With freedom, users can take care of correctness themselves.

That's exactly the problem. The onus shouldn't be on the user. Something like logging in to a site/being able to decrypt should always work (assuming everything else is correct/not tampered with). Your argument feels like 'let's not do validation because the user can do validation themselves', which is generally poor design.

discourages is better then enforcing

Not if you want to guarantee that this isn't a problem. In that case, it's objectively worse.

However, to be honest, I can get onboard with that because that's what lots of other people seem to be doing without caring. If that's 'best practice' and keeps people happy, it's arguably good enough.

The biggest thing that bothers me about normalization is that it's duplicating the password in memory. Now the password already gets duplicated/not cleared properly in Kryptor, but I tried to ensure the password was erased in Cahir. With any sort of non-string password entry, you have to convert it to a string for normalization, with the normalization function also returning a new string. Strings are immutable and not meant to be zeroed as it can potentially crash the runtime, although it seemed to work ok in Cahir. But the point is that you want to minimise the duplication of secrets to limit the risk of them being retrieved/dumped. Normalization defeats half the point of doing an interactive password entry.

This story also shows that remembering passwords can be useful.

I'm not disputing that, but you can't remember passwords for everything. If you do, they'll be terrible. It makes more sense to back up access to your password manager. For example, if you're concerned about MFA, you keep a backup of the recovery code in a safe deposit box at a bank, a box buried in the garden, hidden somewhere at a family member/friend's house, etc. There are ways it can be done, just they're not easy.

I want to be able to download backups from a public storage and successfully decrypt them after I lose all my devices.

Except unless you remember your cloud storage account password and recovery code, you won't have access to that anyway. And it's virtually impossible to remember recovery codes because a) you never use them, b) they're typically gibberish, and c) they're typically long.

But this discussion is going off topic.

[cipriancraciun]

The biggest thing that bothers me about normalization is that it's duplicating the password in memory.

Indeed, the major concern with regard to cryptographic material (like passwords) is erasing their traces after usage, in order to avoid exfiltration. And this is a topic I've thought at length about how to eliminate copies of a password from memory.

And, unfortunately, the conclusion I've reached is that it's almost impossible to make sure copies of the password aren't left anywhere on the system, because:

• if the tool is a CLI, then the password has already traversed the keyboard buffer (in the kernel), X or Wayland (or the user-interface system), GTK or QT (or the toolkit being used), the TTY buffer (in the kernel for the TTY emulation), and finally the CLI process;
• if the tool is a GUI, then it's almost as above, less a few pieces from the stack;
• if the password is supplied from another process, say via pipes, it's still present in the kernel's memory;

Besides zero-ing the buffers in our own systems, we could do the following:

• use allocators that zero on free for all involved software; (this adds performance penalties;)
• use kernel flags (at least on Linux) that zero all memory (used in buffers, pipes, etc.) after use; (which is not enabled by default due to performance penalties;)
• disable SWAP; (one could try to pin a processes memory, but by default non-root users can only do so for a couple of KB;)

Thus, at this point, with today's software, I don't think duplicate copies of a string in a process's memory is much of a concern.

[hakavlad]

The onus shouldn't be on the user.

It is impossible to absolve the user of responsibility.

Either way, the user is responsible for:

• the security of the environment in which the application is run;
• the strength of the keys used;
• preventing the leakage of keys and plaintexts.

you can't remember passwords for everything

Of course, passwords for websites are stored in a password manager. But having the ability to conveniently memorize passphrases for encrypted files is just what we need.

unless you remember your cloud storage account password and recovery code, you won't have access to that anyway

I was talking about public storage. It can be, for example, a public telegram channel (you can store files up to 2 GB free). If you lose your device, you won't need keys to access these files. So, this is exactly the case when it is important to be able to conveniently remember passphrases for encrypted files. This is a way to solve the problem described in the article.

disable SWAP

On Linux, you can use ZRAM to create a compressed swap space, which helps prevent memory page swapping to disk. On Fedora Linux, this is offered as a default option.

[samuel-lucas6]

I've updated the website, adding a section in Known limitations and a warning on the Tutorial pages talking about using a passphrase (e.g., this page).

I'll look into adding a warning in the program based on char values exceeding ASCII.

[hakavlad]

BTW w3c recommends NFC https://www.w3.org/TR/charmod-norm/
Also:

Specifications MUST allow a Unicode character encoding.

[hakavlad]

To eliminate the problem, you have to enforce ASCII characters

It's a redundant solution, a redundant limitation.

It is sufficient not to use certain Unicode symbols when using the tool on different devices.

For example, I can safely use characters from the Russian alphabet; there are no problems with them, as well as with ASCII.

Additionally, I can use ANY Unicode characters as long as I do so on the same device and in the same manner.

Thus, you offer a simple but overly restrictive solution.

[samuel-lucas6]

This is getting a bit silly as we're going around in circles and having multiple separate conversations in different places.

It's a redundant solution, a redundant limitation.

Whether it's redundant depends on the user.

It is sufficient not to use certain Unicode symbols when using the tool on different devices.

Except you have to trust the user, who shouldn't have to know anything about Unicode, to know which Unicode characters to avoid. The average person, even the average computer science student, doesn't know enough about Unicode to do this.

Additionally, I can use ANY Unicode characters as long as I do so on the same device and in the same manner.

I'm not sure that's correct if it depends on the keyboard, operating system, settings, etc. It sounds possible for you to update/upgrade your OS, change your keyboard/settings, whatever and this could potentially be a problem.

Thus, you offer a simple but overly restrictive solution.

However, as I keep saying, it's the only actual solution. Normalization reduces but doesn't eliminate the risk. They're not the same thing.