Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PM-5693] CryptoService using memfd_secret on Linux #7

Open
wants to merge 72 commits into
base: main
Choose a base branch
from

Conversation

dani-garcia
Copy link
Member

🎟️ Tracking

https://bitwarden.atlassian.net/browse/PM-5693

📔 Objective

Migrated from bitwarden/sdk-sm#979

I've been looking into using memfd_secret to protect keys in memory on Linux.

The memfd_secret API is similar to malloc in that we get some memory range allocated for us, but this memory is only visible to the process that has access to the file descriptor, which should protect us from any memory dumping from anything other than a kernel driver.

https://man.archlinux.org/man/memfd_secret.2.en

Using this requires changing how we store keys, so this is the perfect moment to go through removing the raw keys from the API surface.

Changes

  • Added a KeyRef trait and SymmetricKeyRef/AsymmetricKeyRef subtraits to represent the key types. Also a small macro to quickly implement them. KeyRef implementations are user defined types that implements Eq+Hash+Copy, and don't contain any key material
  • KeyEncryptable/KeyDecryptable are now Decryptable/Encryptable, and instead of taking the key as parameter, they take a CryptoServiceContext and a KeyRef. This way keys are never exposed to the clients.
  • KeyLocator trait is replaced by UsesKey. This trait only returns the KeyRef of the key required to decrypt it, instead of fetching the key itself.
  • Created a KeyStore trait, with some simple CRUD methods. We have two implementations, one based on memfd_secret for Linux, and another one based on a Rust boxed slice, that also applies mlock if possible.
  • Because both mlock and memfd_secret apply their protection over a specified memory area, we need a Rust data structure that is laid out linearly and also doesn't reallocate by itself. Also because memfd_secret allocates the memory for us, we can't use a std type like Vec (reason is the Vec must be allocated by the Rust allocator [ref]). This basically only leaves us with a Rust slice, on top of which we'd need to implement insert/get/delete. To allow for reuse and easier testing I've created SliceKeyStore, which implements the CRUD methods from the KeyStore trait on top of a plain slice. Then the actual mlock and memfd_secret implementations just need to implement a trait casting their data to a slice. The data is stored as a slice of Option<(KeyRef, KeyMaterial)>, and the keys are unique and sorted in the slice for easier lookups.
  • Created CryptoService, which contains the KeyStores and some encrypt/decrypt utility functions. From a CryptoService you can also initialize a CryptoServiceContext
  • A CryptoServiceContext contains a read only view of the keys inside the CryptoService, plus a read-write ephemeral key store, for use by Decryptable/Encryptable implementations when they need to temporarily store some keys (Cipher keys, attachment keys, send keys...).

Migrated the codebase to use these changes in a separate PR: bitwarden/sdk-sm#1117

📸 Screenshots

⏰ Reminders before review

  • Contributor guidelines followed
  • All formatters and local linters executed and passed
  • Written new unit and / or integration tests where applicable
  • Protected functional changes with optionality (feature flags)
  • Used internationalization (i18n) for all UI strings
  • CI builds passed
  • Communicated to DevOps any deployment requirements
  • Updated any necessary documentation (Confluence, contributing docs) or informed the documentation
    team

🦮 Reviewer guidelines

  • 👍 (:+1:) or similar for great changes
  • 📝 (:memo:) or ℹ️ (:information_source:) for notes or general info
  • ❓ (:question:) for questions
  • 🤔 (:thinking:) or 💭 (:thought_balloon:) for more open inquiry that's not quite a confirmed
    issue and could potentially benefit from discussion
  • 🎨 (:art:) for suggestions / improvements
  • ❌ (:x:) or ⚠️ (:warning:) for more significant problems or concerns needing attention
  • 🌱 (:seedling:) or ♻️ (:recycle:) for future improvements or indications of technical debt
  • ⛏ (:pick:) for minor or nitpick changes

@dani-garcia
Copy link
Member Author

dani-garcia commented Dec 12, 2024

I've made a few changes while documenting this a bit, plus some of the review suggestions are included here as well:

  • memsec is disabled on windows for now, and we're not using my fork
  • There's been a few renames:
    • CryptoService -> KeyStore. to move away from the service name
    • Keys -> KeyStoreInner. This only exists to be wrapped with the RwLock, and everything in KeyStore is pretty much delegated to it, so it seems like a better name.
    • CryptoServiceContext -> KeyStoreContext. To match the service rename
    • KeyStore -> StoreBackend. To signify there are different backends per platform
    • LinuxMemfdSecretKeyStore -> LinuxMemfdSecretBackend and RustStore -> RustBackend, to match the trait above.
  • Changed the generic params of the KeyStore to be a single KeyRefs, which makes the generic types everywhere much less verbose (Instead of having to do <SymmetricKeyRef, AsymmetricKeyRef> everywhere you can do <KeyRefs>.
  • Removed the generic type over KeyStoreContext global keys, and moved it to an enum.

The next steps for the crypto refactor after this are:

  • Move all the Encryptable implementations to the new trait: [PM-5693] Migrate SDK to CryptoService #8
  • Migrate existing APIs that deal with SymmetricCryptoKey/AsymmetricCryptoKey directly to deal with references instead (init_crypto functions, MasterKey, UserKey, PinKey, Key fingerprint etc).
  • Remove uses of KeyStoreContext::get_key|set_key from the non crypto crates, which shouldn't have any uses left after the previous step (maybe?).
  • Once no other uses of SymmetricCryptoKey/AsymmetricCryptoKey are left, make them private to bitwarden-crypto
  • Remove the now unnecessary boxing around SymmetricCryptoKey/AsymmetricCryptoKey

Some future API improvements:

  • Improve the design around local keys, instead of accepting any user type, create an opaque token and return it in the functions, so the only way for a user to get a valid local key ref is to obtain it from a function.
  • Currently the context implementation can be read-only or read-write over the global keys, depending on how it's initialized, and it's behavior changes slightly in both cases. I think it might be a better design to instead do something like SQL transactions. Contexts are always read-only, but they have a commit function that consumes them and applies the changes to the global KeyStore

@Hinton Hinton self-requested a review January 13, 2025 11:28
Copy link
Member

@Hinton Hinton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need to review the rust backend.

crates/bitwarden-crypto/Cargo.toml Outdated Show resolved Hide resolved
crates/bitwarden-crypto/Cargo.toml Outdated Show resolved Hide resolved
crates/bitwarden-crypto/src/error.rs Outdated Show resolved Hide resolved
crates/bitwarden-crypto/src/lib.rs Outdated Show resolved Hide resolved
crates/bitwarden-crypto/src/keys/mod.rs Outdated Show resolved Hide resolved

// For Send+Sync to be safe, we need to ensure that the memory is only accessed mutably from one
// thread. To do this, we have to make sure that any funcion in `MemfdSecretImplKeyData` that
// accesses the pointer mutably is defined as &mut self, and that the pointer is never copied or
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// accesses the pointer mutably is defined as &mut self, and that the pointer is never copied or
// accesses the pointer mutably is defined as `&mut self`, and that the pointer is never copied or

Comment on lines 51 to 52
let ptr: NonNull<[u8]> = memsec::memfd_secret_sized(capacity * entry_size)
.expect("memfd_secret_sized failed");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would memfd fail? If we run out of memory?

Copy link
Member Author

@dani-garcia dani-garcia Jan 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs (https://man7.org/linux/man-pages/man2/memfd_secret.2.html):

       EINVAL flags included unknown bits.

       EMFILE The per-process limit on the number of open file
              descriptors has been reached.

       EMFILE The system-wide limit on the total number of open files
              has been reached.

       ENOMEM There was insufficient memory to create a new anonymous
              file.

       ENOSYS memfd_secret() is not implemented on this architecture, or
              has not been enabled on the kernel command-line with
              secretmem_enable=1.

So when the system is out of memory or the process has reached the file descriptor limit, or if the feature is disabled and not supported (but we check for that the first time)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we provide a better error message? This seems like an unrecoverable situation but it also sounds like we need to test what happens if we panic on desktop with napi?

Comment on lines 54 to 61
// Initialize the array with Nones using MaybeUninit
let uninit_slice: &mut [MaybeUninit<_>] = std::slice::from_raw_parts_mut(
ptr.as_ptr() as *mut MaybeUninit<Option<(Key, Key::KeyValue)>>,
capacity,
);
for elem in uninit_slice {
elem.write(None);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines 69 to 70
// SAFETY: The pointer is valid and points to a valid slice of the correct size.
// This function is &self so it only takes a immutable *const pointer.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on how we know it's valid?

return Box::new(key_store);
}

Box::new(rust::RustBackend::new().expect("RustKeyStore should always be available"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Box::new(rust::RustBackend::new().expect("RustKeyStore should always be available"))
Box::new(rust::RustBackend::new().expect("RustKeyStore to always be available"))

Copy link
Member

@Hinton Hinton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments, still need to review slice_backend properly.

crates/bitwarden-crypto/src/store/mod.rs Outdated Show resolved Hide resolved
Comment on lines 102 to 140
/// Initiate an encryption/decryption context. This context will have read only access to the
/// global keys, and will have its own local key stores with read/write access. This
/// context-local store will be cleared up when the context is dropped.
///
/// This is an advanced API, use with care. Prefer to instead use
/// `encrypt`/`decrypt`/`encrypt_list`/`decrypt_list` methods.
///
/// One of the pitfalls of the current implementations is that keys stored in the context-local
/// store only get cleared automatically when the context is dropped, and not between
/// operations. This means that if you are using the same context for multiple operations,
/// you may want to clear it manually between them.
pub fn context(&'_ self) -> KeyStoreContext<'_, Refs> {
KeyStoreContext {
global_keys: GlobalKeys::ReadOnly(self.inner.read().expect("RwLock is poisoned")),
local_symmetric_keys: create_store(),
local_asymmetric_keys: create_store(),
_phantom: std::marker::PhantomData,
}
}

/// Initiate an encryption/decryption context. This context will have MUTABLE access to the
/// global keys, and will have its own local key stores with read/write access. This
/// context-local store will be cleared up when the context is dropped.
///
/// This is an advanced API, use with care and ONLY when needing to modify the global keys.
///
/// The same pitfalls as `context` apply here, but with the added risk of accidentally
/// modifying the global keys and leaving the store in an inconsistent state.
///
/// TODO: We should work towards making this pub(crate), and instead providing a safe API for
/// modifying the global keys. (i.e. `derive_master_key`, `derive_user_key`, etc.)
pub fn context_mut(&'_ self) -> KeyStoreContext<'_, Refs> {
KeyStoreContext {
global_keys: GlobalKeys::ReadWrite(self.inner.write().expect("RwLock is poisoned")),
local_symmetric_keys: create_store(),
local_asymmetric_keys: create_store(),
_phantom: std::marker::PhantomData,
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Can we provide some examples of when it's alright to use these APIs?

crates/bitwarden-crypto/src/store/mod.rs Show resolved Hide resolved

// In this case, the minimum chunk size is 50. This was chosen pretty arbitrarily,
// but it seems to work well in practice.
usize::max(1 + len / rayon::current_num_threads(), 50)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: Magic number, extract it as a constant with a good name?

crates/bitwarden-crypto/src/store/mod.rs Show resolved Hide resolved

// Note: munlock is zeroing the memory, which leaves the data in an undefined
// state, so we set it to None/Default again to avoid UB in the Drop implementation.
let uninit_slice: &mut [MaybeUninit<_>] = std::slice::from_raw_parts_mut(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Just checking but we fulfill all safety criteria defined in https://doc.rust-lang.org/std/slice/fn.from_raw_parts_mut.html?

Comment on lines 69 to 77
// Note: munlock is zeroing the memory, which leaves the data in an undefined
// state, so we set it to None/Default again to avoid UB in the Drop implementation.
let uninit_slice: &mut [MaybeUninit<_>] = std::slice::from_raw_parts_mut(
data.as_mut_ptr() as *mut MaybeUninit<T>,
data.len(),
);
for elem in uninit_slice {
elem.write(T::default());
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Why is munlock zeroizing memory undefined state?

And do we have documentation that we're re-initializing it correctly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the items inside the store can implement Drop themselves, and if they try to access their values after they have been zeroed it would cause UB.

}

impl<Key: KeyRef, Data: SliceLike<Key>> StoreBackend<Key> for SliceBackend<Key, Data> {
fn insert(&mut self, key_ref: Key, key: Key::KeyValue) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue: This sounds like an upsert. I think we should name it accurately if that is the expected behaviour.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I based it on the std HashMap API, where the insert does insert and replace. I'll rename it

Comment on lines 32 to 44
/// This represents a key store over an arbitrary fixed size slice.
/// This is meant to abstract over the different ways to store keys in memory, whether we're
/// using a Box<[u8]> or a NonNull<u8>, regardless of if that memory was allocated by Rust or not.
///
/// Internally this is represented as a slice of `Option<(Key, Key::KeyValue)>`
/// and elements are sorted based on Key for performance. In essence, this is almost a homegrown
/// `HashMap`.
///
/// The reason why we're not using a Rust collection like `Vec` or `HashMap` is that those types
/// expect their memory to be allocated by Rust, and they will deallocate/reallocate it as needed.
/// That will not work for our usecases where we want to have control over allocations/deallocations
/// and where some of our strategies rely on using system-allocated secure memory for the storage,
/// like the Linux-only `memfd_secret` API.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: We seem to rely on this being an ordered list but it's not documented? Can we add this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering is an implementation detail of the backend, I chose it because it allows us to do binary searches for lookup performance. It shouldn't be exposed outside its own API.

That said it would be good to document it yes.

@dani-garcia
Copy link
Member Author

Split the secure storage implementations to #125 so this is easier to review and merge.

@dani-garcia dani-garcia requested a review from Hinton January 22, 2025 14:43
}
}

// Key::KeyValue already implements ZeroizeOnDrop,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Key::KeyValue already implements ZeroizeOnDrop,
// [Key::KeyValue] already implements ZeroizeOnDrop,


/// This trait represents a platform that can store and return keys. If possible,
/// it will try to enable as many security protections on the keys as it can.
/// The keys themselves implement ZeroizeOnDrop, so the store will only need to make sure
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The keys themselves implement ZeroizeOnDrop, so the store will only need to make sure
/// The keys themselves implement [ZeroizeOnDrop], so the store will only need to make sure

pub trait StoreBackend<Key: KeyId>: ZeroizeOnDrop + Send + Sync {
/// Inserts a key into the store. If the key already exists, it will be replaced.
fn upsert(&mut self, key_id: Key, key: Key::KeyValue);
/// Retrieves a key from the store.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: New line between functions?

/// This will usually be accessed from an implementation of [crate::Decryptable] or
/// [crate::Encryptable], but can also be obtained through [super::KeyStore::context]
///
/// This context contains access to the user keys stored in the [super::KeyStore] (sometimes refered
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// This context contains access to the user keys stored in the [super::KeyStore] (sometimes refered
/// This context contains access to the user keys stored in the [super::KeyStore] (sometimes referred

// A KeyStoreContext is usually limited to a read only access to the global keys,
// which allows us to have multiple read only contexts at the same time and do multitheaded
// encryption/decryption. We also have the option to create a read/write context, which allows us to
// modify the global keys, but only allows one context at a time. This is controlled by a RWLock on
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// modify the global keys, but only allows one context at a time. This is controlled by a RWLock on
// modify the global keys, but only allows one context at a time. This is controlled by a [RwLock] on

/// Represents a key identifier that can be used to identify cryptographic keys in the
/// key store. It is used to avoid exposing the key material directly in the public API.
///
/// This trait is user-implemented, and our recommended implementation is using enums with variants
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// This trait is user-implemented, and our recommended implementation is using enums with variants
/// This trait is user-implemented, and the recommended implementation is using enums with variants

Comment on lines +50 to +68
impl<Ids: KeyIds> Encryptable<Ids, Ids::Symmetric, EncString> for &str {
fn encrypt(
&self,
ctx: &mut KeyStoreContext<Ids>,
key: Ids::Symmetric,
) -> Result<EncString, CryptoError> {
self.as_bytes().encrypt(ctx, key)
}
}

impl<Ids: KeyIds> Encryptable<Ids, Ids::Asymmetric, AsymmetricEncString> for &str {
fn encrypt(
&self,
ctx: &mut KeyStoreContext<Ids>,
key: Ids::Asymmetric,
) -> Result<AsymmetricEncString, CryptoError> {
self.as_bytes().encrypt(ctx, key)
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we can generalize this code slightly.

Suggested change
impl<Ids: KeyIds> Encryptable<Ids, Ids::Symmetric, EncString> for &str {
fn encrypt(
&self,
ctx: &mut KeyStoreContext<Ids>,
key: Ids::Symmetric,
) -> Result<EncString, CryptoError> {
self.as_bytes().encrypt(ctx, key)
}
}
impl<Ids: KeyIds> Encryptable<Ids, Ids::Asymmetric, AsymmetricEncString> for &str {
fn encrypt(
&self,
ctx: &mut KeyStoreContext<Ids>,
key: Ids::Asymmetric,
) -> Result<AsymmetricEncString, CryptoError> {
self.as_bytes().encrypt(ctx, key)
}
}
impl<Ids, K, E> Encryptable<Ids, K, E> for &str
where
Ids: KeyIds,
K: KeyId,
E: EncStringType,
[u8]: Encryptable<Ids, K, E>,
{
fn encrypt(&self, ctx: &mut KeyStoreContext<Ids>, key: K) -> Result<E, CryptoError> {
self.as_bytes().encrypt(ctx, key)
}
}

And define a common trait for EncString.

pub trait EncStringType {}
impl EncStringType for EncString {}
impl EncStringType for AsymmetricEncString {}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some quick module docs? I.e. //!

inner: Arc<RwLock<KeyStoreInner<Ids>>>,
}

impl<Ids: KeyIds> std::fmt::Debug for KeyStore<Ids> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
impl<Ids: KeyIds> std::fmt::Debug for KeyStore<Ids> {
// [KeyStore] contains sensitive data, provide a dummy [Debug] implementation.
impl<Ids: KeyIds> std::fmt::Debug for KeyStore<Ids> {

/// * `new_key_id` - The key id where the decrypted key will be stored. If it already exists, it
/// will be overwritten
/// * `encrypted_key` - The key to decrypt
pub fn decrypt_symmetric_key_with_symmetric_key(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the naming here is slightly misleading. We're not decrypting it we're adding it to the context?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants