-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PM-5693] CryptoService using memfd_secret on Linux #7
base: main
Are you sure you want to change the base?
Conversation
Otherwise some Drop implementations can crash
I've made a few changes while documenting this a bit, plus some of the review suggestions are included here as well:
The next steps for the crypto refactor after this are:
Some future API improvements:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still need to review the rust backend.
|
||
// For Send+Sync to be safe, we need to ensure that the memory is only accessed mutably from one | ||
// thread. To do this, we have to make sure that any funcion in `MemfdSecretImplKeyData` that | ||
// accesses the pointer mutably is defined as &mut self, and that the pointer is never copied or |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// accesses the pointer mutably is defined as &mut self, and that the pointer is never copied or | |
// accesses the pointer mutably is defined as `&mut self`, and that the pointer is never copied or |
let ptr: NonNull<[u8]> = memsec::memfd_secret_sized(capacity * entry_size) | ||
.expect("memfd_secret_sized failed"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When would memfd fail? If we run out of memory?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to the docs (https://man7.org/linux/man-pages/man2/memfd_secret.2.html):
EINVAL flags included unknown bits.
EMFILE The per-process limit on the number of open file
descriptors has been reached.
EMFILE The system-wide limit on the total number of open files
has been reached.
ENOMEM There was insufficient memory to create a new anonymous
file.
ENOSYS memfd_secret() is not implemented on this architecture, or
has not been enabled on the kernel command-line with
secretmem_enable=1.
So when the system is out of memory or the process has reached the file descriptor limit, or if the feature is disabled and not supported (but we check for that the first time)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we provide a better error message? This seems like an unrecoverable situation but it also sounds like we need to test what happens if we panic on desktop with napi?
// Initialize the array with Nones using MaybeUninit | ||
let uninit_slice: &mut [MaybeUninit<_>] = std::slice::from_raw_parts_mut( | ||
ptr.as_ptr() as *mut MaybeUninit<Option<(Key, Key::KeyValue)>>, | ||
capacity, | ||
); | ||
for elem in uninit_slice { | ||
elem.write(None); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is probably unsafe? https://doc.rust-lang.org/std/slice/fn.from_raw_parts_mut.html
// SAFETY: The pointer is valid and points to a valid slice of the correct size. | ||
// This function is &self so it only takes a immutable *const pointer. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you elaborate on how we know it's valid?
return Box::new(key_store); | ||
} | ||
|
||
Box::new(rust::RustBackend::new().expect("RustKeyStore should always be available")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Box::new(rust::RustBackend::new().expect("RustKeyStore should always be available")) | |
Box::new(rust::RustBackend::new().expect("RustKeyStore to always be available")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some more comments, still need to review slice_backend properly.
/// Initiate an encryption/decryption context. This context will have read only access to the | ||
/// global keys, and will have its own local key stores with read/write access. This | ||
/// context-local store will be cleared up when the context is dropped. | ||
/// | ||
/// This is an advanced API, use with care. Prefer to instead use | ||
/// `encrypt`/`decrypt`/`encrypt_list`/`decrypt_list` methods. | ||
/// | ||
/// One of the pitfalls of the current implementations is that keys stored in the context-local | ||
/// store only get cleared automatically when the context is dropped, and not between | ||
/// operations. This means that if you are using the same context for multiple operations, | ||
/// you may want to clear it manually between them. | ||
pub fn context(&'_ self) -> KeyStoreContext<'_, Refs> { | ||
KeyStoreContext { | ||
global_keys: GlobalKeys::ReadOnly(self.inner.read().expect("RwLock is poisoned")), | ||
local_symmetric_keys: create_store(), | ||
local_asymmetric_keys: create_store(), | ||
_phantom: std::marker::PhantomData, | ||
} | ||
} | ||
|
||
/// Initiate an encryption/decryption context. This context will have MUTABLE access to the | ||
/// global keys, and will have its own local key stores with read/write access. This | ||
/// context-local store will be cleared up when the context is dropped. | ||
/// | ||
/// This is an advanced API, use with care and ONLY when needing to modify the global keys. | ||
/// | ||
/// The same pitfalls as `context` apply here, but with the added risk of accidentally | ||
/// modifying the global keys and leaving the store in an inconsistent state. | ||
/// | ||
/// TODO: We should work towards making this pub(crate), and instead providing a safe API for | ||
/// modifying the global keys. (i.e. `derive_master_key`, `derive_user_key`, etc.) | ||
pub fn context_mut(&'_ self) -> KeyStoreContext<'_, Refs> { | ||
KeyStoreContext { | ||
global_keys: GlobalKeys::ReadWrite(self.inner.write().expect("RwLock is poisoned")), | ||
local_symmetric_keys: create_store(), | ||
local_asymmetric_keys: create_store(), | ||
_phantom: std::marker::PhantomData, | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: Can we provide some examples of when it's alright to use these APIs?
|
||
// In this case, the minimum chunk size is 50. This was chosen pretty arbitrarily, | ||
// but it seems to work well in practice. | ||
usize::max(1 + len / rayon::current_num_threads(), 50) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
issue: Magic number, extract it as a constant with a good name?
|
||
// Note: munlock is zeroing the memory, which leaves the data in an undefined | ||
// state, so we set it to None/Default again to avoid UB in the Drop implementation. | ||
let uninit_slice: &mut [MaybeUninit<_>] = std::slice::from_raw_parts_mut( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: Just checking but we fulfill all safety criteria defined in https://doc.rust-lang.org/std/slice/fn.from_raw_parts_mut.html?
// Note: munlock is zeroing the memory, which leaves the data in an undefined | ||
// state, so we set it to None/Default again to avoid UB in the Drop implementation. | ||
let uninit_slice: &mut [MaybeUninit<_>] = std::slice::from_raw_parts_mut( | ||
data.as_mut_ptr() as *mut MaybeUninit<T>, | ||
data.len(), | ||
); | ||
for elem in uninit_slice { | ||
elem.write(T::default()); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: Why is munlock zeroizing memory undefined state?
And do we have documentation that we're re-initializing it correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because the items inside the store can implement Drop themselves, and if they try to access their values after they have been zeroed it would cause UB.
} | ||
|
||
impl<Key: KeyRef, Data: SliceLike<Key>> StoreBackend<Key> for SliceBackend<Key, Data> { | ||
fn insert(&mut self, key_ref: Key, key: Key::KeyValue) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
issue: This sounds like an upsert. I think we should name it accurately if that is the expected behaviour.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I based it on the std HashMap API, where the insert
does insert and replace. I'll rename it
/// This represents a key store over an arbitrary fixed size slice. | ||
/// This is meant to abstract over the different ways to store keys in memory, whether we're | ||
/// using a Box<[u8]> or a NonNull<u8>, regardless of if that memory was allocated by Rust or not. | ||
/// | ||
/// Internally this is represented as a slice of `Option<(Key, Key::KeyValue)>` | ||
/// and elements are sorted based on Key for performance. In essence, this is almost a homegrown | ||
/// `HashMap`. | ||
/// | ||
/// The reason why we're not using a Rust collection like `Vec` or `HashMap` is that those types | ||
/// expect their memory to be allocated by Rust, and they will deallocate/reallocate it as needed. | ||
/// That will not work for our usecases where we want to have control over allocations/deallocations | ||
/// and where some of our strategies rely on using system-allocated secure memory for the storage, | ||
/// like the Linux-only `memfd_secret` API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
question: We seem to rely on this being an ordered list but it's not documented? Can we add this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ordering is an implementation detail of the backend, I chose it because it allows us to do binary searches for lookup performance. It shouldn't be exposed outside its own API.
That said it would be good to document it yes.
Split the secure storage implementations to #125 so this is easier to review and merge. |
} | ||
} | ||
|
||
// Key::KeyValue already implements ZeroizeOnDrop, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// Key::KeyValue already implements ZeroizeOnDrop, | |
// [Key::KeyValue] already implements ZeroizeOnDrop, |
|
||
/// This trait represents a platform that can store and return keys. If possible, | ||
/// it will try to enable as many security protections on the keys as it can. | ||
/// The keys themselves implement ZeroizeOnDrop, so the store will only need to make sure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// The keys themselves implement ZeroizeOnDrop, so the store will only need to make sure | |
/// The keys themselves implement [ZeroizeOnDrop], so the store will only need to make sure |
pub trait StoreBackend<Key: KeyId>: ZeroizeOnDrop + Send + Sync { | ||
/// Inserts a key into the store. If the key already exists, it will be replaced. | ||
fn upsert(&mut self, key_id: Key, key: Key::KeyValue); | ||
/// Retrieves a key from the store. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggestion: New line between functions?
/// This will usually be accessed from an implementation of [crate::Decryptable] or | ||
/// [crate::Encryptable], but can also be obtained through [super::KeyStore::context] | ||
/// | ||
/// This context contains access to the user keys stored in the [super::KeyStore] (sometimes refered |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// This context contains access to the user keys stored in the [super::KeyStore] (sometimes refered | |
/// This context contains access to the user keys stored in the [super::KeyStore] (sometimes referred |
// A KeyStoreContext is usually limited to a read only access to the global keys, | ||
// which allows us to have multiple read only contexts at the same time and do multitheaded | ||
// encryption/decryption. We also have the option to create a read/write context, which allows us to | ||
// modify the global keys, but only allows one context at a time. This is controlled by a RWLock on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
// modify the global keys, but only allows one context at a time. This is controlled by a RWLock on | |
// modify the global keys, but only allows one context at a time. This is controlled by a [RwLock] on |
/// Represents a key identifier that can be used to identify cryptographic keys in the | ||
/// key store. It is used to avoid exposing the key material directly in the public API. | ||
/// | ||
/// This trait is user-implemented, and our recommended implementation is using enums with variants |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// This trait is user-implemented, and our recommended implementation is using enums with variants | |
/// This trait is user-implemented, and the recommended implementation is using enums with variants |
impl<Ids: KeyIds> Encryptable<Ids, Ids::Symmetric, EncString> for &str { | ||
fn encrypt( | ||
&self, | ||
ctx: &mut KeyStoreContext<Ids>, | ||
key: Ids::Symmetric, | ||
) -> Result<EncString, CryptoError> { | ||
self.as_bytes().encrypt(ctx, key) | ||
} | ||
} | ||
|
||
impl<Ids: KeyIds> Encryptable<Ids, Ids::Asymmetric, AsymmetricEncString> for &str { | ||
fn encrypt( | ||
&self, | ||
ctx: &mut KeyStoreContext<Ids>, | ||
key: Ids::Asymmetric, | ||
) -> Result<AsymmetricEncString, CryptoError> { | ||
self.as_bytes().encrypt(ctx, key) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe we can generalize this code slightly.
impl<Ids: KeyIds> Encryptable<Ids, Ids::Symmetric, EncString> for &str { | |
fn encrypt( | |
&self, | |
ctx: &mut KeyStoreContext<Ids>, | |
key: Ids::Symmetric, | |
) -> Result<EncString, CryptoError> { | |
self.as_bytes().encrypt(ctx, key) | |
} | |
} | |
impl<Ids: KeyIds> Encryptable<Ids, Ids::Asymmetric, AsymmetricEncString> for &str { | |
fn encrypt( | |
&self, | |
ctx: &mut KeyStoreContext<Ids>, | |
key: Ids::Asymmetric, | |
) -> Result<AsymmetricEncString, CryptoError> { | |
self.as_bytes().encrypt(ctx, key) | |
} | |
} | |
impl<Ids, K, E> Encryptable<Ids, K, E> for &str | |
where | |
Ids: KeyIds, | |
K: KeyId, | |
E: EncStringType, | |
[u8]: Encryptable<Ids, K, E>, | |
{ | |
fn encrypt(&self, ctx: &mut KeyStoreContext<Ids>, key: K) -> Result<E, CryptoError> { | |
self.as_bytes().encrypt(ctx, key) | |
} | |
} |
And define a common trait for EncString
.
pub trait EncStringType {}
impl EncStringType for EncString {}
impl EncStringType for AsymmetricEncString {}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add some quick module docs? I.e. //!
inner: Arc<RwLock<KeyStoreInner<Ids>>>, | ||
} | ||
|
||
impl<Ids: KeyIds> std::fmt::Debug for KeyStore<Ids> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
impl<Ids: KeyIds> std::fmt::Debug for KeyStore<Ids> { | |
// [KeyStore] contains sensitive data, provide a dummy [Debug] implementation. | |
impl<Ids: KeyIds> std::fmt::Debug for KeyStore<Ids> { |
/// * `new_key_id` - The key id where the decrypted key will be stored. If it already exists, it | ||
/// will be overwritten | ||
/// * `encrypted_key` - The key to decrypt | ||
pub fn decrypt_symmetric_key_with_symmetric_key( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if the naming here is slightly misleading. We're not decrypting it we're adding it to the context?
🎟️ Tracking
https://bitwarden.atlassian.net/browse/PM-5693
📔 Objective
Migrated from bitwarden/sdk-sm#979
I've been looking into using
memfd_secret
to protect keys in memory on Linux.The
memfd_secret
API is similar to malloc in that we get some memory range allocated for us, but this memory is only visible to the process that has access to the file descriptor, which should protect us from any memory dumping from anything other than a kernel driver.https://man.archlinux.org/man/memfd_secret.2.en
Using this requires changing how we store keys, so this is the perfect moment to go through removing the raw keys from the API surface.
Changes
KeyRef
trait andSymmetricKeyRef
/AsymmetricKeyRef
subtraits to represent the key types. Also a small macro to quickly implement them.KeyRef
implementations are user defined types that implementsEq+Hash+Copy
, and don't contain any key materialKeyEncryptable/KeyDecryptable
are nowDecryptable/Encryptable
, and instead of taking the key as parameter, they take aCryptoServiceContext
and aKeyRef
. This way keys are never exposed to the clients.KeyLocator
trait is replaced byUsesKey
. This trait only returns theKeyRef
of the key required to decrypt it, instead of fetching the key itself.KeyStore
trait, with some simple CRUD methods. We have two implementations, one based onmemfd_secret
for Linux, and another one based on a Rust boxed slice, that also appliesmlock
if possible.mlock
andmemfd_secret
apply their protection over a specified memory area, we need a Rust data structure that is laid out linearly and also doesn't reallocate by itself. Also becausememfd_secret
allocates the memory for us, we can't use a std type like Vec (reason is the Vec must be allocated by the Rust allocator [ref]). This basically only leaves us with a Rust slice, on top of which we'd need to implement insert/get/delete. To allow for reuse and easier testing I've createdSliceKeyStore
, which implements the CRUD methods from theKeyStore
trait on top of a plain slice. Then the actualmlock
andmemfd_secret
implementations just need to implement a trait casting their data to a slice. The data is stored as a slice ofOption<(KeyRef, KeyMaterial)>
, and the keys are unique and sorted in the slice for easier lookups.CryptoService
, which contains theKeyStore
s and some encrypt/decrypt utility functions. From aCryptoService
you can also initialize aCryptoServiceContext
CryptoServiceContext
contains a read only view of the keys inside theCryptoService
, plus a read-write ephemeral key store, for use byDecryptable/Encryptable
implementations when they need to temporarily store some keys (Cipher keys, attachment keys, send keys...).Migrated the codebase to use these changes in a separate PR: bitwarden/sdk-sm#1117
📸 Screenshots
⏰ Reminders before review
team
🦮 Reviewer guidelines
:+1:
) or similar for great changes:memo:
) or ℹ️ (:information_source:
) for notes or general info:question:
) for questions:thinking:
) or 💭 (:thought_balloon:
) for more open inquiry that's not quite a confirmedissue and could potentially benefit from discussion
:art:
) for suggestions / improvements:x:
) or:warning:
) for more significant problems or concerns needing attention:seedling:
) or ♻️ (:recycle:
) for future improvements or indications of technical debt:pick:
) for minor or nitpick changes