| Message ID | 20240528122352.2485958-5-Jason@zx2c4.com (mailing list archive) |
|---|---|
| State | Not Applicable |
| Delegated to: | Herbert Xu |
| Headers | show |
| Series | implement getrandom() in vDSO | expand |
On 5/28/24 5:19 AM, Jason A. Donenfeld wrote: > +/** > + * type vdso_kernel_ulong - unsigned long type that matches kernel's unsigned long s/type/typedef/ (for first "type" only) > + * > + * Data shared between userspace and the kernel must operate the same way in both 64-bit code and in > + * 32-bit compat code, over the same potentially 64-bit kernel. This type represents the size of an > + * unsigned long as used by kernel code. This isn't necessarily the same as an unsigned long as used > + * by userspace, however. > + * > + * +-------------------+-------------------+------------------+-------------------+ > + * | 32-bit userspace | 32-bit userspace | 64-bit userspace | 64-bit userspace | > + * | unsigned long | vdso_kernel_ulong | unsigned long | vdso_kernel_ulong | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + * | 32-bit kernel | ✓ same size | ✓ same size | > + * | unsigned long | | | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + * | 64-bit kernel | ✘ different size! | ✓ same size | ✓ same size | ✓ same size | > + * | unsigned long | | | | | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + */ > +#ifdef CONFIG_64BIT > +typedef u64 vdso_kernel_ulong; > +#else > +typedef u32 vdso_kernel_ulong; > +#endif
On 5/28/24 5:19 AM, Jason A. Donenfeld wrote: > +/** > + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. > + * @rng_info: Describes state of kernel RNG, memory shared with kernel. > + * @buffer: Destination buffer to fill with random bytes. > + * @len: Size of @buffer in bytes. > + * @flags: Zero or more GRND_* flags. > + * @opaque_state: Pointer to an opaque state area. > + * > + * This implements a "fast key erasure" RNG using ChaCha20, in the same way that the kernel's > + * getrandom() syscall does. It periodically reseeds its key from the kernel's RNG, at the same > + * schedule that the kernel's RNG is reseeded. If the kernel's RNG is not ready, then this always > + * calls into the syscall. > + * > + * @opaque_state *must* be allocated using the vgetrandom_alloc() syscall. Unless external locking > + * is used, one state must be allocated per thread, as it is not safe to call this function > + * concurrently with the same @opaque_state. However, it is safe to call this using the same > + * @opaque_state that is shared between main code and signal handling code, within the same thread. > + * > + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. * Returns: > + */
> On May 28, 2024, at 5:25 AM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Provide a generic C vDSO getrandom() implementation, which operates on > an opaque state returned by vgetrandom_alloc() and produces random bytes > the same way as getrandom(). This has a the API signature: > > ssize_t vgetrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state); > +/** > + * type vdso_kernel_ulong - unsigned long type that matches kernel's unsigned long > + * > + * Data shared between userspace and the kernel must operate the same way in both 64-bit code and in > + * 32-bit compat code, over the same potentially 64-bit kernel. This type represents the size of an > + * unsigned long as used by kernel code. This isn't necessarily the same as an unsigned long as used > + * by userspace, however. Why is this better than using plain u64? It’s certainly more complicated. It also rather fundamentally breaks CRIU on 32-bit userspace (although CRIU may well be unable to keep vgetrandom working after a restore onto a different kernel anyway). Admittedly 32-bit userspace is a slowly dying breed, but still. > + * > + * +-------------------+-------------------+------------------+-------------------+ > + * | 32-bit userspace | 32-bit userspace | 64-bit userspace | 64-bit userspace | > + * | unsigned long | vdso_kernel_ulong | unsigned long | vdso_kernel_ulong | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + * | 32-bit kernel | ✓ same size | ✓ same size | > + * | unsigned long | | | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + * | 64-bit kernel | ✘ different size! | ✓ same size | ✓ same size | ✓ same size | > + * | unsigned long | | | | | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + */ > +#ifdef CONFIG_64BIT > +typedef u64 vdso_kernel_ulong; > +#else > +typedef u32 vdso_kernel_ulong; > +#endif > + > +#endif /* __VDSO_TYPES_H */ > diff --git a/lib/vdso/getrandom.c b/lib/vdso/getrandom.c > new file mode 100644 > index 000000000000..4d9bb59985f8 > --- /dev/null > +++ b/lib/vdso/getrandom.c > @@ -0,0 +1,226 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright (C) 2022 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved. > + */ > + > +#include <linux/cache.h> > +#include <linux/kernel.h> > +#include <linux/time64.h> > +#include <vdso/datapage.h> > +#include <vdso/getrandom.h> > +#include <asm/vdso/getrandom.h> > +#include <asm/vdso/vsyscall.h> > + > +#define MEMCPY_AND_ZERO_SRC(type, dst, src, len) do { \ > + while (len >= sizeof(type)) { \ > + __put_unaligned_t(type, __get_unaligned_t(type, src), dst); \ > + __put_unaligned_t(type, 0, src); \ > + dst += sizeof(type); \ > + src += sizeof(type); \ > + len -= sizeof(type); \ > + } \ > +} while (0) > + > +static void memcpy_and_zero_src(void *dst, void *src, size_t len) > +{ > + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { > + if (IS_ENABLED(CONFIG_64BIT)) > + MEMCPY_AND_ZERO_SRC(u64, dst, src, len); > + MEMCPY_AND_ZERO_SRC(u32, dst, src, len); > + MEMCPY_AND_ZERO_SRC(u16, dst, src, len); > + } > + MEMCPY_AND_ZERO_SRC(u8, dst, src, len); > +} > + > +/** > + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. > + * @rng_info: Describes state of kernel RNG, memory shared with kernel. > + * @buffer: Destination buffer to fill with random bytes. > + * @len: Size of @buffer in bytes. > + * @flags: Zero or more GRND_* flags. > + * @opaque_state: Pointer to an opaque state area. > + * > + * This implements a "fast key erasure" RNG using ChaCha20, in the same way that the kernel's > + * getrandom() syscall does. It periodically reseeds its key from the kernel's RNG, at the same > + * schedule that the kernel's RNG is reseeded. If the kernel's RNG is not ready, then this always > + * calls into the syscall. > + * > + * @opaque_state *must* be allocated using the vgetrandom_alloc() syscall. Unless external locking > + * is used, one state must be allocated per thread, as it is not safe to call this function > + * concurrently with the same @opaque_state. However, it is safe to call this using the same > + * @opaque_state that is shared between main code and signal handling code, within the same thread. > + * > + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. > + */ > +static __always_inline ssize_t > +__cvdso_getrandom_data(const struct vdso_rng_data *rng_info, void *buffer, size_t len, > + unsigned int flags, void *opaque_state) I don’t love this function signature. I generally think that, if you’re going to have user code pass a pointer to kernel code, either make the buffer have a well defined, constant size or pass a length. As it stands, one cannot locally prove that user code that calls it is memory-safe. In fact, any caller that has the misfortune of running under CRIU is *not* memory safe if CRIU allows the vDSO to be preserved. Ouch. (CRIU has some special code for this. I'm not 100% clear on all the details.) One could maybe sort of get away with treating the provided opaque_state as a completely opaque value and not a pointer, but then the mechanism for allocating these states should be adjusted accordingly. One thing that occurs to me is that, if this thing were to be made CRIU-safe, the buffer could have a magic number that changes any time the data structure changes, and the vDSO could check, at the beginning and end of the call, that the magic number is correct. Doing this would require using a special VMA type instead of just a wipe-on-fork mapping, which could plausibly be a good thing anyway. (Hmm, we don't just want WIPEONFORK. We should probably also wipe on swap-out and, more importantly, we should absolutely wipe on any sort of CRIU-style checkpointing. Perhaps a special VMA would be a good thing for multiple reasons. > +{ > + ssize_t ret = min_t(size_t, INT_MAX & PAGE_MASK /* = MAX_RW_COUNT */, len); > + struct vgetrandom_state *state = opaque_state; > + size_t batch_len, nblocks, orig_len = len; > + unsigned long current_generation; > + void *orig_buffer = buffer; > + u32 counter[2] = { 0 }; > + bool in_use, have_retried = false; > + > + /* The state must not straddle a page, since pages can be zeroed at any time. */ > + if (unlikely(((unsigned long)opaque_state & ~PAGE_MASK) + sizeof(*state) > PAGE_SIZE)) > + goto fallback_syscall; This is weird. Either the provided pointer is valid or it isn’t. Reasonable outcomes are a segfault if the pointer is bad or success (or fallback if needed for some reason) if the pointer is good. Why is there specific code to catch a specific sort of pointer screwup here?
Jason! On Tue, May 28 2024 at 14:19, Jason A. Donenfeld wrote: > diff --git a/include/vdso/getrandom.h b/include/vdso/getrandom.h > index e3ceb1976386..7dc93d5f72dc 100644 > --- a/include/vdso/getrandom.h > +++ b/include/vdso/getrandom.h > @@ -6,11 +6,39 @@ > #ifndef _VDSO_GETRANDOM_H > #define _VDSO_GETRANDOM_H > > +#include <crypto/chacha.h> Can you please split the required defines into a seperate header preferrably in include/vdso/ and include that from crypto/chacha.h The point is that VDSO is very intentionally not using anything outside include/uapi/ and include/vdso/ except for include/linux/compiler.h and include/linux/types.h. We've had too much trouble of random include chains which magically break the build dependent on architectures and configurations. VDSO is a userspace library after all. > +#include <vdso/types.h> > + > /** > * struct vgetrandom_state - State used by vDSO getrandom() and allocated by vgetrandom_alloc(). > * > - * Currently empty, as the vDSO getrandom() function has not yet been implemented. > + * @batch: One and a half ChaCha20 blocks of buffered RNG output. > + * > + * @key: Key to be used for generating next batch. > + * > + * @batch_key: Union of the prior two members, which is exactly two full > + * ChaCha20 blocks in size, so that @batch and @key can be filled > + * together. > + * > + * @generation: Snapshot of @rng_info->generation in the vDSO data page at > + * the time @key was generated. > + * > + * @pos: Offset into @batch of the next available random byte. > + * > + * @in_use: Reentrancy guard for reusing a state within the same thread > + * due to signal handlers. > */ > -struct vgetrandom_state { int placeholder; }; > +struct vgetrandom_state { > + union { > + struct { > + u8 batch[CHACHA_BLOCK_SIZE * 3 / 2]; > + u32 key[CHACHA_KEY_SIZE / sizeof(u32)]; CHACHA_STATE_WORDS ? > + }; > + u8 batch_key[CHACHA_BLOCK_SIZE * 2]; Lot's of magic constants here *3/2 *2 .... > + }; > + vdso_kernel_ulong generation; > + u8 pos; What does the u8 buy here over a simple unsigned int? > + bool in_use; > +}; > > #endif /* _VDSO_GETRANDOM_H */ > diff --git a/include/vdso/types.h b/include/vdso/types.h > new file mode 100644 > index 000000000000..ce131463aeff > --- /dev/null > +++ b/include/vdso/types.h > @@ -0,0 +1,35 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ Why does this need an extra header when it's clearly getrandom specific? Please put this into getrandom.h > +/* > + * Copyright (C) 2022 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved. > + */ > +#ifndef __VDSO_TYPES_H > +#define __VDSO_TYPES_H > + > +#include <linux/types.h> > + > +/** > + * type vdso_kernel_ulong - unsigned long type that matches kernel's unsigned long > + * > + * Data shared between userspace and the kernel must operate the same way in both 64-bit code and in > + * 32-bit compat code, over the same potentially 64-bit kernel. This type represents the size of an > + * unsigned long as used by kernel code. This isn't necessarily the same as an unsigned long as used > + * by userspace, however. This is confusing at best. First of all 64-bit code can run only on a 64-bit kernel, so what does 'the same potentially 64-bit kernel' even mean in that sentence? What means: 'This type represents the size of an unsigned long as used by kernel code'? > + * +-------------------+-------------------+------------------+-------------------+ > + * | 32-bit userspace | 32-bit userspace | 64-bit userspace | 64-bit userspace | > + * | unsigned long | vdso_kernel_ulong | unsigned long | vdso_kernel_ulong | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + * | 32-bit kernel | ✓ same size | ✓ same size | > + * | unsigned long | | | > + * +---------------+-------------------+-------------------+------------------+-------------------+ > + * | 64-bit kernel | ✘ different size! | ✓ same size | ✓ same size | ✓ same size | > + * | unsigned long | | | | | > + * +---------------+-------------------+-------------------+------------------+-------------------+ I have no idea what this table tries to tell me, but I clearly can see what you are trying to achieve here: > + */ > +#ifdef CONFIG_64BIT > +typedef u64 vdso_kernel_ulong; > +#else > +typedef u32 vdso_kernel_ulong; > +#endif All of this is pointless because if a 32-bit application runs on a 64-bit kernel it has to use the 64-bit 'generation'. So why on earth do we need magic here for a 32-bit kernel? Just use u64 for both and spare all this voodoo. We're seriously not "optimizing" for 32-bit kernels. > +/** > + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. > + * @rng_info: Describes state of kernel RNG, memory shared with kernel. > + * @buffer: Destination buffer to fill with random bytes. > + * @len: Size of @buffer in bytes. > + * @flags: Zero or more GRND_* flags. > + * @opaque_state: Pointer to an opaque state area. > + * > + * This implements a "fast key erasure" RNG using ChaCha20, in the same way that the kernel's > + * getrandom() syscall does. It periodically reseeds its key from the kernel's RNG, at the same > + * schedule that the kernel's RNG is reseeded. If the kernel's RNG is not ready, then this always > + * calls into the syscall. > + * > + * @opaque_state *must* be allocated using the vgetrandom_alloc() syscall. Unless external locking > + * is used, one state must be allocated per thread, as it is not safe to call this function > + * concurrently with the same @opaque_state. However, it is safe to call this using the same > + * @opaque_state that is shared between main code and signal handling code, within the same thread. > + * > + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. > + */ > +static __always_inline ssize_t > +__cvdso_getrandom_data(const struct vdso_rng_data *rng_info, void *buffer, size_t len, > + unsigned int flags, void *opaque_state) > +{ > + ssize_t ret = min_t(size_t, INT_MAX & PAGE_MASK /* = MAX_RW_COUNT */, len); We really need to allow reading almost 2GB of random data in one go? > + struct vgetrandom_state *state = opaque_state; > + size_t batch_len, nblocks, orig_len = len; > + unsigned long current_generation; > + void *orig_buffer = buffer; > + u32 counter[2] = { 0 }; > + bool in_use, have_retried = false; Please keep the reverse fir tree order. > + /* The state must not straddle a page, since pages can be zeroed at any time. */ > + if (unlikely(((unsigned long)opaque_state & ~PAGE_MASK) + sizeof(*state) > PAGE_SIZE)) > + goto fallback_syscall; > + > + /* > + * If the kernel's RNG is not yet ready, then it's not possible to provide random bytes from > + * userspace, because A) the various @flags require this to block, or not, depending on > + * various factors unavailable to userspace, and B) the kernel's behavior before the RNG is > + * ready is to reseed from the entropy pool at every invocation. > + */ > + if (unlikely(!READ_ONCE(rng_info->is_ready))) > + goto fallback_syscall; > + > + /* > + * This condition is checked after @rng_info->is_ready, because before the kernel's RNG is > + * initialized, the @flags parameter may require this to block or return an error, even when > + * len is zero. > + */ > + if (unlikely(!len)) > + return 0; > + > + /* > + * @state->in_use is basic reentrancy protection against this running in a signal handler > + * with the same @opaque_state, but obviously not atomic wrt multiple CPUs or more than one > + * level of reentrancy. If a signal interrupts this after reading @state->in_use, but before > + * writing @state->in_use, there is still no race, because the signal handler will run to > + * its completion before returning execution. Can you please add an explanation that the syscall does not touch the state and just fills the buffer? > + */ > + in_use = READ_ONCE(state->in_use); > + if (unlikely(in_use)) > + goto fallback_syscall; > + WRITE_ONCE(state->in_use, true); > + > +retry_generation: > + /* > + * @rng_info->generation must always be read here, as it serializes @state->key with the > + * kernel's RNG reseeding schedule. > + */ > + current_generation = READ_ONCE(rng_info->generation); > + > + /* > + * If @state->generation doesn't match the kernel RNG's generation, then it means the > + * kernel's RNG has reseeded, and so @state->key is reseeded as well. > + */ > + if (unlikely(state->generation != current_generation)) { > + /* > + * Write the generation before filling the key, in case of fork. If there is a fork > + * just after this line, the two forks will get different random bytes from the the two forks? You mean the parent and the child, no? > + * syscall, which is good. However, were this line to occur after the getrandom > + * syscall, then both child and parent could have the same bytes and the same > + * generation counter, so the fork would not be detected. Therefore, write > + * @state->generation before the call to the getrandom syscall. > + */ > + WRITE_ONCE(state->generation, current_generation); > + > + /* Prevent the syscall from being reordered wrt current_generation. */ > + barrier(); > + > + /* Reseed @state->key using fresh bytes from the kernel. */ > + if (getrandom_syscall(state->key, sizeof(state->key), 0) != sizeof(state->key)) { > + /* > + * If the syscall failed to refresh the key, then @state->key is now > + * invalid, so invalidate the generation so that it is not used again, and > + * fallback to using the syscall entirely. > + */ > + WRITE_ONCE(state->generation, 0); > + > + /* > + * Set @state->in_use to false only after the last write to @state in the > + * line above. > + */ > + WRITE_ONCE(state->in_use, false); So here you rely on the compiler not reordering vs. WRITE_ONCE(), i.e. volatile, but above you have a barrier() to prevent the write being reordered vs. the syscall, confused. But even when the compiler does not reorder, what prevents a weakly ordered CPU from doing so? > + goto fallback_syscall; > + } > + > + /* > + * Set @state->pos to beyond the end of the batch, so that the batch is refilled > + * using the new key. > + */ > + state->pos = sizeof(state->batch); > + } > + > + /* Set len to the total amount of bytes that this function is allowed to read, ret. */ > + len = ret; > +more_batch: > + /* > + * First use bytes out of @state->batch, which may have been filled by the last call to this > + * function. > + */ > + batch_len = min_t(size_t, sizeof(state->batch) - state->pos, len); > + if (batch_len) { > + /* Zeroing at the same time as memcpying helps preserve forward secrecy. */ > + memcpy_and_zero_src(buffer, state->batch + state->pos, batch_len); > + state->pos += batch_len; > + buffer += batch_len; > + len -= batch_len; > + } > + > + if (!len) { > + /* Prevent the loop from being reordered wrt ->generation. */ > + barrier(); Same question as above. > + /* > + * Since @rng_info->generation will never be 0, re-read @state->generation, rather > + * than using the local current_generation variable, to learn whether a fork > + * occurred or if @state was zeroed due to memory pressure. Primarily, though, this > + * indicates whether the kernel's RNG has reseeded, in which case generate a new key > + * and start over. > + */ > + if (unlikely(READ_ONCE(state->generation) != READ_ONCE(rng_info->generation))) { > + /* > + * Prevent this from looping forever in case of low memory or racing with a > + * user force-reseeding the kernel's RNG using the ioctl. > + */ > + if (have_retried) { > + WRITE_ONCE(state->in_use, false); > + goto fallback_syscall; > + } > + > + have_retried = true; > + buffer = orig_buffer; > + goto retry_generation; > + } > + > + /* > + * Set @state->in_use to false only when there will be no more reads or writes of > + * @state. > + */ > + WRITE_ONCE(state->in_use, false); > + return ret; > + } > + > + /* Generate blocks of RNG output directly into @buffer while there's enough room left. */ > + nblocks = len / CHACHA_BLOCK_SIZE; > + if (nblocks) { > + __arch_chacha20_blocks_nostack(buffer, state->key, counter, nblocks); > + buffer += nblocks * CHACHA_BLOCK_SIZE; > + len -= nblocks * CHACHA_BLOCK_SIZE; > + } > + > + BUILD_BUG_ON(sizeof(state->batch_key) % CHACHA_BLOCK_SIZE != 0); > + > + /* Refill the batch and then overwrite the key, in order to preserve forward secrecy. */ 'and then overwrite'? Isn't this overwriting it implicitely because batch_key and key are at the same place in the union? > + __arch_chacha20_blocks_nostack(state->batch_key, state->key, counter, > + sizeof(state->batch_key) / CHACHA_BLOCK_SIZE); > + > + /* Since the batch was just refilled, set the position back to 0 to indicate a full batch. */ > + state->pos = 0; > + goto more_batch; Thanks, tglx
Jason! On Wed, Jun 05 2024 at 23:03, Thomas Gleixner wrote: > On Tue, May 28 2024 at 14:19, Jason A. Donenfeld wrote: >> + */ >> +#ifdef CONFIG_64BIT >> +typedef u64 vdso_kernel_ulong; >> +#else >> +typedef u32 vdso_kernel_ulong; >> +#endif > > All of this is pointless because if a 32-bit application runs on a > 64-bit kernel it has to use the 64-bit 'generation'. So why on earth do > we need magic here for a 32-bit kernel? > > Just use u64 for both and spare all this voodoo. We're seriously not > "optimizing" for 32-bit kernels. All what happens on a 32-bit kernel is that the RNG will store the unsigned long (32bit) generation into a 64bit variable: smp_store_release(&_vdso_rng_data.generation, next_gen + 1); As the upper 32bit are always zero, there is no issue vs. load store tearing at all. So there is zero benefit for this aside of slightly "better" user space code when running on a 32-bit kernel. Who cares? While staring at this I wonder where the corresponding smp_load_acquire() is. I haven't found one in the VDSO code. READ_ONCE() is only equivalent on a few architectures. But, what does that store_release() buy at all? There is zero ordering vs. anything in the kernel and neither against user space. If that smp_store_release() serves a purpose then it really has to be extensively documented especially as the kernel itself simply uses WRITE/READ_ONCE() for base_rng.generation. Thanks, tglx
On Fri, May 31, 2024 at 12:15:16PM -0700, Randy Dunlap wrote: > > > On 5/28/24 5:19 AM, Jason A. Donenfeld wrote: > > +/** > > + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. > > + * @rng_info: Describes state of kernel RNG, memory shared with kernel. > > + * @buffer: Destination buffer to fill with random bytes. > > + * @len: Size of @buffer in bytes. > > + * @flags: Zero or more GRND_* flags. > > + * @opaque_state: Pointer to an opaque state area. > > + * > > + * This implements a "fast key erasure" RNG using ChaCha20, in the same way that the kernel's > > + * getrandom() syscall does. It periodically reseeds its key from the kernel's RNG, at the same > > + * schedule that the kernel's RNG is reseeded. If the kernel's RNG is not ready, then this always > > + * calls into the syscall. > > + * > > + * @opaque_state *must* be allocated using the vgetrandom_alloc() syscall. Unless external locking > > + * is used, one state must be allocated per thread, as it is not safe to call this function > > + * concurrently with the same @opaque_state. However, it is safe to call this using the same > > + * @opaque_state that is shared between main code and signal handling code, within the same thread. > > + * > > + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. > > * Returns: Ack. Thanks.
On Fri, May 31, 2024 at 04:06:37PM -0700, Andy Lutomirski wrote: > > On May 28, 2024, at 5:25 AM, Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > > > Provide a generic C vDSO getrandom() implementation, which operates on > > an opaque state returned by vgetrandom_alloc() and produces random bytes > > the same way as getrandom(). This has a the API signature: > > > > ssize_t vgetrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state); > > > +/** > > + * type vdso_kernel_ulong - unsigned long type that matches kernel's unsigned long > > + * > > + * Data shared between userspace and the kernel must operate the same way in both 64-bit code and in > > + * 32-bit compat code, over the same potentially 64-bit kernel. This type represents the size of an > > + * unsigned long as used by kernel code. This isn't necessarily the same as an unsigned long as used > > + * by userspace, however. > > Why is this better than using plain u64? It’s certainly more > complicated. It also rather fundamentally breaks CRIU on 32-bit > userspace (although CRIU may well be unable to keep vgetrandom working > after a restore onto a different kernel anyway). Admittedly 32-bit > userspace is a slowly dying breed, but still. That came out of this conversation: https://lore.kernel.org/all/878rjs7mcx.fsf@oldenburg.str.redhat.com/ (And I'd like single instruction increments, which means long, not u64 on 32-bit machines.) > > +{ > > + ssize_t ret = min_t(size_t, INT_MAX & PAGE_MASK /* = MAX_RW_COUNT */, len); > > + struct vgetrandom_state *state = opaque_state; > > + size_t batch_len, nblocks, orig_len = len; > > + unsigned long current_generation; > > + void *orig_buffer = buffer; > > + u32 counter[2] = { 0 }; > > + bool in_use, have_retried = false; > > + > > + /* The state must not straddle a page, since pages can be zeroed at any time. */ > > + if (unlikely(((unsigned long)opaque_state & ~PAGE_MASK) + sizeof(*state) > PAGE_SIZE)) > > + goto fallback_syscall; > > This is weird. Either the provided pointer is valid or it isn’t. > Reasonable outcomes are a segfault if the pointer is bad or success > (or fallback if needed for some reason) if the pointer is good. Why > is there specific code to catch a specific sort of pointer screwup > here? I guess I could make it return -EFAULT in this case, rather than silently succeeding. Jason
On Thu, Jun 06, 2024 at 12:10:00AM +0200, Thomas Gleixner wrote: > Jason! > > On Wed, Jun 05 2024 at 23:03, Thomas Gleixner wrote: > > On Tue, May 28 2024 at 14:19, Jason A. Donenfeld wrote: > >> + */ > >> +#ifdef CONFIG_64BIT > >> +typedef u64 vdso_kernel_ulong; > >> +#else > >> +typedef u32 vdso_kernel_ulong; > >> +#endif > > > > All of this is pointless because if a 32-bit application runs on a > > 64-bit kernel it has to use the 64-bit 'generation'. So why on earth do > > we need magic here for a 32-bit kernel? > > > > Just use u64 for both and spare all this voodoo. We're seriously not > > "optimizing" for 32-bit kernels. > > All what happens on a 32-bit kernel is that the RNG will store the > unsigned long (32bit) generation into a 64bit variable: > > smp_store_release(&_vdso_rng_data.generation, next_gen + 1); > > As the upper 32bit are always zero, there is no issue vs. load store > tearing at all. So there is zero benefit for this aside of slightly > "better" user space code when running on a 32-bit kernel. Who cares? Oh yea. Okay, great. I was concerned about the tearing, but I guess it's really a non issue. So I'll just make it a u64 and all of this complexity can just go away. Thanks for thinking about it in a less convoluted way than me. > While staring at this I wonder where the corresponding > smp_load_acquire() is. I haven't found one in the VDSO code. > READ_ONCE() is only equivalent on a few architectures. > > But, what does that store_release() buy at all? There is zero ordering > vs. anything in the kernel and neither against user space. > > If that smp_store_release() serves a purpose then it really has to be > extensively documented especially as the kernel itself simply uses > WRITE/READ_ONCE() for base_rng.generation. This came up here too: https://lore.kernel.org/all/Y3l6ocn1dTN0+1GK@zx2c4.com/ It's to order the writes to the generation counter and is_ready. Jason
On Wed, Jun 05, 2024 at 11:03:00PM +0200, Thomas Gleixner wrote: > Jason! Thomas! > Can you please split the required defines into a seperate header > preferrably in include/vdso/ and include that from crypto/chacha.h Sure. It only actually uses two straight forward constants from there. > > + u32 key[CHACHA_KEY_SIZE / sizeof(u32)]; > > CHACHA_STATE_WORDS ? Nah, that's for CHACHA_BLOCK_SIZE / sizeof(u32), but here is CHACHA_KEY_SIZE. > > > + }; > > + u8 batch_key[CHACHA_BLOCK_SIZE * 2]; > > What does the u8 buy here over a simple unsigned int? > > > + bool in_use; It means that the structure can be more compact, because `pos` and the `in_use` boolean will be closer together. > > diff --git a/include/vdso/types.h b/include/vdso/types.h > > new file mode 100644 > > index 000000000000..ce131463aeff > > --- /dev/null > > +++ b/include/vdso/types.h > > @@ -0,0 +1,35 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > Why does this need an extra header when it's clearly getrandom specific? > Please put this into getrandom.h From your followup, I just killed the whole thing and now use u64. > > +/** > > + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. > > + * @rng_info: Describes state of kernel RNG, memory shared with kernel. > > + * @buffer: Destination buffer to fill with random bytes. > > + * @len: Size of @buffer in bytes. > > + * @flags: Zero or more GRND_* flags. > > + * @opaque_state: Pointer to an opaque state area. > > + * > > + * This implements a "fast key erasure" RNG using ChaCha20, in the same way that the kernel's > > + * getrandom() syscall does. It periodically reseeds its key from the kernel's RNG, at the same > > + * schedule that the kernel's RNG is reseeded. If the kernel's RNG is not ready, then this always > > + * calls into the syscall. > > + * > > + * @opaque_state *must* be allocated using the vgetrandom_alloc() syscall. Unless external locking > > + * is used, one state must be allocated per thread, as it is not safe to call this function > > + * concurrently with the same @opaque_state. However, it is safe to call this using the same > > + * @opaque_state that is shared between main code and signal handling code, within the same thread. > > + * > > + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. > > + */ > > +static __always_inline ssize_t > > +__cvdso_getrandom_data(const struct vdso_rng_data *rng_info, void *buffer, size_t len, > > + unsigned int flags, void *opaque_state) > > +{ > > + ssize_t ret = min_t(size_t, INT_MAX & PAGE_MASK /* = MAX_RW_COUNT */, len); > > We really need to allow reading almost 2GB of random data in one go? It's just copying the precise semantics as the syscall by bounding the requested length. The idea is to make this have basically identical semantics as the syscall (while being way faster). > > + /* > > + * @state->in_use is basic reentrancy protection against this running in a signal handler > > + * with the same @opaque_state, but obviously not atomic wrt multiple CPUs or more than one > > + * level of reentrancy. If a signal interrupts this after reading @state->in_use, but before > > + * writing @state->in_use, there is still no race, because the signal handler will run to > > + * its completion before returning execution. > > Can you please add an explanation that the syscall does not touch the > state and just fills the buffer? Will do. > > + */ > > + in_use = READ_ONCE(state->in_use); > > + if (unlikely(in_use)) > > + goto fallback_syscall; > > + WRITE_ONCE(state->in_use, true); > > + > > +retry_generation: > > + /* > > + * @rng_info->generation must always be read here, as it serializes @state->key with the > > + * kernel's RNG reseeding schedule. > > + */ > > + current_generation = READ_ONCE(rng_info->generation); > > + > > + /* > > + * If @state->generation doesn't match the kernel RNG's generation, then it means the > > + * kernel's RNG has reseeded, and so @state->key is reseeded as well. > > + */ > > + if (unlikely(state->generation != current_generation)) { > > + /* > > + * Write the generation before filling the key, in case of fork. If there is a fork > > + * just after this line, the two forks will get different random bytes from the > > the two forks? You mean the parent and the child, no? Yes, nice catch, thanks. > > + * syscall, which is good. However, were this line to occur after the getrandom > > + * syscall, then both child and parent could have the same bytes and the same > > + * generation counter, so the fork would not be detected. Therefore, write > > + * @state->generation before the call to the getrandom syscall. > > + */ > > + WRITE_ONCE(state->generation, current_generation); > > + > > + /* Prevent the syscall from being reordered wrt current_generation. */ > > + barrier(); > > + > > + /* Reseed @state->key using fresh bytes from the kernel. */ > > + if (getrandom_syscall(state->key, sizeof(state->key), 0) != sizeof(state->key)) { > > + /* > > + * If the syscall failed to refresh the key, then @state->key is now > > + * invalid, so invalidate the generation so that it is not used again, and > > + * fallback to using the syscall entirely. > > + */ > > + WRITE_ONCE(state->generation, 0); > > + > > + /* > > + * Set @state->in_use to false only after the last write to @state in the > > + * line above. > > + */ > > + WRITE_ONCE(state->in_use, false); > > So here you rely on the compiler not reordering vs. WRITE_ONCE(), > i.e. volatile, but above you have a barrier() to prevent the write being > reordered vs. the syscall, confused. > > But even when the compiler does not reorder, what prevents a weakly > ordered CPU from doing so? The issue isn't that this code will race between CPUs -- that's explicitly disallowed by the design. The issue is that this code is signal-reentrant. So it's mostly a matter of compiler ordering the instructions correctly. Then, in addition, the write of current_generation to state->generation must come before the syscall fires. > > + if (!len) { > > + /* Prevent the loop from being reordered wrt ->generation. */ > > + barrier(); > > Same question as above. This came out of discussions here: https://lore.kernel.org/all/878rjlr85s.fsf@oldenburg.str.redhat.com/ And on IRC with Jann. > > + /* Refill the batch and then overwrite the key, in order to preserve forward secrecy. */ > > 'and then overwrite'? > > Isn't this overwriting it implicitely because batch_key and key are at > the same place in the union? Yes, I'll remove the `then`. Thanks for the review! Jason
More comments... On Tue, May 28, 2024 at 5:25 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote: > > Provide a generic C vDSO getrandom() implementation, which operates on > an opaque state returned by vgetrandom_alloc() and produces random bytes > the same way as getrandom(). This has a the API signature: > > ssize_t vgetrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state); > > The return value and the first 3 arguments are the same as ordinary > getrandom(), while the last argument is a pointer to the opaque > allocated state. Were all four arguments passed to the getrandom() > syscall, nothing different would happen, and the functions would have > the exact same behavior. > > The actual vDSO RNG algorithm implemented is the same one implemented by > drivers/char/random.c, using the same fast-erasure techniques as that. > Should the in-kernel implementation change, so too will the vDSO one. > > It requires an implementation of ChaCha20 that does not use any stack, > in order to maintain forward secrecy if a multi-threaded program forks > (though this does not account for a similar issue with SA_SIGINFO > copying registers to the stack), so this is left as an > architecture-specific fill-in. Stack-less ChaCha20 is an easy algorithm > to implement on a variety of architectures, so this shouldn't be too > onerous. Can you clarify this, because I'm a bit confused. First, if a multi-threaded program forks, bascially all bets are off -- fork() is extremely poorly behaved in multithreaded programs, and the child should take care to execve() or exit() in short order. But more to the point: If I do: some_bytes = get_awesome_random_bytes(); <-- other thread forks here! The bytes get copied. Is the concern that the fork might happen *in the middle* of the vDSO code, causing the child to end up inadvertently possessing a copy of the parent's random state and thus being able to predict future outputs? If so, I think this could be much more cleanly fixed by making sure that the vDSO state gets wiped *for the parent and the child* on a fork. > + /* > + * If @state->generation doesn't match the kernel RNG's generation, then it means the > + * kernel's RNG has reseeded, and so @state->key is reseeded as well. > + */ > + if (unlikely(state->generation != current_generation)) { > + /* > + * Write the generation before filling the key, in case of fork. If there is a fork > + * just after this line, the two forks will get different random bytes from the > + * syscall, which is good. However, were this line to occur after the getrandom > + * syscall, then both child and parent could have the same bytes and the same > + * generation counter, so the fork would not be detected. Therefore, write > + * @state->generation before the call to the getrandom syscall. > + */ > + WRITE_ONCE(state->generation, current_generation); Farther down the thread I think you were saying this had something to do with signals, not fork. As for fork, if you make sure that rng_info->generation can never be 0, then, after a fork, the vDSO will always retry or fall back, and I think there will be no complexity in the middle related to forking, which could end up simplifying a few things.
diff --git a/MAINTAINERS b/MAINTAINERS index 522c88b38550..55ffa68426bc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -18698,6 +18698,7 @@ F: Documentation/devicetree/bindings/rng/microsoft,vmgenid.yaml F: drivers/char/random.c F: drivers/virt/vmgenid.c F: include/vdso/getrandom.h +F: lib/vdso/getrandom.c RAPIDIO SUBSYSTEM M: Matt Porter <mporter@kernel.crashing.org> diff --git a/drivers/char/random.c b/drivers/char/random.c index b066b8e0bbcb..3e66827f06ea 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -60,6 +60,7 @@ #include <crypto/blake2s.h> #ifdef CONFIG_VDSO_GETRANDOM #include <vdso/getrandom.h> +#include <vdso/datapage.h> #endif #include <asm/archrandom.h> #include <asm/processor.h> @@ -403,6 +404,13 @@ static void crng_reseed(struct work_struct *work) if (next_gen == ULONG_MAX) ++next_gen; WRITE_ONCE(base_crng.generation, next_gen); +#ifdef CONFIG_VDSO_GETRANDOM + /* base_crng.generation's invalid value is ULONG_MAX, while + * _vdso_rng_data.generation's invalid value is 0, so add one to the + * former to arrive at the latter. + */ + smp_store_release(&_vdso_rng_data.generation, next_gen + 1); +#endif if (!static_branch_likely(&crng_is_ready)) crng_init = CRNG_READY; spin_unlock_irqrestore(&base_crng.lock, flags); @@ -853,6 +861,9 @@ static void __cold _credit_init_bits(size_t bits) if (static_key_initialized && system_unbound_wq) queue_work(system_unbound_wq, &set_ready); atomic_notifier_call_chain(&random_ready_notifier, 0, NULL); +#ifdef CONFIG_VDSO_GETRANDOM + smp_store_release(&_vdso_rng_data.is_ready, true); +#endif wake_up_interruptible(&crng_init_wait); kill_fasync(&fasync, SIGIO, POLL_IN); pr_notice("crng init done\n"); diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index d04d394db064..3ca19296ce28 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -18,6 +18,7 @@ #include <vdso/time.h> #include <vdso/time32.h> #include <vdso/time64.h> +#include <vdso/types.h> #ifdef CONFIG_ARCH_HAS_VDSO_DATA #include <asm/vdso/data.h> @@ -113,6 +114,16 @@ struct vdso_data { struct arch_vdso_data arch_data; }; +/** + * struct vdso_rng_data - vdso RNG state information + * @generation: counter representing the number of RNG reseeds + * @is_ready: boolean signaling whether the RNG is initialized + */ +struct vdso_rng_data { + vdso_kernel_ulong generation; + u8 is_ready; +}; + /* * We use the hidden visibility to prevent the compiler from generating a GOT * relocation. Not only is going through a GOT useless (the entry couldn't and @@ -124,6 +135,7 @@ struct vdso_data { */ extern struct vdso_data _vdso_data[CS_BASES] __attribute__((visibility("hidden"))); extern struct vdso_data _timens_data[CS_BASES] __attribute__((visibility("hidden"))); +extern struct vdso_rng_data _vdso_rng_data __attribute__((visibility("hidden"))); /** * union vdso_data_store - Generic vDSO data page diff --git a/include/vdso/getrandom.h b/include/vdso/getrandom.h index e3ceb1976386..7dc93d5f72dc 100644 --- a/include/vdso/getrandom.h +++ b/include/vdso/getrandom.h @@ -6,11 +6,39 @@ #ifndef _VDSO_GETRANDOM_H #define _VDSO_GETRANDOM_H +#include <crypto/chacha.h> +#include <vdso/types.h> + /** * struct vgetrandom_state - State used by vDSO getrandom() and allocated by vgetrandom_alloc(). * - * Currently empty, as the vDSO getrandom() function has not yet been implemented. + * @batch: One and a half ChaCha20 blocks of buffered RNG output. + * + * @key: Key to be used for generating next batch. + * + * @batch_key: Union of the prior two members, which is exactly two full + * ChaCha20 blocks in size, so that @batch and @key can be filled + * together. + * + * @generation: Snapshot of @rng_info->generation in the vDSO data page at + * the time @key was generated. + * + * @pos: Offset into @batch of the next available random byte. + * + * @in_use: Reentrancy guard for reusing a state within the same thread + * due to signal handlers. */ -struct vgetrandom_state { int placeholder; }; +struct vgetrandom_state { + union { + struct { + u8 batch[CHACHA_BLOCK_SIZE * 3 / 2]; + u32 key[CHACHA_KEY_SIZE / sizeof(u32)]; + }; + u8 batch_key[CHACHA_BLOCK_SIZE * 2]; + }; + vdso_kernel_ulong generation; + u8 pos; + bool in_use; +}; #endif /* _VDSO_GETRANDOM_H */ diff --git a/include/vdso/types.h b/include/vdso/types.h new file mode 100644 index 000000000000..ce131463aeff --- /dev/null +++ b/include/vdso/types.h @@ -0,0 +1,35 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved. + */ +#ifndef __VDSO_TYPES_H +#define __VDSO_TYPES_H + +#include <linux/types.h> + +/** + * type vdso_kernel_ulong - unsigned long type that matches kernel's unsigned long + * + * Data shared between userspace and the kernel must operate the same way in both 64-bit code and in + * 32-bit compat code, over the same potentially 64-bit kernel. This type represents the size of an + * unsigned long as used by kernel code. This isn't necessarily the same as an unsigned long as used + * by userspace, however. + * + * +-------------------+-------------------+------------------+-------------------+ + * | 32-bit userspace | 32-bit userspace | 64-bit userspace | 64-bit userspace | + * | unsigned long | vdso_kernel_ulong | unsigned long | vdso_kernel_ulong | + * +---------------+-------------------+-------------------+------------------+-------------------+ + * | 32-bit kernel | ✓ same size | ✓ same size | + * | unsigned long | | | + * +---------------+-------------------+-------------------+------------------+-------------------+ + * | 64-bit kernel | ✘ different size! | ✓ same size | ✓ same size | ✓ same size | + * | unsigned long | | | | | + * +---------------+-------------------+-------------------+------------------+-------------------+ + */ +#ifdef CONFIG_64BIT +typedef u64 vdso_kernel_ulong; +#else +typedef u32 vdso_kernel_ulong; +#endif + +#endif /* __VDSO_TYPES_H */ diff --git a/lib/vdso/getrandom.c b/lib/vdso/getrandom.c new file mode 100644 index 000000000000..4d9bb59985f8 --- /dev/null +++ b/lib/vdso/getrandom.c @@ -0,0 +1,226 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2022 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved. + */ + +#include <linux/cache.h> +#include <linux/kernel.h> +#include <linux/time64.h> +#include <vdso/datapage.h> +#include <vdso/getrandom.h> +#include <asm/vdso/getrandom.h> +#include <asm/vdso/vsyscall.h> + +#define MEMCPY_AND_ZERO_SRC(type, dst, src, len) do { \ + while (len >= sizeof(type)) { \ + __put_unaligned_t(type, __get_unaligned_t(type, src), dst); \ + __put_unaligned_t(type, 0, src); \ + dst += sizeof(type); \ + src += sizeof(type); \ + len -= sizeof(type); \ + } \ +} while (0) + +static void memcpy_and_zero_src(void *dst, void *src, size_t len) +{ + if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)) { + if (IS_ENABLED(CONFIG_64BIT)) + MEMCPY_AND_ZERO_SRC(u64, dst, src, len); + MEMCPY_AND_ZERO_SRC(u32, dst, src, len); + MEMCPY_AND_ZERO_SRC(u16, dst, src, len); + } + MEMCPY_AND_ZERO_SRC(u8, dst, src, len); +} + +/** + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. + * @rng_info: Describes state of kernel RNG, memory shared with kernel. + * @buffer: Destination buffer to fill with random bytes. + * @len: Size of @buffer in bytes. + * @flags: Zero or more GRND_* flags. + * @opaque_state: Pointer to an opaque state area. + * + * This implements a "fast key erasure" RNG using ChaCha20, in the same way that the kernel's + * getrandom() syscall does. It periodically reseeds its key from the kernel's RNG, at the same + * schedule that the kernel's RNG is reseeded. If the kernel's RNG is not ready, then this always + * calls into the syscall. + * + * @opaque_state *must* be allocated using the vgetrandom_alloc() syscall. Unless external locking + * is used, one state must be allocated per thread, as it is not safe to call this function + * concurrently with the same @opaque_state. However, it is safe to call this using the same + * @opaque_state that is shared between main code and signal handling code, within the same thread. + * + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. + */ +static __always_inline ssize_t +__cvdso_getrandom_data(const struct vdso_rng_data *rng_info, void *buffer, size_t len, + unsigned int flags, void *opaque_state) +{ + ssize_t ret = min_t(size_t, INT_MAX & PAGE_MASK /* = MAX_RW_COUNT */, len); + struct vgetrandom_state *state = opaque_state; + size_t batch_len, nblocks, orig_len = len; + unsigned long current_generation; + void *orig_buffer = buffer; + u32 counter[2] = { 0 }; + bool in_use, have_retried = false; + + /* The state must not straddle a page, since pages can be zeroed at any time. */ + if (unlikely(((unsigned long)opaque_state & ~PAGE_MASK) + sizeof(*state) > PAGE_SIZE)) + goto fallback_syscall; + + /* + * If the kernel's RNG is not yet ready, then it's not possible to provide random bytes from + * userspace, because A) the various @flags require this to block, or not, depending on + * various factors unavailable to userspace, and B) the kernel's behavior before the RNG is + * ready is to reseed from the entropy pool at every invocation. + */ + if (unlikely(!READ_ONCE(rng_info->is_ready))) + goto fallback_syscall; + + /* + * This condition is checked after @rng_info->is_ready, because before the kernel's RNG is + * initialized, the @flags parameter may require this to block or return an error, even when + * len is zero. + */ + if (unlikely(!len)) + return 0; + + /* + * @state->in_use is basic reentrancy protection against this running in a signal handler + * with the same @opaque_state, but obviously not atomic wrt multiple CPUs or more than one + * level of reentrancy. If a signal interrupts this after reading @state->in_use, but before + * writing @state->in_use, there is still no race, because the signal handler will run to + * its completion before returning execution. + */ + in_use = READ_ONCE(state->in_use); + if (unlikely(in_use)) + goto fallback_syscall; + WRITE_ONCE(state->in_use, true); + +retry_generation: + /* + * @rng_info->generation must always be read here, as it serializes @state->key with the + * kernel's RNG reseeding schedule. + */ + current_generation = READ_ONCE(rng_info->generation); + + /* + * If @state->generation doesn't match the kernel RNG's generation, then it means the + * kernel's RNG has reseeded, and so @state->key is reseeded as well. + */ + if (unlikely(state->generation != current_generation)) { + /* + * Write the generation before filling the key, in case of fork. If there is a fork + * just after this line, the two forks will get different random bytes from the + * syscall, which is good. However, were this line to occur after the getrandom + * syscall, then both child and parent could have the same bytes and the same + * generation counter, so the fork would not be detected. Therefore, write + * @state->generation before the call to the getrandom syscall. + */ + WRITE_ONCE(state->generation, current_generation); + + /* Prevent the syscall from being reordered wrt current_generation. */ + barrier(); + + /* Reseed @state->key using fresh bytes from the kernel. */ + if (getrandom_syscall(state->key, sizeof(state->key), 0) != sizeof(state->key)) { + /* + * If the syscall failed to refresh the key, then @state->key is now + * invalid, so invalidate the generation so that it is not used again, and + * fallback to using the syscall entirely. + */ + WRITE_ONCE(state->generation, 0); + + /* + * Set @state->in_use to false only after the last write to @state in the + * line above. + */ + WRITE_ONCE(state->in_use, false); + + goto fallback_syscall; + } + + /* + * Set @state->pos to beyond the end of the batch, so that the batch is refilled + * using the new key. + */ + state->pos = sizeof(state->batch); + } + + /* Set len to the total amount of bytes that this function is allowed to read, ret. */ + len = ret; +more_batch: + /* + * First use bytes out of @state->batch, which may have been filled by the last call to this + * function. + */ + batch_len = min_t(size_t, sizeof(state->batch) - state->pos, len); + if (batch_len) { + /* Zeroing at the same time as memcpying helps preserve forward secrecy. */ + memcpy_and_zero_src(buffer, state->batch + state->pos, batch_len); + state->pos += batch_len; + buffer += batch_len; + len -= batch_len; + } + + if (!len) { + /* Prevent the loop from being reordered wrt ->generation. */ + barrier(); + + /* + * Since @rng_info->generation will never be 0, re-read @state->generation, rather + * than using the local current_generation variable, to learn whether a fork + * occurred or if @state was zeroed due to memory pressure. Primarily, though, this + * indicates whether the kernel's RNG has reseeded, in which case generate a new key + * and start over. + */ + if (unlikely(READ_ONCE(state->generation) != READ_ONCE(rng_info->generation))) { + /* + * Prevent this from looping forever in case of low memory or racing with a + * user force-reseeding the kernel's RNG using the ioctl. + */ + if (have_retried) { + WRITE_ONCE(state->in_use, false); + goto fallback_syscall; + } + + have_retried = true; + buffer = orig_buffer; + goto retry_generation; + } + + /* + * Set @state->in_use to false only when there will be no more reads or writes of + * @state. + */ + WRITE_ONCE(state->in_use, false); + return ret; + } + + /* Generate blocks of RNG output directly into @buffer while there's enough room left. */ + nblocks = len / CHACHA_BLOCK_SIZE; + if (nblocks) { + __arch_chacha20_blocks_nostack(buffer, state->key, counter, nblocks); + buffer += nblocks * CHACHA_BLOCK_SIZE; + len -= nblocks * CHACHA_BLOCK_SIZE; + } + + BUILD_BUG_ON(sizeof(state->batch_key) % CHACHA_BLOCK_SIZE != 0); + + /* Refill the batch and then overwrite the key, in order to preserve forward secrecy. */ + __arch_chacha20_blocks_nostack(state->batch_key, state->key, counter, + sizeof(state->batch_key) / CHACHA_BLOCK_SIZE); + + /* Since the batch was just refilled, set the position back to 0 to indicate a full batch. */ + state->pos = 0; + goto more_batch; + +fallback_syscall: + return getrandom_syscall(orig_buffer, orig_len, flags); +} + +static __always_inline ssize_t +__cvdso_getrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state) +{ + return __cvdso_getrandom_data(__arch_get_vdso_rng_data(), buffer, len, flags, opaque_state); +} diff --git a/tools/testing/selftests/vDSO/.gitignore b/tools/testing/selftests/vDSO/.gitignore index a8dc51af5a9c..7dbfdec53f3d 100644 --- a/tools/testing/selftests/vDSO/.gitignore +++ b/tools/testing/selftests/vDSO/.gitignore @@ -6,3 +6,4 @@ vdso_test_correctness vdso_test_gettimeofday vdso_test_getcpu vdso_standalone_test_x86 +vdso_test_getrandom diff --git a/tools/testing/selftests/vDSO/Makefile b/tools/testing/selftests/vDSO/Makefile index d53a4d8008f9..a33b4d200a32 100644 --- a/tools/testing/selftests/vDSO/Makefile +++ b/tools/testing/selftests/vDSO/Makefile @@ -11,6 +11,7 @@ ifeq ($(ARCH),$(filter $(ARCH),x86 x86_64)) TEST_GEN_PROGS += $(OUTPUT)/vdso_standalone_test_x86 endif TEST_GEN_PROGS += $(OUTPUT)/vdso_test_correctness +TEST_GEN_PROGS += $(OUTPUT)/vdso_test_getrandom CFLAGS := -std=gnu99 CFLAGS_vdso_standalone_test_x86 := -nostdlib -fno-asynchronous-unwind-tables -fno-stack-protector @@ -33,3 +34,4 @@ $(OUTPUT)/vdso_test_correctness: vdso_test_correctness.c vdso_test_correctness.c \ -o $@ \ $(LDFLAGS_vdso_test_correctness) +$(OUTPUT)/vdso_test_getrandom: parse_vdso.c diff --git a/tools/testing/selftests/vDSO/vdso_test_getrandom.c b/tools/testing/selftests/vDSO/vdso_test_getrandom.c new file mode 100644 index 000000000000..80b9849fe911 --- /dev/null +++ b/tools/testing/selftests/vDSO/vdso_test_getrandom.c @@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2022 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved. + */ + +#include <assert.h> +#include <pthread.h> +#include <stdint.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <time.h> +#include <unistd.h> +#include <sys/auxv.h> +#include <sys/mman.h> +#include <sys/random.h> +#include <sys/syscall.h> +#include <sys/types.h> + +#include "../kselftest.h" +#include "parse_vdso.h" + +#ifndef timespecsub +#define timespecsub(tsp, usp, vsp) \ + do { \ + (vsp)->tv_sec = (tsp)->tv_sec - (usp)->tv_sec; \ + (vsp)->tv_nsec = (tsp)->tv_nsec - (usp)->tv_nsec; \ + if ((vsp)->tv_nsec < 0) { \ + (vsp)->tv_sec--; \ + (vsp)->tv_nsec += 1000000000L; \ + } \ + } while (0) +#endif + +static void *vgetrandom_alloc(unsigned int *num, unsigned int *size_per_each) +{ + enum { __NR_vgetrandom_alloc = 462 }; + *size_per_each = 0; + return (void *)syscall(__NR_vgetrandom_alloc, num, size_per_each, 0, 0); +} + +static struct { + pthread_mutex_t lock; + void **states; + size_t len, cap; +} grnd_allocator = { + .lock = PTHREAD_MUTEX_INITIALIZER +}; + +static void *vgetrandom_get_state(void) +{ + void *state = NULL; + + pthread_mutex_lock(&grnd_allocator.lock); + if (!grnd_allocator.len) { + size_t new_cap; + size_t page_size = getpagesize(); + unsigned int num = sysconf(_SC_NPROCESSORS_ONLN); /* Could be arbitrary, just a hint. */ + unsigned int size_per_each; + void *new_block = vgetrandom_alloc(&num, &size_per_each); + void *new_states; + + if (new_block == MAP_FAILED) + goto out; + new_cap = grnd_allocator.cap + num; + new_states = reallocarray(grnd_allocator.states, new_cap, sizeof(*grnd_allocator.states)); + if (!new_states) { + munmap(new_block, num * size_per_each); + goto out; + } + grnd_allocator.cap = new_cap; + grnd_allocator.states = new_states; + + for (size_t i = 0; i < num; ++i) { + grnd_allocator.states[i] = new_block; + if (((uintptr_t)new_block & (page_size - 1)) + size_per_each > page_size) + new_block = (void *)(((uintptr_t)new_block + page_size) & (page_size - 1)); + else + new_block += size_per_each; + } + grnd_allocator.len = num; + } + state = grnd_allocator.states[--grnd_allocator.len]; + +out: + pthread_mutex_unlock(&grnd_allocator.lock); + return state; +} + +static void vgetrandom_put_state(void *state) +{ + if (!state) + return; + pthread_mutex_lock(&grnd_allocator.lock); + grnd_allocator.states[grnd_allocator.len++] = state; + pthread_mutex_unlock(&grnd_allocator.lock); +} + +static struct { + ssize_t(*fn)(void *buf, size_t len, unsigned long flags, void *state); + pthread_key_t key; + pthread_once_t initialized; +} grnd_ctx = { + .initialized = PTHREAD_ONCE_INIT +}; + +static void vgetrandom_init(void) +{ + if (pthread_key_create(&grnd_ctx.key, vgetrandom_put_state) != 0) + return; + unsigned long sysinfo_ehdr = getauxval(AT_SYSINFO_EHDR); + if (!sysinfo_ehdr) { + printf("AT_SYSINFO_EHDR is not present!\n"); + exit(KSFT_SKIP); + } + vdso_init_from_sysinfo_ehdr(sysinfo_ehdr); + grnd_ctx.fn = (__typeof__(grnd_ctx.fn))vdso_sym("LINUX_2.6", "__vdso_getrandom"); + if (!grnd_ctx.fn) { + printf("__vdso_getrandom is missing!\n"); + exit(KSFT_FAIL); + } +} + +static ssize_t vgetrandom(void *buf, size_t len, unsigned long flags) +{ + void *state; + + pthread_once(&grnd_ctx.initialized, vgetrandom_init); + state = pthread_getspecific(grnd_ctx.key); + if (!state) { + state = vgetrandom_get_state(); + if (pthread_setspecific(grnd_ctx.key, state) != 0) { + vgetrandom_put_state(state); + state = NULL; + } + if (!state) { + printf("vgetrandom_get_state failed!\n"); + exit(KSFT_FAIL); + } + } + return grnd_ctx.fn(buf, len, flags, state); +} + +enum { TRIALS = 25000000, THREADS = 256 }; + +static void *test_vdso_getrandom(void *) +{ + for (size_t i = 0; i < TRIALS; ++i) { + unsigned int val; + ssize_t ret = vgetrandom(&val, sizeof(val), 0); + assert(ret == sizeof(val)); + } + return NULL; +} + +static void *test_libc_getrandom(void *) +{ + for (size_t i = 0; i < TRIALS; ++i) { + unsigned int val; + ssize_t ret = getrandom(&val, sizeof(val), 0); + assert(ret == sizeof(val)); + } + return NULL; +} + +static void *test_syscall_getrandom(void *) +{ + for (size_t i = 0; i < TRIALS; ++i) { + unsigned int val; + ssize_t ret = syscall(SYS_getrandom, &val, sizeof(val), 0); + assert(ret == sizeof(val)); + } + return NULL; +} + +static void bench_single(void) +{ + struct timespec start, end, diff; + + clock_gettime(CLOCK_MONOTONIC, &start); + test_vdso_getrandom(NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + timespecsub(&end, &start, &diff); + printf(" vdso: %u times in %lu.%09lu seconds\n", TRIALS, diff.tv_sec, diff.tv_nsec); + + clock_gettime(CLOCK_MONOTONIC, &start); + test_libc_getrandom(NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + timespecsub(&end, &start, &diff); + printf(" libc: %u times in %lu.%09lu seconds\n", TRIALS, diff.tv_sec, diff.tv_nsec); + + clock_gettime(CLOCK_MONOTONIC, &start); + test_syscall_getrandom(NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + timespecsub(&end, &start, &diff); + printf("syscall: %u times in %lu.%09lu seconds\n", TRIALS, diff.tv_sec, diff.tv_nsec); +} + +static void bench_multi(void) +{ + struct timespec start, end, diff; + pthread_t threads[THREADS]; + + clock_gettime(CLOCK_MONOTONIC, &start); + for (size_t i = 0; i < THREADS; ++i) + assert(pthread_create(&threads[i], NULL, test_vdso_getrandom, NULL) == 0); + for (size_t i = 0; i < THREADS; ++i) + pthread_join(threads[i], NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + timespecsub(&end, &start, &diff); + printf(" vdso: %u x %u times in %lu.%09lu seconds\n", TRIALS, THREADS, diff.tv_sec, diff.tv_nsec); + + clock_gettime(CLOCK_MONOTONIC, &start); + for (size_t i = 0; i < THREADS; ++i) + assert(pthread_create(&threads[i], NULL, test_libc_getrandom, NULL) == 0); + for (size_t i = 0; i < THREADS; ++i) + pthread_join(threads[i], NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + timespecsub(&end, &start, &diff); + printf(" libc: %u x %u times in %lu.%09lu seconds\n", TRIALS, THREADS, diff.tv_sec, diff.tv_nsec); + + clock_gettime(CLOCK_MONOTONIC, &start); + for (size_t i = 0; i < THREADS; ++i) + assert(pthread_create(&threads[i], NULL, test_syscall_getrandom, NULL) == 0); + for (size_t i = 0; i < THREADS; ++i) + pthread_join(threads[i], NULL); + clock_gettime(CLOCK_MONOTONIC, &end); + timespecsub(&end, &start, &diff); + printf(" syscall: %u x %u times in %lu.%09lu seconds\n", TRIALS, THREADS, diff.tv_sec, diff.tv_nsec); +} + +static void fill(void) +{ + uint8_t weird_size[323929]; + for (;;) + vgetrandom(weird_size, sizeof(weird_size), 0); +} + +static void kselftest(void) +{ + uint8_t weird_size[1263]; + + ksft_print_header(); + ksft_set_plan(1); + + for (size_t i = 0; i < 1000; ++i) { + ssize_t ret = vgetrandom(weird_size, sizeof(weird_size), 0); + if (ret != sizeof(weird_size)) + exit(KSFT_FAIL); + } + + ksft_test_result_pass("getrandom: PASS\n"); + exit(KSFT_PASS); +} + +static void usage(const char *argv0) +{ + fprintf(stderr, "Usage: %s [bench-single|bench-multi|fill]\n", argv0); +} + +int main(int argc, char *argv[]) +{ + if (argc == 1) { + kselftest(); + return 0; + } + + if (argc != 2) { + usage(argv[0]); + return 1; + } + if (!strcmp(argv[1], "bench-single")) + bench_single(); + else if (!strcmp(argv[1], "bench-multi")) + bench_multi(); + else if (!strcmp(argv[1], "fill")) + fill(); + else { + usage(argv[0]); + return 1; + } + return 0; +}
Provide a generic C vDSO getrandom() implementation, which operates on an opaque state returned by vgetrandom_alloc() and produces random bytes the same way as getrandom(). This has a the API signature: ssize_t vgetrandom(void *buffer, size_t len, unsigned int flags, void *opaque_state); The return value and the first 3 arguments are the same as ordinary getrandom(), while the last argument is a pointer to the opaque allocated state. Were all four arguments passed to the getrandom() syscall, nothing different would happen, and the functions would have the exact same behavior. The actual vDSO RNG algorithm implemented is the same one implemented by drivers/char/random.c, using the same fast-erasure techniques as that. Should the in-kernel implementation change, so too will the vDSO one. It requires an implementation of ChaCha20 that does not use any stack, in order to maintain forward secrecy if a multi-threaded program forks (though this does not account for a similar issue with SA_SIGINFO copying registers to the stack), so this is left as an architecture-specific fill-in. Stack-less ChaCha20 is an easy algorithm to implement on a variety of architectures, so this shouldn't be too onerous. Initially, the state is keyless, and so the first call makes a getrandom() syscall to generate that key, and then uses it for subsequent calls. By keeping track of a generation counter, it knows when its key is invalidated and it should fetch a new one using the syscall. Later, more than just a generation counter might be used. Since MADV_WIPEONFORK is set on the opaque state, the key and related state is wiped during a fork(), so secrets don't roll over into new processes, and the same state doesn't accidentally generate the same random stream. The generation counter, as well, is always >0, so that the 0 counter is a useful indication of a fork() or otherwise uninitialized state. If the kernel RNG is not yet initialized, then the vDSO always calls the syscall, because that behavior cannot be emulated in userspace, but fortunately that state is short lived and only during early boot. If it has been initialized, then there is no need to inspect the `flags` argument, because the behavior does not change post-initialization regardless of the `flags` value. Since the opaque state passed to it is mutated, vDSO getrandom() is not reentrant, when used with the same opaque state, which libc should be mindful of. vgetrandom_alloc() and vDSO getrandom() provide the ability for userspace to generate random bytes quickly and safely, and are intended to be integrated into libc's thread management. As an illustrative example, together with the example code from "random: add vgetrandom_alloc() syscall", the following code might be used to do the same outside of libc. In a libc, only the non-static vgetrandom() function at the end would be exported as part of a getrandom() implementations, and the various pthread-isms are expected to be elided into libc internals. static struct { ssize_t(*fn)(void *buf, size_t len, unsigned long flags, void *state); pthread_key_t key; pthread_once_t initialized; } grnd_ctx = { .initialized = PTHREAD_ONCE_INIT }; static void vgetrandom_init(void) { if (pthread_key_create(&grnd_ctx.key, vgetrandom_put_state) != 0) return; grnd_ctx.fn = vdso_sym("LINUX_2.6", "__vdso_getrandom"); } ssize_t vgetrandom(void *buf, size_t len, unsigned long flags) { void *state; pthread_once(&grnd_ctx.initialized, vgetrandom_init); if (!grnd_ctx.fn) return getrandom(buf, len, flags); state = pthread_getspecific(grnd_ctx.key); if (!state) { state = vgetrandom_get_state(); if (pthread_setspecific(grnd_ctx.key, state) != 0) { vgetrandom_put_state(state); state = NULL; } if (!state) return getrandom(buf, len, flags); } return grnd_ctx.fn(buf, len, flags, state); } Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com> --- MAINTAINERS | 1 + drivers/char/random.c | 11 + include/vdso/datapage.h | 12 + include/vdso/getrandom.h | 32 +- include/vdso/types.h | 35 +++ lib/vdso/getrandom.c | 226 ++++++++++++++ tools/testing/selftests/vDSO/.gitignore | 1 + tools/testing/selftests/vDSO/Makefile | 2 + .../selftests/vDSO/vdso_test_getrandom.c | 283 ++++++++++++++++++ 9 files changed, 601 insertions(+), 2 deletions(-) create mode 100644 include/vdso/types.h create mode 100644 lib/vdso/getrandom.c create mode 100644 tools/testing/selftests/vDSO/vdso_test_getrandom.c