From patchwork Thu Jan 12 17:02:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yann Droneaud X-Patchwork-Id: 13098509 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8E39C61DB3 for ; Thu, 12 Jan 2023 17:45:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236181AbjALRpi (ORCPT ); Thu, 12 Jan 2023 12:45:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237251AbjALRop (ORCPT ); Thu, 12 Jan 2023 12:44:45 -0500 Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1878414037; Thu, 12 Jan 2023 09:03:36 -0800 (PST) Received: from localhost (unknown [IPv6:2a01:e35:39f2:1220:dc8b:b602:9bcd:3004]) by smtp6-g21.free.fr (Postfix) with ESMTPS id D59497803AE; Thu, 12 Jan 2023 18:03:18 +0100 (CET) From: Yann Droneaud To: "Jason A. Donenfeld" , "Theodore Ts'o" Cc: Yann Droneaud , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , x86@kernel.org, linux-crypto@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, Florian Weimer , Adhemerval Zanella Netto , "Carlos O'Donell" Subject: [RFC PATCH 1/4] random: introduce getrandom() GRND_TIMESTAMP Date: Thu, 12 Jan 2023 18:02:33 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In the pursuit of implementing an userspace arc4random() fast enough to be used in place of rand(), random(), lrand48(), mrand48(), etc., and as strong as getrandom(), it was found that calling getrandom() to generate one uint32_t at a time is not fast enough (see [1] for example). As noted by Florian Weimer in [2]: "The performance numbers suggest that we benefit from buffering in user space. It might not be necessary to implement expansion in userspace. getrandom (or /dev/urandom) with a moderately-sized buffer could be sufficient." Generating multiple values ahead of time with a proper CSPRNG helps achieve better performances. But buffering in userspace come with a lot of security related hurdles and pitfalls. As noted by Jason A. Donenfeld in [3] "For example, the kernel reseeds itself when virtual machines fork using an identifier passed to the kernel via ACPI. It also reseeds itself on system resume, both from ordinary S3 sleep but also, more importantly, from hibernation." Ignoring for now the issue of securily store the buffered random values in memory to achieve forward secrecy [4], it's possible to devise a mechanism to help userspace to know when to discard the values generated/buffered from a previous call to getrandom() so that the VM and/or resume issue can be dealt with at userspace level. Instead of adding a new system call, this patch shoehorns a query mechanism in getrandom() syscall by adding a mean to get and test a "timestamp". Currently, the "timestamp" is a single 64bit integer, that maps to the kernel's base CSPRNG generation, inverted, so that 0 means unintialized. GRND_TIMESTAMP allows userspace to ask the kernel if previous "timestamp" has changed as the result of an event that triggered kernel CSPRNG reseed, and to update the "timestamp". In case the "timestamp" hasn't changed, userspace CSPRNG can consume a slice of its buffered random stream. If it has changed, remaining userspace buffered random values should be discarded. Userspace should call getrandom() to fill and/or generate its buffer with updated seed. It's advised to test again the "timestamp" to deal with the race condition, where the kernel reseed just after the call to getrandom() to get entropy. How to not use it (because it doesn't have reseed on fork(), aka. MADV_WIPEONFORK, and forward secrecy buffer protection aka. mlock(), see [4]): static bool expired(void) { static uint64_t grnd_ts; ret = getrandom(&grnd_ts, sizeof(grnd_ts), GRND_TIMESTAMP); if (ret < 0) abort(); /* TODO: proper fallback to unbuffered getrandom() */ return !!ret; } uint32_t arc4random(void) { static uint32_t buffer[128]; /* TODO: mlock() buffer memory */ static unsigned int avail; uint32_t val; while (expired() || !avail) { getrandom(buffer, sizeof(buffer), 0); avail = 128; } avail--; val = buffer[avail]; buffer[avail] = 0; return val; } As the "timestamp" query has to be made for each uint32_t value generated by arc4random(), the query should be as lightweight as possible, thus it's expected GRND_TIMESTAMP to be handled by at the vDSO level to prevent a system call. [1] https://lore.kernel.org/all/874jt0kndq.fsf@oldenburg.str.redhat.com/ [2] https://sourceware.org/pipermail/libc-alpha/2022-July/140963.html [3] https://sourceware.org/pipermail/libc-alpha/2022-July/140939.html [4] https://lore.kernel.org/all/20230101162910.710293-1-Jason@zx2c4.com/ Link: https://lore.kernel.org/all/cover.1673539719.git.ydroneaud@opteya.com/ Cc: linux-crypto@vger.kernel.org Cc: Theodore Ts'o Cc: Adhemerval Zanella Netto Cc: Florian Weimer Cc: Jason A. Donenfeld Signed-off-by: Yann Droneaud --- drivers/char/random.c | 46 ++++++++++++++++++++++++++++++++++++- include/linux/random.h | 31 +++++++++++++++++++++++++ include/uapi/linux/random.h | 2 ++ 3 files changed, 78 insertions(+), 1 deletion(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index ce3ccd172cc8..9e2a37e432c0 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1361,15 +1361,59 @@ static void __cold try_to_generate_entropy(void) * **********************************************************************/ +static ssize_t get_random_timestamp(char __user *ubuf, size_t len, unsigned int flags) +{ + u64 ts; + + /* other combination not supported */ + if (WARN(flags != GRND_TIMESTAMP, "GRND_TIMESTAMP cannot be used with other flags")) + return -EINVAL; + + /* shorter structure not supported */ + if (len < sizeof(ts)) + return -EINVAL; + + if (copy_from_user(&ts, ubuf, sizeof(ts))) + return -EFAULT; + + /* longer structure supported, only if 0-padded, + timestamp might be extended in the future with more fields */ + if (len > sizeof(ts)) { + char __user *p = ubuf + sizeof(ts); + size_t l = len - sizeof(ts); + + while (l) { + char b; + + if (get_user(b, p++)) + return -EFAULT; + + if (b) + return -EINVAL; + } + } + + if (!get_random_timestamp_update(&ts, READ_ONCE(base_crng.generation))) + return 0; + + if (copy_to_user(ubuf, &ts, sizeof(ts))) + return -EFAULT; + + return sizeof(ts); +} + SYSCALL_DEFINE3(getrandom, char __user *, ubuf, size_t, len, unsigned int, flags) { struct iov_iter iter; struct iovec iov; int ret; - if (flags & ~(GRND_NONBLOCK | GRND_RANDOM | GRND_INSECURE)) + if (flags & ~(GRND_NONBLOCK | GRND_RANDOM | GRND_INSECURE | GRND_TIMESTAMP)) return -EINVAL; + if (unlikely(flags & GRND_TIMESTAMP)) + return get_random_timestamp(ubuf, len, flags); + /* * Requesting insecure and blocking randomness at the same time makes * no sense. diff --git a/include/linux/random.h b/include/linux/random.h index b0a940af4fff..bc219b5a96a5 100644 --- a/include/linux/random.h +++ b/include/linux/random.h @@ -161,4 +161,35 @@ int random_online_cpu(unsigned int cpu); extern const struct file_operations random_fops, urandom_fops; #endif +/* + * get_random_timestamp_update() + * + * @generation: current CRNG generation (from base_crng.generation + * or _vdso_rng_data.generation) + * + * Return: timestamp size if previous timestamp is expired and is updated, + * 0 if not expired (and not updated) + */ +static inline bool get_random_timestamp_update(u64 *user_ts, + u64 generation) +{ + u64 ts; + + /* base_crng.generation is never ULONG_MAX, + * OTOH userspace will initialize its timestamp + * to 0, so inverting base_crng.generation ensure + * first call to getrandom(,,GRND_TIMESTAMP) will + * update + */ + ts = ~generation; + + /* not expired ? no refresh suggested */ + if (*user_ts == ts) + return false; + + *user_ts = ts; + + return true; +} + #endif /* _LINUX_RANDOM_H */ diff --git a/include/uapi/linux/random.h b/include/uapi/linux/random.h index e744c23582eb..b433fb8d79ac 100644 --- a/include/uapi/linux/random.h +++ b/include/uapi/linux/random.h @@ -50,9 +50,11 @@ struct rand_pool_info { * GRND_NONBLOCK Don't block and return EAGAIN instead * GRND_RANDOM No effect * GRND_INSECURE Return non-cryptographic random bytes + * GRND_TIMESTAMP Interpret buffer as an opaque timestamp structure */ #define GRND_NONBLOCK 0x0001 #define GRND_RANDOM 0x0002 #define GRND_INSECURE 0x0004 +#define GRND_TIMESTAMP 0x0008 #endif /* _UAPI_LINUX_RANDOM_H */ From patchwork Thu Jan 12 17:02:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yann Droneaud X-Patchwork-Id: 13098510 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5053CC677F1 for ; Thu, 12 Jan 2023 17:46:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232065AbjALRqD (ORCPT ); Thu, 12 Jan 2023 12:46:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237643AbjALRo7 (ORCPT ); Thu, 12 Jan 2023 12:44:59 -0500 Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [IPv6:2a01:e0c:1:1599::15]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7760644378; Thu, 12 Jan 2023 09:03:57 -0800 (PST) Received: from localhost (unknown [IPv6:2a01:e35:39f2:1220:dc8b:b602:9bcd:3004]) by smtp6-g21.free.fr (Postfix) with ESMTPS id 87B7078036C; Thu, 12 Jan 2023 18:03:39 +0100 (CET) From: Yann Droneaud To: "Jason A. Donenfeld" , "Theodore Ts'o" Cc: Yann Droneaud , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , x86@kernel.org, linux-crypto@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, Florian Weimer , Adhemerval Zanella Netto , "Carlos O'Donell" Subject: [RFC PATCH 2/4] random: introduce generic vDSO getrandom(,, GRND_TIMESTAMP) fast path Date: Thu, 12 Jan 2023 18:02:34 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: "Jason A. Donenfeld" Exports base_crng.generation to vDSO and adds getrandom() to the vDSO. Based on Jason A. Donenfeld patch [1] "[PATCH v14 6/7] random: introduce generic vDSO getrandom() implementation", but deal only with GRND_TIMESTAMP in vDSO: generating random stream is left to the getrandom() syscall. [1] https://lore.kernel.org/all/20230101162910.710293-7-Jason@zx2c4.com/ Link: https://lore.kernel.org/all/cover.1673539719.git.ydroneaud@opteya.com/ Signed-off-by: Yann Droneaud --- MAINTAINERS | 1 + drivers/char/random.c | 6 +++++ include/vdso/datapage.h | 9 ++++++++ lib/vdso/Kconfig | 5 ++++ lib/vdso/getrandom.c | 51 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 72 insertions(+) create mode 100644 lib/vdso/getrandom.c diff --git a/MAINTAINERS b/MAINTAINERS index 7f86d02cb427..20e1fabcb2e9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17521,6 +17521,7 @@ T: git https://git.kernel.org/pub/scm/linux/kernel/git/crng/random.git S: Maintained F: drivers/char/random.c F: drivers/virt/vmgenid.c +F: lib/vdso/getrandom.c RAPIDIO SUBSYSTEM M: Matt Porter diff --git a/drivers/char/random.c b/drivers/char/random.c index 9e2a37e432c0..a60f50c95ab1 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -56,6 +56,9 @@ #include #include #include +#ifdef CONFIG_VDSO_GETRANDOM +#include +#endif #include #include #include @@ -271,6 +274,9 @@ static void crng_reseed(struct work_struct *work) if (next_gen == ULONG_MAX) ++next_gen; WRITE_ONCE(base_crng.generation, next_gen); +#ifdef CONFIG_VDSO_GETRANDOM + smp_store_release(&_vdso_rng_data.generation, next_gen); +#endif if (!static_branch_likely(&crng_is_ready)) crng_init = CRNG_READY; spin_unlock_irqrestore(&base_crng.lock, flags); diff --git a/include/vdso/datapage.h b/include/vdso/datapage.h index 73eb622e7663..7ae8e7ffe3ba 100644 --- a/include/vdso/datapage.h +++ b/include/vdso/datapage.h @@ -109,6 +109,14 @@ struct vdso_data { struct arch_vdso_data arch_data; }; +/** + * struct vdso_rng_data - vdso RNG state information + * @generation: counter representing the number of RNG reseeds + */ +struct vdso_rng_data { + u64 generation; +}; + /* * We use the hidden visibility to prevent the compiler from generating a GOT * relocation. Not only is going through a GOT useless (the entry couldn't and @@ -120,6 +128,7 @@ struct vdso_data { */ extern struct vdso_data _vdso_data[CS_BASES] __attribute__((visibility("hidden"))); extern struct vdso_data _timens_data[CS_BASES] __attribute__((visibility("hidden"))); +extern struct vdso_rng_data _vdso_rng_data __attribute__((visibility("hidden"))); /* * The generic vDSO implementation requires that gettimeofday.h diff --git a/lib/vdso/Kconfig b/lib/vdso/Kconfig index d883ac299508..3b394fa83f65 100644 --- a/lib/vdso/Kconfig +++ b/lib/vdso/Kconfig @@ -31,3 +31,8 @@ config GENERIC_VDSO_TIME_NS VDSO endif + +config VDSO_GETRANDOM + bool + help + Selected by architectures that support vDSO getrandom(). diff --git a/lib/vdso/getrandom.c b/lib/vdso/getrandom.c new file mode 100644 index 000000000000..827351a87002 --- /dev/null +++ b/lib/vdso/getrandom.c @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ + +#include +#include +#include +#include +#include +#include + +/** + * __cvdso_getrandom_data - Generic vDSO implementation of getrandom() syscall. + * @rng_info: Describes state of kernel RNG, memory shared with kernel. + * @buffer: Input/Output buffer. + * @len: Size of @buffer in bytes. + * @flags: Zero or more GRND_* flags. + */ +static __always_inline ssize_t +__cvdso_getrandom_data(const struct vdso_rng_data *rng_info, void *buffer, size_t len, + unsigned int flags) +{ + if (flags != GRND_TIMESTAMP) + goto fallback; + + if (unlikely(!buffer)) + goto fallback; + + /* want aligned access */ + if (unlikely(!IS_ALIGNED((uintptr_t)buffer, __alignof__(u64)))) + goto fallback; + + if (unlikely(len != sizeof(u64))) + goto fallback; + + if (!get_random_timestamp_update((u64 *)buffer, + READ_ONCE(rng_info->generation))) + return 0; + + return sizeof(u64); + +fallback: + return getrandom_syscall(buffer, len, flags); +} + +static __always_inline ssize_t +__cvdso_getrandom(void *buffer, size_t len, unsigned int flags) +{ + return __cvdso_getrandom_data(__arch_get_vdso_rng_data(), buffer, len, flags); +} From patchwork Thu Jan 12 17:02:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yann Droneaud X-Patchwork-Id: 13098511 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B33BBC61DB3 for ; Thu, 12 Jan 2023 17:46:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238033AbjALRqS (ORCPT ); Thu, 12 Jan 2023 12:46:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232594AbjALRpW (ORCPT ); Thu, 12 Jan 2023 12:45:22 -0500 Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [IPv6:2a01:e0c:1:1599::15]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F199448CD8; Thu, 12 Jan 2023 09:04:16 -0800 (PST) Received: from localhost (unknown [IPv6:2a01:e35:39f2:1220:dc8b:b602:9bcd:3004]) by smtp6-g21.free.fr (Postfix) with ESMTPS id 90EBF780357; Thu, 12 Jan 2023 18:03:58 +0100 (CET) From: Yann Droneaud To: "Jason A. Donenfeld" , "Theodore Ts'o" Cc: Yann Droneaud , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , x86@kernel.org, linux-crypto@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, Florian Weimer , Adhemerval Zanella Netto , "Carlos O'Donell" Subject: [RFC PATCH 3/4] x86: vdso: Wire up getrandom() vDSO implementation. Date: Thu, 12 Jan 2023 18:02:35 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: "Jason A. Donenfeld" Hook up the generic vDSO implementation to the x86 vDSO data page. Since the existing vDSO infrastructure is heavily based on the timekeeping functionality, which works over arrays of bases, a new macro is introduced for vvars that are not arrays. Based on Jason A. Donenfeld patch [1] "[PATCH v14 7/7] x86: vdso: Wire up getrandom() vDSO implementation" removing the ChaCha20 implementation and opaque state argument from vDSO getrandom(). [1] https://lore.kernel.org/all/20230101162910.710293-8-Jason@zx2c4.com/ Link: https://lore.kernel.org/all/cover.1673539719.git.ydroneaud@opteya.com/ Signed-off-by: Yann Droneaud --- arch/x86/Kconfig | 1 + arch/x86/entry/vdso/Makefile | 3 +- arch/x86/entry/vdso/vdso.lds.S | 2 ++ arch/x86/entry/vdso/vgetrandom.c | 17 +++++++++++ arch/x86/include/asm/vdso/getrandom.h | 42 +++++++++++++++++++++++++++ arch/x86/include/asm/vdso/vsyscall.h | 2 ++ arch/x86/include/asm/vvar.h | 16 ++++++++++ 7 files changed, 82 insertions(+), 1 deletion(-) create mode 100644 arch/x86/entry/vdso/vgetrandom.c create mode 100644 arch/x86/include/asm/vdso/getrandom.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..df48387f019f 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -272,6 +272,7 @@ config X86 select HAVE_UNSTABLE_SCHED_CLOCK select HAVE_USER_RETURN_NOTIFIER select HAVE_GENERIC_VDSO + select VDSO_GETRANDOM select HOTPLUG_SMT if SMP select IRQ_FORCED_THREADING select NEED_PER_CPU_EMBED_FIRST_CHUNK diff --git a/arch/x86/entry/vdso/Makefile b/arch/x86/entry/vdso/Makefile index 838613ac15b8..2565c4702f54 100644 --- a/arch/x86/entry/vdso/Makefile +++ b/arch/x86/entry/vdso/Makefile @@ -27,7 +27,7 @@ VDSO32-$(CONFIG_X86_32) := y VDSO32-$(CONFIG_IA32_EMULATION) := y # files to link into the vdso -vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o +vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o vgetrandom.o vobjs32-y := vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o vobjs32-y += vdso32/vclock_gettime.o vobjs-$(CONFIG_X86_SGX) += vsgx.o @@ -105,6 +105,7 @@ CFLAGS_REMOVE_vclock_gettime.o = -pg CFLAGS_REMOVE_vdso32/vclock_gettime.o = -pg CFLAGS_REMOVE_vgetcpu.o = -pg CFLAGS_REMOVE_vsgx.o = -pg +CFLAGS_REMOVE_vgetrandom.o = -pg # # X32 processes use x32 vDSO to access 64bit kernel data. diff --git a/arch/x86/entry/vdso/vdso.lds.S b/arch/x86/entry/vdso/vdso.lds.S index e8c60ae7a7c8..0bab5f4af6d1 100644 --- a/arch/x86/entry/vdso/vdso.lds.S +++ b/arch/x86/entry/vdso/vdso.lds.S @@ -30,6 +30,8 @@ VERSION { #ifdef CONFIG_X86_SGX __vdso_sgx_enter_enclave; #endif + getrandom; + __vdso_getrandom; local: *; }; } diff --git a/arch/x86/entry/vdso/vgetrandom.c b/arch/x86/entry/vdso/vgetrandom.c new file mode 100644 index 000000000000..157a6f7dbc44 --- /dev/null +++ b/arch/x86/entry/vdso/vgetrandom.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ +#include + +#include "../../../../lib/vdso/getrandom.c" + +ssize_t __vdso_getrandom(void *buffer, size_t len, unsigned int flags); + +ssize_t __vdso_getrandom(void *buffer, size_t len, unsigned int flags) +{ + return __cvdso_getrandom(buffer, len, flags); +} + +ssize_t getrandom(void *, size_t, unsigned int) + __attribute__((weak, alias("__vdso_getrandom"))); diff --git a/arch/x86/include/asm/vdso/getrandom.h b/arch/x86/include/asm/vdso/getrandom.h new file mode 100644 index 000000000000..14247ddc431a --- /dev/null +++ b/arch/x86/include/asm/vdso/getrandom.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2022 Jason A. Donenfeld . All Rights Reserved. + */ +#ifndef __ASM_VDSO_GETRANDOM_H +#define __ASM_VDSO_GETRANDOM_H + +#ifndef __ASSEMBLY__ + +#include +#include + +/** + * getrandom_syscall - Invoke the getrandom() syscall. + * @buffer: Input/Output buffer. + * @len: Size of @buffer in bytes. + * @flags: Zero or more GRND_* flags. + * Returns the number of random bytes written to @buffer, or a negative value indicating an error. + */ +static __always_inline ssize_t getrandom_syscall(void *buffer, size_t len, unsigned int flags) +{ + long ret; + + asm ("syscall" : "=a" (ret) : + "0" (__NR_getrandom), "D" (buffer), "S" (len), "d" (flags) : + "rcx", "r11", "memory"); + + return ret; +} + +#define __vdso_rng_data (VVAR(_vdso_rng_data)) + +static __always_inline const struct vdso_rng_data *__arch_get_vdso_rng_data(void) +{ + if (__vdso_data->clock_mode == VDSO_CLOCKMODE_TIMENS) + return (void *)&__vdso_rng_data + ((void *)&__timens_vdso_data - (void *)&__vdso_data); + return &__vdso_rng_data; +} + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ASM_VDSO_GETRANDOM_H */ diff --git a/arch/x86/include/asm/vdso/vsyscall.h b/arch/x86/include/asm/vdso/vsyscall.h index be199a9b2676..71c56586a22f 100644 --- a/arch/x86/include/asm/vdso/vsyscall.h +++ b/arch/x86/include/asm/vdso/vsyscall.h @@ -11,6 +11,8 @@ #include DEFINE_VVAR(struct vdso_data, _vdso_data); +DEFINE_VVAR_SINGLE(struct vdso_rng_data, _vdso_rng_data); + /* * Update the vDSO data page to keep in sync with kernel timekeeping. */ diff --git a/arch/x86/include/asm/vvar.h b/arch/x86/include/asm/vvar.h index 183e98e49ab9..9d9af37f7cab 100644 --- a/arch/x86/include/asm/vvar.h +++ b/arch/x86/include/asm/vvar.h @@ -26,6 +26,8 @@ */ #define DECLARE_VVAR(offset, type, name) \ EMIT_VVAR(name, offset) +#define DECLARE_VVAR_SINGLE(offset, type, name) \ + EMIT_VVAR(name, offset) #else @@ -37,6 +39,10 @@ extern char __vvar_page; extern type timens_ ## name[CS_BASES] \ __attribute__((visibility("hidden"))); \ +#define DECLARE_VVAR_SINGLE(offset, type, name) \ + extern type vvar_ ## name \ + __attribute__((visibility("hidden"))); \ + #define VVAR(name) (vvar_ ## name) #define TIMENS(name) (timens_ ## name) @@ -44,12 +50,22 @@ extern char __vvar_page; type name[CS_BASES] \ __attribute__((section(".vvar_" #name), aligned(16))) __visible +#define DEFINE_VVAR_SINGLE(type, name) \ + type name \ + __attribute__((section(".vvar_" #name), aligned(16))) __visible + #endif /* DECLARE_VVAR(offset, type, name) */ DECLARE_VVAR(128, struct vdso_data, _vdso_data) +#if !defined(_SINGLE_DATA) +#define _SINGLE_DATA +DECLARE_VVAR_SINGLE(640, struct vdso_rng_data, _vdso_rng_data) +#endif + #undef DECLARE_VVAR +#undef DECLARE_VVAR_SINGLE #endif From patchwork Thu Jan 12 17:02:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yann Droneaud X-Patchwork-Id: 13098512 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32693C63797 for ; Thu, 12 Jan 2023 17:46:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238133AbjALRqg (ORCPT ); Thu, 12 Jan 2023 12:46:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231349AbjALRp5 (ORCPT ); Thu, 12 Jan 2023 12:45:57 -0500 Received: from smtp6-g21.free.fr (smtp6-g21.free.fr [212.27.42.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7500952749; Thu, 12 Jan 2023 09:04:35 -0800 (PST) Received: from localhost (unknown [IPv6:2a01:e35:39f2:1220:dc8b:b602:9bcd:3004]) by smtp6-g21.free.fr (Postfix) with ESMTPS id 52CCC78036C; Thu, 12 Jan 2023 18:04:17 +0100 (CET) From: Yann Droneaud To: "Jason A. Donenfeld" , "Theodore Ts'o" Cc: Yann Droneaud , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Vincenzo Frascino , x86@kernel.org, linux-crypto@vger.kernel.org, linux-api@vger.kernel.org, linux-kernel@vger.kernel.org, Florian Weimer , Adhemerval Zanella Netto , "Carlos O'Donell" Subject: [RFC PATCH 4/4] testing: add a getrandom() GRND_TIMESTAMP vDSO demonstration/benchmark Date: Thu, 12 Jan 2023 18:02:36 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Link: https://lore.kernel.org/all/cover.1673539719.git.ydroneaud@opteya.com/ Signed-off-by: Yann Droneaud --- tools/testing/crypto/getrandom/Makefile | 4 + .../testing/crypto/getrandom/test-getrandom.c | 307 ++++++++++++++++++ 2 files changed, 311 insertions(+) create mode 100644 tools/testing/crypto/getrandom/Makefile create mode 100644 tools/testing/crypto/getrandom/test-getrandom.c diff --git a/tools/testing/crypto/getrandom/Makefile b/tools/testing/crypto/getrandom/Makefile new file mode 100644 index 000000000000..1370b6f1ae94 --- /dev/null +++ b/tools/testing/crypto/getrandom/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0+ + +test-getrandom: test-getrandom.c + $(CC) $(CPPFLAGS) $(CFLAGS) -I ../../../../usr/include/ -O2 -Wall -Wextra -o $@ $^ -ldl diff --git a/tools/testing/crypto/getrandom/test-getrandom.c b/tools/testing/crypto/getrandom/test-getrandom.c new file mode 100644 index 000000000000..311eef503f50 --- /dev/null +++ b/tools/testing/crypto/getrandom/test-getrandom.c @@ -0,0 +1,307 @@ +// SPDX-License-Identifier: GPL-2.0+ +/* + * Copyright (C) 2022 Yann Droneaud. All Rights Reserved. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +static size_t pagesz; +static size_t discarded; + +typedef ssize_t(*getrandom_fn) (void *, size_t, int); + +static bool grnd_timestamp; +static getrandom_fn getrandom_vDSO; + +static ssize_t getrandom_syscall(void *buffer, size_t size, int flags) +{ + return syscall(SYS_getrandom, buffer, size, flags); +} + +static ssize_t timestamp(getrandom_fn _getrandom, uint64_t *grnd_ts, + size_t size) +{ + ssize_t ret; + + ret = _getrandom(grnd_ts, size, GRND_TIMESTAMP); + if (ret < 0) { + fprintf(stderr, + "getrandom(,,GRND_TIMESTAMP) failed: %ld (%s)\n", -ret, + strerror((int)-ret)); + return -1; + } + + return ret; +} + +static void fetch(getrandom_fn _getrandom, void *buffer, size_t size) +{ + ssize_t ret; + + ret = _getrandom(buffer, size, 0); + if (ret < 0) { + fprintf(stderr, "getrandom(,,0) failed: %ld (%s)\n", -ret, + strerror((int)-ret)); + exit(EXIT_FAILURE); + } +} + +struct rng { + uint64_t grnd_ts; + size_t availsz; /* available bytes in buffer */ + size_t buffersz; /* buffer size */ + uint8_t buffer[]; +}; + +static struct rng *rng; + +static void init_rng(void) +{ + int r; + ssize_t s; + void *p; + + r = getpagesize(); + if (r == -1) { + fprintf(stderr, "getpagesize() failed: %d\n", errno); + exit(EXIT_FAILURE); + } + + pagesz = (size_t)r; + + p = mmap(NULL, pagesz, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (p == MAP_FAILED) { + fprintf(stderr, "mmap() failed: %d\n", errno); + exit(EXIT_FAILURE); + } + + r = madvise(p, pagesz, MADV_DONTDUMP | MADV_WIPEONFORK); + if (r == -1) { + fprintf(stderr, "madvise() failed: %d\n", errno); + exit(EXIT_FAILURE); + } + + r = mlock(p, pagesz); + if (r == -1) + fprintf(stderr, "mlock() failed: %d\n", errno); + + rng = p; + + s = timestamp(getrandom_syscall, &rng->grnd_ts, sizeof(rng->grnd_ts)); + if (s == -1) + return; + + printf("getrandom() support GRND_TIMESTAMP\n"); + + grnd_timestamp = true; +} + +static void init_vdso(void) +{ + void *h; + void *p; + + h = dlopen("linux-vdso.so.1", RTLD_LAZY | RTLD_LOCAL | RTLD_NOLOAD); + if (!h) { + fprintf(stderr, "failed to open vDSO: %s\n", dlerror()); + return; + } + + p = dlsym(h, "__vdso_getrandom"); + if (!p) { + fprintf(stderr, "getrandom() not found in vDSO: %s\n", + dlerror()); + return; + } + + printf("found getrandom() in vDSO at %p\n", p); + + getrandom_vDSO = p; +} + +/* + * 1) check timestamp isn't expired + * 2) if expired or there's not enough data in buffer + * a) if expired, reset buffer size, + * b) fetch new random stream + * c) check timestamp + * d) if expired, reset buffer size, go to b) + * + */ +static void ensure(getrandom_fn _getrandom, size_t request) +{ + ssize_t r; + + r = timestamp(_getrandom, &rng->grnd_ts, sizeof(rng->grnd_ts)); + switch (r) { + case 0: /* timestamp didn't change */ + /* enough available random bytes ? */ + if (rng->availsz >= request) + return; + + /* increase buffer size when drained */ + if (rng->buffersz < pagesz - sizeof(*rng)) + rng->buffersz *= 2; + + /* no less than 32 */ + if (rng->buffersz < 32) + rng->buffersz = 32; + + /* no more than a full page minus the rng structure */ + if (rng->buffersz > pagesz - sizeof(*rng)) + rng->buffersz = pagesz - sizeof(*rng); + + break; + + case sizeof(rng->grnd_ts): /* timestamp did change, random bytes must be discarded */ + rng->buffersz = 32; /* reset size */ + break; + + default: + fprintf(stderr, "unexpected timestamp size %zd\n", r); + exit(EXIT_FAILURE); + } + + /* keep fetching if timestamp is updated */ + for (;;) { + if (rng->availsz) + discarded += rng->availsz; + + fetch(_getrandom, rng->buffer, rng->buffersz); + rng->availsz = rng->buffersz; + + r = timestamp(_getrandom, &rng->grnd_ts, sizeof(rng->grnd_ts)); + + switch (r) { + case 0: /* timestamp didn't change between previous check and last fetch */ + return; + + case sizeof(rng->grnd_ts): /* timestamp did change, random bytes just fetched must be discarded */ + rng->buffersz = 32; /* reset size */ + continue; /* retry again */ + + default: + fprintf(stderr, "unexpected timestamp size %zd\n", r); + exit(EXIT_FAILURE); + } + } +} + +/* arc4random() */ +static void get_direct(getrandom_fn _getrandom) +{ + uint32_t v; + fetch(_getrandom, &v, sizeof(v)); +} + +static void get_pooled(getrandom_fn _getrandom) +{ + ensure(_getrandom, sizeof(uint32_t)); + rng->availsz -= sizeof(uint32_t); +} + +static inline struct timespec timespec_sub(const struct timespec *a, + const struct timespec *b) +{ + struct timespec res; + + res.tv_sec = a->tv_sec - b->tv_sec; + res.tv_nsec = a->tv_nsec - b->tv_nsec; + if (res.tv_nsec < 0) { + res.tv_sec--; + res.tv_nsec += 1000000000L; + } + + return res; +} + +#define SAMPLES 13 +#define VALUES (16 * 1024 * 1024) + +static void test_direct(getrandom_fn _getrandom, const char *method) +{ + struct timespec start, end, diff; + + for (int i = 0; i < SAMPLES; i++) { + clock_gettime(CLOCK_MONOTONIC, &start); + + for (uint32_t j = 0; j < VALUES; j++) + get_direct(_getrandom); + + clock_gettime(CLOCK_MONOTONIC, &end); + + diff = timespec_sub(&end, &start); + + printf("== direct %s getrandom(), %u u32, %lu.%09lu s, %.3f M u32/s, %.3f ns/u32\n", + method, VALUES, diff.tv_sec, diff.tv_nsec, + VALUES / (1000000 * + (diff.tv_sec + + (double)diff.tv_nsec / 1000000000UL)), + (double)(diff.tv_sec * 1000000000UL + + diff.tv_nsec) / VALUES); + } +} + +static void test_pooled(getrandom_fn _getrandom, const char *method) +{ + struct timespec start, end, diff; + + for (int i = 0; i < SAMPLES; i++) { + discarded = 0; + + clock_gettime(CLOCK_MONOTONIC, &start); + + for (uint32_t j = 0; j < VALUES; j++) + get_pooled(_getrandom); + + clock_gettime(CLOCK_MONOTONIC, &end); + + diff = timespec_sub(&end, &start); + + printf("== pooled %s getrandom(), %u u32, %lu.%09lu s, %.3f M u32/s, %.3f ns/u32, (%zu bytes discarded)\n", + method, VALUES, diff.tv_sec, diff.tv_nsec, + VALUES / (1000000 * + (diff.tv_sec + + (double)diff.tv_nsec / 1000000000UL)), + (double)(diff.tv_sec * 1000000000UL + + diff.tv_nsec) / VALUES, + discarded); + } +} + +int main(void) +{ + printf("getrandom(,,GRND_TIMESTAMP) test\n"); + + init_rng(); + init_vdso(); + + while (1) { + test_direct(getrandom_syscall, "syscall"); + + if (getrandom_vDSO) + test_direct(getrandom_vDSO, "vDSO"); + + if (grnd_timestamp) + test_pooled(getrandom_syscall, "syscall"); + + if (getrandom_vDSO && grnd_timestamp) + test_pooled(getrandom_vDSO, "vDSO"); + } + + return 0; +}