From patchwork Mon Sep 2 19:17:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 13787673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37732CA0ED3 for ; Mon, 2 Sep 2024 19:18:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C08F38D010D; Mon, 2 Sep 2024 15:17:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B91748D00EF; Mon, 2 Sep 2024 15:17:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E5DA8D010D; Mon, 2 Sep 2024 15:17:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7D66C8D00EF for ; Mon, 2 Sep 2024 15:17:59 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ED08F120792 for ; Mon, 2 Sep 2024 19:17:58 +0000 (UTC) X-FDA: 82520758236.05.9B28798 Received: from pegase2.c-s.fr (pegase2.c-s.fr [93.17.235.10]) by imf28.hostedemail.com (Postfix) with ESMTP id D00CCC0007 for ; Mon, 2 Sep 2024 19:17:56 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf28.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.235.10 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725304654; a=rsa-sha256; cv=none; b=c4PgqZQnzfp12WhNpSuzjvNrbXtKTapQOt/ULZbn5ITwD06a5/70IzxzytJzUzRD5YgLLA cZOmP4Lhi1yRE+yiippgxrFjbofcSQXWk3LywW9DhR4z4Fk5+zHek7XLJUiMPIDiKaMdBK 8vaSvmqa/reAZOud7DNiJ6v9rhApFek= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=csgroup.eu; spf=pass (imf28.hostedemail.com: domain of christophe.leroy@csgroup.eu designates 93.17.235.10 as permitted sender) smtp.mailfrom=christophe.leroy@csgroup.eu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725304654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TbUZIPNhHKK4TCKKF2Lc46zC0ioL7fEsik29vmA7pLw=; b=acf6bnQR2wE+toWs+pfSnFWmdghs8JSSINXm1vvDsFFif2P7+f8NnFS1a7xoIFLPhA/b6Q lEt2HRHBA8BBsZtM89M1SCMzDj9zKL5YzaeBP4S9SOsDj9U8W/K5NI3KAe0Ja3/lMDE3CR P4Fkn0vfuajhK3bf4hkllEVXOPrd4rM= Received: from localhost (mailhub3.si.c-s.fr [172.26.127.67]) by localhost (Postfix) with ESMTP id 4WyJRg4Yrzz9sSS; Mon, 2 Sep 2024 21:17:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from pegase2.c-s.fr ([172.26.127.65]) by localhost (pegase2.c-s.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UHuwOflyJahA; Mon, 2 Sep 2024 21:17:35 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase2.c-s.fr (Postfix) with ESMTP id 4WyJRg3FM1z9sS7; Mon, 2 Sep 2024 21:17:35 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 621068B76E; Mon, 2 Sep 2024 21:17:35 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id nRvxAuIibIDy; Mon, 2 Sep 2024 21:17:35 +0200 (CEST) Received: from PO20335.idsi0.si.c-s.fr (unknown [192.168.234.167]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 657548B773; Mon, 2 Sep 2024 21:17:34 +0200 (CEST) From: Christophe Leroy To: Andrew Morton , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Michael Ellerman , Nicholas Piggin , Naveen N Rao , Nathan Chancellor , Nick Desaulniers , Bill Wendling , Justin Stitt , Shuah Khan , "Jason A . Donenfeld" Cc: Christophe Leroy , linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-kselftest@vger.kernel.org, llvm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, Adhemerval Zanella , Xi Ruoyao Subject: [PATCH v5 5/5] powerpc/vdso: Wire up getrandom() vDSO implementation on VDSO64 Date: Mon, 2 Sep 2024 21:17:22 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1725304638; l=7245; i=christophe.leroy@csgroup.eu; s=20211009; h=from:subject:message-id; bh=1u98q0W43xTrsnG50liffh4ZTxumv6Ho4dq/SBaH6jA=; b=lM1uhuWNjxthaHQDf7uVaaSIKJ42G+9/6cAo7nbd76NGIaBABIDpKtUqexJkcPnaewmAS32q3 h+Nv5kPU5p2DUadEnB+UvL81zdHxza+GA275Mngu9p5wM9cdC8ZywAr X-Developer-Key: i=christophe.leroy@csgroup.eu; a=ed25519; pk=HIzTzUj91asvincQGOFx6+ZF5AoUuP9GdOtQChs7Mm0= X-Rspam-User: X-Rspamd-Queue-Id: D00CCC0007 X-Rspamd-Server: rspam01 X-Stat-Signature: raykg3mwzu59zncuyrnwb5gekyrwij97 X-HE-Tag: 1725304676-775548 X-HE-Meta: U2FsdGVkX18goKVQYtvqqR2z+MN7AQrZdyFb03TXOSUfzzKiIFjfANwy1AiLEI41/CN96zK61RZ3LbjYUXlwyQrkBxMFinkjd/HJo9Oo8By/qP9LvVTgDlj2ENwtFNeQ+yPkmIG9sGcPXEGrn6VKE/jWsnawBgELQN0WruGiRt7eWkU5vqOQCbK+PC1vfHPa0yNmYSwRnHgXDfalNcvdutcFIvrTmwLFwyn/8hFsNdc/En9Q/DlrC44T3nvfHmo7QlZrAV9FyK1T3Frr2FY6RsNRJudaza0LFdJ1XR/082B4Fniy1BJMhHJ6xyb6F0/CacFYZXmZB6G8qauk5aeChe/Y9Q1M1LN0wNizEtJpjZsVXUeRtgBtWqx7ydqCw1hcsdvECGQMwFDSDBh19TOjAF/HlM/27XvztOnedz19uCRrb2PSnBX/QP7eIWSjNZXfGag+WmEIZvLwLjV9uw0y9FDpVuXDyxSF9jjUBMux0duDI+PB8SoafrzKhAq0bOHcmfxjOlQxgLICHJ0UkPcIoAhfbCGyKphE71WWLr7nHc7WA4+qDR3sWG7stp9DsTYWxJeLc+QooL201OR0Bnje1v9ndYpIMizvqApBdiivrULxl4ebjPKHCa3FsVoGJW8MVE1cyz1kvvaXiRsR0tgOG+IX8iTtxaH+nFzOIPK1bR5sGbVt7Gv726m1tCuZq7uD2wyxHnDmB9j1WmNTScIsv4j+RiMlhBC+KTSOJqAMvjFPA0DMzMS/0J61UHSLdF/ePAaBKV6vqxd/BQ6Ozst/U/+l1NNYJw1FrlbIXC1ac+qS2ftYO6w/umWDF3Iuf2UzltLNNrYZoSPBS4TuqaVknV9KzNffuEBr8RtAWPFYN+JHyZm9n8FW/FEIPX2MtBD5th9rfgzc1JLGXwn/2d889kMceAr1zh+W3l5pi5zEF/qLdVDeqG26uXTTR/OOpC7E5LKNVztjDeXF2nHDgxh Yx6WdAIn N0Cii7rf/zs8dsJmBBtTAyCaJxI2dvwcVvvDMQ2YT1NTFnTOz2x0xdGQxWi5Ee5TjncBX9nJGR1W+2+BZzZYjLUfcy/Of8Izkt5x5gMX88nIdXpp0A3A8p279uw4iXhcigGlB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Extend getrandom() vDSO implementation to VDSO64 Tested on QEMU on both ppc64_defconfig and ppc64le_defconfig. The results are not precise as it is QEMU on an x86 laptop, but no need to be precise to see the benefit. ~ # ./vdso_test_getrandom bench-single vdso: 25000000 times in 4.977777162 seconds libc: 25000000 times in 75.516749981 seconds syscall: 25000000 times in 86.842242014 seconds ~ # ./vdso_test_getrandom bench-single vdso: 25000000 times in 6.473814156 seconds libc: 25000000 times in 73.875109463 seconds syscall: 25000000 times in 71.805066229 seconds Signed-off-by: Christophe Leroy Tested-by: Madhavan Srinivasan --- v5: - VDSO32 for both PPC32 and PPC64 is in previous patch. This patch have the logic for VDSO64. v4: - Use __BIG_ENDIAN__ which is defined by GCC instead of CONFIG_CPU_BIG_ENDIAN which is unknown by selftests - Implement a cleaner/smaller output copy for little endian instead of keeping compat macro. v3: New (split out of previous patch) --- arch/powerpc/Kconfig | 2 +- arch/powerpc/kernel/vdso/Makefile | 8 ++- arch/powerpc/kernel/vdso/getrandom.S | 8 +++ arch/powerpc/kernel/vdso/vdso64.lds.S | 1 + arch/powerpc/kernel/vdso/vgetrandom-chacha.S | 53 ++++++++++++++++++++ 5 files changed, 69 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e500a59ddecc..b45452ac4a73 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -311,7 +311,7 @@ config PPC select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK select TRACE_IRQFLAGS_SUPPORT - select VDSO_GETRANDOM if VDSO32 + select VDSO_GETRANDOM # # Please keep this list sorted alphabetically. # diff --git a/arch/powerpc/kernel/vdso/Makefile b/arch/powerpc/kernel/vdso/Makefile index 7a4a935406d8..56fb1633529a 100644 --- a/arch/powerpc/kernel/vdso/Makefile +++ b/arch/powerpc/kernel/vdso/Makefile @@ -9,6 +9,7 @@ obj-vdso32 = sigtramp32-32.o gettimeofday-32.o datapage-32.o cacheflush-32.o not obj-vdso64 = sigtramp64-64.o gettimeofday-64.o datapage-64.o cacheflush-64.o note-64.o getcpu-64.o obj-vdso32 += getrandom-32.o vgetrandom-chacha-32.o +obj-vdso64 += getrandom-64.o vgetrandom-chacha-64.o ifneq ($(c-gettimeofday-y),) CFLAGS_vgettimeofday-32.o += -include $(c-gettimeofday-y) @@ -21,6 +22,7 @@ endif ifneq ($(c-getrandom-y),) CFLAGS_vgetrandom-32.o += -include $(c-getrandom-y) + CFLAGS_vgetrandom-64.o += -include $(c-getrandom-y) $(call cc-option, -ffixed-r30) endif # Build rules @@ -34,7 +36,7 @@ endif targets := $(obj-vdso32) vdso32.so.dbg vgettimeofday-32.o vgetrandom-32.o targets += crtsavres-32.o obj-vdso32 := $(addprefix $(obj)/, $(obj-vdso32)) -targets += $(obj-vdso64) vdso64.so.dbg vgettimeofday-64.o +targets += $(obj-vdso64) vdso64.so.dbg vgettimeofday-64.o vgetrandom-64.o obj-vdso64 := $(addprefix $(obj)/, $(obj-vdso64)) ccflags-y := -fno-common -fno-builtin -DBUILD_VDSO @@ -71,7 +73,7 @@ CPPFLAGS_vdso64.lds += -P -C # link rule for the .so file, .lds has to be first $(obj)/vdso32.so.dbg: $(obj)/vdso32.lds $(obj-vdso32) $(obj)/vgettimeofday-32.o $(obj)/vgetrandom-32.o $(obj)/crtsavres-32.o FORCE $(call if_changed,vdso32ld_and_check) -$(obj)/vdso64.so.dbg: $(obj)/vdso64.lds $(obj-vdso64) $(obj)/vgettimeofday-64.o FORCE +$(obj)/vdso64.so.dbg: $(obj)/vdso64.lds $(obj-vdso64) $(obj)/vgettimeofday-64.o $(obj)/vgetrandom-64.o FORCE $(call if_changed,vdso64ld_and_check) # assembly rules for the .S files @@ -87,6 +89,8 @@ $(obj-vdso64): %-64.o: %.S FORCE $(call if_changed_dep,vdso64as) $(obj)/vgettimeofday-64.o: %-64.o: %.c FORCE $(call if_changed_dep,cc_o_c) +$(obj)/vgetrandom-64.o: %-64.o: %.c FORCE + $(call if_changed_dep,cc_o_c) # Generate VDSO offsets using helper script gen-vdso32sym := $(src)/gen_vdso32_offsets.sh diff --git a/arch/powerpc/kernel/vdso/getrandom.S b/arch/powerpc/kernel/vdso/getrandom.S index 21773ef3fc1d..a957cd2b2b03 100644 --- a/arch/powerpc/kernel/vdso/getrandom.S +++ b/arch/powerpc/kernel/vdso/getrandom.S @@ -27,10 +27,18 @@ .cfi_adjust_cfa_offset PPC_MIN_STKFRM PPC_STL r0, PPC_MIN_STKFRM + PPC_LR_STKOFF(r1) .cfi_rel_offset lr, PPC_MIN_STKFRM + PPC_LR_STKOFF +#ifdef __powerpc64__ + PPC_STL r2, PPC_MIN_STKFRM + STK_GOT(r1) + .cfi_rel_offset r2, PPC_MIN_STKFRM + STK_GOT +#endif get_datapage r8 addi r8, r8, VDSO_RNG_DATA_OFFSET bl CFUNC(DOTSYM(\funct)) PPC_LL r0, PPC_MIN_STKFRM + PPC_LR_STKOFF(r1) +#ifdef __powerpc64__ + PPC_LL r2, PPC_MIN_STKFRM + STK_GOT(r1) + .cfi_restore r2 +#endif cmpwi r3, 0 mtlr r0 addi r1, r1, 2 * PPC_MIN_STKFRM diff --git a/arch/powerpc/kernel/vdso/vdso64.lds.S b/arch/powerpc/kernel/vdso/vdso64.lds.S index 400819258c06..9481e4b892ed 100644 --- a/arch/powerpc/kernel/vdso/vdso64.lds.S +++ b/arch/powerpc/kernel/vdso/vdso64.lds.S @@ -123,6 +123,7 @@ VERSION __kernel_sigtramp_rt64; __kernel_getcpu; __kernel_time; + __kernel_getrandom; local: *; }; diff --git a/arch/powerpc/kernel/vdso/vgetrandom-chacha.S b/arch/powerpc/kernel/vdso/vgetrandom-chacha.S index ac85788205cb..7f9061a9e8b4 100644 --- a/arch/powerpc/kernel/vdso/vgetrandom-chacha.S +++ b/arch/powerpc/kernel/vdso/vgetrandom-chacha.S @@ -124,6 +124,26 @@ */ SYM_FUNC_START(__arch_chacha20_blocks_nostack) #ifdef __powerpc64__ + std counter, -216(r1) + + std r14, -144(r1) + std r15, -136(r1) + std r16, -128(r1) + std r17, -120(r1) + std r18, -112(r1) + std r19, -104(r1) + std r20, -96(r1) + std r21, -88(r1) + std r22, -80(r1) + std r23, -72(r1) + std r24, -64(r1) + std r25, -56(r1) + std r26, -48(r1) + std r27, -40(r1) + std r28, -32(r1) + std r29, -24(r1) + std r30, -16(r1) + std r31, -8(r1) #else stwu r1, -96(r1) stw counter, 20(r1) @@ -149,9 +169,13 @@ SYM_FUNC_START(__arch_chacha20_blocks_nostack) stw r30, 88(r1) stw r31, 92(r1) #endif +#endif /* __powerpc64__ */ lwz counter0, 0(counter) lwz counter1, 4(counter) +#ifdef __powerpc64__ + rldimi counter0, counter1, 32, 0 +#endif mr idx_r0, nblocks subi dst_bytes, dst_bytes, 4 @@ -267,12 +291,21 @@ SYM_FUNC_START(__arch_chacha20_blocks_nostack) subic. idx_r0, idx_r0, 1 /* subi. can't use r0 as source */ +#ifdef __powerpc64__ + addi counter0, counter0, 1 + srdi counter1, counter0, 32 +#else addic counter0, counter0, 1 addze counter1, counter1 +#endif bne .Lblock +#ifdef __powerpc64__ + ld counter, -216(r1) +#else lwz counter, 20(r1) +#endif stw counter0, 0(counter) stw counter1, 4(counter) @@ -284,6 +317,26 @@ SYM_FUNC_START(__arch_chacha20_blocks_nostack) li r11, 0 li r12, 0 +#ifdef __powerpc64__ + ld r14, -144(r1) + ld r15, -136(r1) + ld r16, -128(r1) + ld r17, -120(r1) + ld r18, -112(r1) + ld r19, -104(r1) + ld r20, -96(r1) + ld r21, -88(r1) + ld r22, -80(r1) + ld r23, -72(r1) + ld r24, -64(r1) + ld r25, -56(r1) + ld r26, -48(r1) + ld r27, -40(r1) + ld r28, -32(r1) + ld r29, -24(r1) + ld r30, -16(r1) + ld r31, -8(r1) +#else #ifdef __BIG_ENDIAN__ lmw r14, 24(r1) #else