From patchwork Fri Dec 8 11:32:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13485341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C6B12C4167B for ; Fri, 8 Dec 2023 11:33:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=2Nt64+CTc/+V5X/oClaYUnheiUuncKIY7AvN3tuJi0o=; b=2yqLWLkPQ4MJ+XcyEIbCqJRo4r QpPvq01WzRVgf7ji26WlThLnUefy9uqMv7UFmqoA4dRDNYPXOizpoqbu8vVgdOaVC/A2y/JF980cZ +V359bvtoWhmS6S7riNW16ZxCWMotkHtNu6auQpZvIV81WSZM8k9r0bvQH8VndWe7uhOVYX0l6vcP UD6+xHm3FahPQ+uxVI746h08s0vZZjCRLF/+1DzkoX4R0amd57ydxEWEcciYe8cfEQ5xmofrsj5ON XYGKld2F7Ya/7J2UVMqD6iPCDCv3oKX0iusTi06aN2+hC13mEgvRsiR5QtmCxxz6yWx10TBstcLfc ElPrafdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rBZ6G-00FVh1-2s; Fri, 08 Dec 2023 11:32:56 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rBZ6B-00FVer-1h for linux-arm-kernel@lists.infradead.org; Fri, 08 Dec 2023 11:32:53 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5d42c43d8daso12752557b3.0 for ; Fri, 08 Dec 2023 03:32:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1702035170; x=1702639970; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tIW6OxkQLD6ojtSOM3oZEkZa7hR7XnG6NZiYsQLCSik=; b=Kgc80n+t/ENSeWQDt38KQiXPds3e4UuMZiqe/rlzJT6/rPK8CY4j8hNsyUiXmdZUEp MQzs3d+nhNYnxLFcaEJ1M3XQpWiS0MNQlVO/6TxrCwZIb4GR+I6dWhVZYBMrY+yot7z4 8sv5Zy3zreeb/yHsmNdgzRd7PI3j0AMxaXv8hnSRpCLPaXQ4dbP8kl1kmQR815U/xwGF F3vDk4ViOC88SROOh9LB8lr8tTzE4sAiFSRiu/iXuKlc8AbAnBFci6swUJ79zymc8koZ neV4IMgyHx9/OZUVosNfHGr7N/5y2/PrwffnMsx5aPKFogI4jIeuX+Jb5wYh9HJndmdN 6njQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702035170; x=1702639970; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tIW6OxkQLD6ojtSOM3oZEkZa7hR7XnG6NZiYsQLCSik=; b=XbSxJWdXewOCdR6K+O5HJXbFkSziKeXLTvwyBbBxP7bHGI5PFHwHc9SrYQlUyr9qql axyyDCEHJLsPDGg2+Vkmx5UOgxOlVqGmAjE9rRF/jYsujBiA0RdDiVm99HFamUy7ltGN Olt5mTLC/Wlm25FER7NtdMywfn+r83S4xCO3u0hxQ9jaetgnCOirjMyKXffmcLatUXLO zXZjzxYvjcOANCiXYTLVutGflE8ECbDkr6NxPFLC0k2QhFWPp6zRymGRBHd0RMExZe3m yOxsHgNlxD7+5eE8Dap9ud8qprWTXeTAwSClyLSbBIWYyWrK2nDa+DMjX7GFOIxNPYPj +w9g== X-Gm-Message-State: AOJu0YzLUelT7ddLfiv/LpyWlkVLmxfUmLjLXoXn8UmdQs9EBLqZStwZ hhN/arCe6MuvzqG6DPjUJ+wwm9yCVEBC5BbyLXrO2bj/j/79dnQt2elEoj9OE46YzGmy81wJ4le NnXWzK8b4JiAOhjtZDo03aStOUiieOxbhcDN83eh6OeOx2c3ad8CPph9Jq3NKiaqy0vZVfaFBdu 0= X-Google-Smtp-Source: AGHT+IGmi88gIiTU25m/DqJSHEAZeeNr73MSkwK61S+3OU/oHCtmTGtgJXqoqfhL2wBZUn5l7SaTiMG7 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:fe09:0:b0:5d3:d44a:578d with SMTP id j9-20020a81fe09000000b005d3d44a578dmr11434ywn.4.1702035169745; Fri, 08 Dec 2023 03:32:49 -0800 (PST) Date: Fri, 8 Dec 2023 12:32:23 +0100 In-Reply-To: <20231208113218.3001940-6-ardb@google.com> Mime-Version: 1.0 References: <20231208113218.3001940-6-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6423; i=ardb@kernel.org; h=from:subject; bh=CrHTaHJltEK6PeCTI4LoBKpJ8PhFKRqD4hhGSJUvD44=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIbXo37Ft8nUvf9254Rmw993GMzGn3L8e/NO2JereI4mHm 6oSVCKSO0pZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEklkY/medMDjxp/i3kaqn vNqdY+7F9XPTTzA+N1x0Y8qkfQIPOjoYGfY7zdJgzWI+tuI8ywOudlmPaqG9V7KDbz+pCk5bzRP AyQAA X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208113218.3001940-10-ardb@google.com> Subject: [PATCH v4 4/4] arm64: crypto: Disable yielding logic unless CONFIG_PREEMPT_VOLUNTARY=y From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Eric Biggers , Sebastian Andrzej Siewior X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231208_033251_566139_B4F73AB2 X-CRM114-Status: GOOD ( 17.01 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Now that kernel mode use of SIMD runs with preemption enabled, the explicit yield logic is redundant for preemptible builds, and since it should not actually be used at all on non-preemptible builds (where kernel work is supposed to run to completion and not give up its time slice prematurely), let's make it depend on CONFIG_PREEMPT_VOLUNTARY. Once CONFIG_PREEMPT_VOLUNTARY is removed, all the logic it guards can be removed as well. Signed-off-by: Ard Biesheuvel --- arch/arm64/crypto/aes-ce-ccm-glue.c | 8 ++++++-- arch/arm64/crypto/chacha-neon-glue.c | 5 ++++- arch/arm64/crypto/crct10dif-ce-glue.c | 6 ++++-- arch/arm64/crypto/nhpoly1305-neon-glue.c | 5 ++++- arch/arm64/crypto/poly1305-glue.c | 5 ++++- arch/arm64/crypto/polyval-ce-glue.c | 9 +++++++-- arch/arm64/include/asm/assembler.h | 4 ++-- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index 25cd3808ecbe..82e293a698ff 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -125,13 +125,17 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) scatterwalk_start(&walk, sg_next(walk.sg)); n = scatterwalk_clamp(&walk, len); } - n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */ + + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) + n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */ + p = scatterwalk_map(&walk); macp = ce_aes_ccm_auth_data(mac, p, n, macp, ctx->key_enc, num_rounds(ctx)); - if (len / SZ_4K > (len - n) / SZ_4K) { + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) && + len / SZ_4K > (len - n) / SZ_4K) { kernel_neon_end(); kernel_neon_begin(); } diff --git a/arch/arm64/crypto/chacha-neon-glue.c b/arch/arm64/crypto/chacha-neon-glue.c index af2bbca38e70..655b250cef4a 100644 --- a/arch/arm64/crypto/chacha-neon-glue.c +++ b/arch/arm64/crypto/chacha-neon-glue.c @@ -88,7 +88,10 @@ void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes, return chacha_crypt_generic(state, dst, src, bytes, nrounds); do { - unsigned int todo = min_t(unsigned int, bytes, SZ_4K); + unsigned int todo = bytes; + + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) + todo = min_t(unsigned int, todo, SZ_4K); kernel_neon_begin(); chacha_doneon(state, dst, src, todo, nrounds); diff --git a/arch/arm64/crypto/crct10dif-ce-glue.c b/arch/arm64/crypto/crct10dif-ce-glue.c index 09eb1456aed4..c6e8cf4f56da 100644 --- a/arch/arm64/crypto/crct10dif-ce-glue.c +++ b/arch/arm64/crypto/crct10dif-ce-glue.c @@ -40,7 +40,8 @@ static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data, do { unsigned int chunk = length; - if (chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE) + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) && + chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE) chunk = SZ_4K; kernel_neon_begin(); @@ -65,7 +66,8 @@ static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data, do { unsigned int chunk = length; - if (chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE) + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY) && + chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE) chunk = SZ_4K; kernel_neon_begin(); diff --git a/arch/arm64/crypto/nhpoly1305-neon-glue.c b/arch/arm64/crypto/nhpoly1305-neon-glue.c index e4a0b463f080..cbbc51b27d93 100644 --- a/arch/arm64/crypto/nhpoly1305-neon-glue.c +++ b/arch/arm64/crypto/nhpoly1305-neon-glue.c @@ -23,7 +23,10 @@ static int nhpoly1305_neon_update(struct shash_desc *desc, return crypto_nhpoly1305_update(desc, src, srclen); do { - unsigned int n = min_t(unsigned int, srclen, SZ_4K); + unsigned int n = srclen; + + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) + n = min_t(unsigned int, n, SZ_4K); kernel_neon_begin(); crypto_nhpoly1305_update_helper(desc, src, n, nh_neon); diff --git a/arch/arm64/crypto/poly1305-glue.c b/arch/arm64/crypto/poly1305-glue.c index 1fae18ba11ed..27f84f5bfc98 100644 --- a/arch/arm64/crypto/poly1305-glue.c +++ b/arch/arm64/crypto/poly1305-glue.c @@ -144,7 +144,10 @@ void poly1305_update_arch(struct poly1305_desc_ctx *dctx, const u8 *src, if (static_branch_likely(&have_neon) && crypto_simd_usable()) { do { - unsigned int todo = min_t(unsigned int, len, SZ_4K); + unsigned int todo = len; + + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) + todo = min_t(unsigned int, todo, SZ_4K); kernel_neon_begin(); poly1305_blocks_neon(&dctx->h, src, todo, 1); diff --git a/arch/arm64/crypto/polyval-ce-glue.c b/arch/arm64/crypto/polyval-ce-glue.c index 0a3b5718df85..c4c0fb3fcaf4 100644 --- a/arch/arm64/crypto/polyval-ce-glue.c +++ b/arch/arm64/crypto/polyval-ce-glue.c @@ -123,8 +123,13 @@ static int polyval_arm64_update(struct shash_desc *desc, } while (srclen >= POLYVAL_BLOCK_SIZE) { - /* allow rescheduling every 4K bytes */ - nblocks = min(srclen, 4096U) / POLYVAL_BLOCK_SIZE; + unsigned int len = srclen; + + if (IS_ENABLED(CONFIG_PREEMPT_VOLUNTARY)) + /* allow rescheduling every 4K bytes */ + len = min(len, 4096U); + + nblocks = len / POLYVAL_BLOCK_SIZE; internal_polyval_update(tctx, src, nblocks, dctx->buffer); srclen -= nblocks * POLYVAL_BLOCK_SIZE; src += nblocks * POLYVAL_BLOCK_SIZE; diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 376a980f2bad..0180ac1f9b8b 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -769,6 +769,7 @@ alternative_endif * field) */ .macro cond_yield, lbl:req, tmp:req, tmp2:req +#ifdef CONFIG_PREEMPT_VOLUNTARY get_current_task \tmp ldr \tmp, [\tmp, #TSK_TI_PREEMPT] /* @@ -777,15 +778,14 @@ alternative_endif * run to completion as quickly as we can. */ tbnz \tmp, #SOFTIRQ_SHIFT, .Lnoyield_\@ -#ifdef CONFIG_PREEMPTION sub \tmp, \tmp, #PREEMPT_DISABLE_OFFSET cbz \tmp, \lbl -#endif adr_l \tmp, irq_stat + IRQ_CPUSTAT_SOFTIRQ_PENDING get_this_cpu_offset \tmp2 ldr w\tmp, [\tmp, \tmp2] cbnz w\tmp, \lbl // yield on pending softirq in task context .Lnoyield_\@: +#endif .endm /*