From patchwork Mon Dec 30 00:13:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922906 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BAB125948E; Mon, 30 Dec 2024 00:15:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; cv=none; b=KXju8bVI88dJlm7Z/R/vso5mMSs9ouQ3bViYfvtCRO29kD4QgN26eefl0nnCFwidbvbEy0W9O5x+h2cECNqBsQGLQPJylZQ84bPPA6n0vb5AvbQxKTruB1JoebSemX4O2l6X7upWu+RD7q60TTDQk7ihvytHzC2P/VKVRByXTZE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; c=relaxed/simple; bh=5+ZF0ImlbQo//JHYBtR4hUhu8cd/k3zSglYQL5MyI1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y8objp7o1ZEbd95siolE0Lvl3TcdnOu08oINwSrlU9ARNvYogBJbs1ZCPDLs+v+vIgoLufaYTO3nk2sSscmURYjIYg1WdBg8pTbfqbtQTWLtm0UCRrHrYmaHnMv3FLBXUrqvKaHTj3ClhS4q/55vFFWAZgD3vObjnNX7qds87wc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CajwGQWs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CajwGQWs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 51452C4CED7; Mon, 30 Dec 2024 00:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517759; bh=5+ZF0ImlbQo//JHYBtR4hUhu8cd/k3zSglYQL5MyI1I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CajwGQWsxx05ZrJZe2LCuWC1feYaE6hpCpmCosLC9+nd9cABcywaDkOtw+FfJFDKs EDUrpQO5btCgpYAo8cUbvLnpnqDK4JqDQgJ7hY3GDsMPTk5vxUihNLbrBCdyWDnwL4 KO+BY2iZg0D47vV+bSog6PydDkilzlIgdam8Y5vXE89LEAWTVTX60CYCVa6pxl78n9 eE/XkAI0rC8035n/rPCyKI+QP5mvI0p3H/O/TAWH2+4MXZ7itSFE863g2FHGVGjQVR xwQ4wzHr9WHwMfsG52Wbfm7b/HkqYS+aazwxFrXj4uI6STp5hPNRqxRZeLwtO6e4df HBEmF/JgoUpKQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 01/29] crypto: skcipher - document skcipher_walk_done() and rename some vars Date: Sun, 29 Dec 2024 16:13:50 -0800 Message-ID: <20241230001418.74739-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers skcipher_walk_done() has an unusual calling convention, and some of its local variables have unclear names. Document it and rename variables to make it a bit clearer what is going on. No change in behavior. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 50 ++++++++++++++++++++---------- include/crypto/internal/skcipher.h | 2 +- 2 files changed, 35 insertions(+), 17 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index d5fe0eca3826..8749c44f98a2 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -87,21 +87,39 @@ static int skcipher_done_slow(struct skcipher_walk *walk, unsigned int bsize) addr = skcipher_get_spot(addr, bsize); scatterwalk_copychunks(addr, &walk->out, bsize, 1); return 0; } -int skcipher_walk_done(struct skcipher_walk *walk, int err) +/** + * skcipher_walk_done() - finish one step of a skcipher_walk + * @walk: the skcipher_walk + * @res: number of bytes *not* processed (>= 0) from walk->nbytes, + * or a -errno value to terminate the walk due to an error + * + * This function cleans up after one step of walking through the source and + * destination scatterlists, and advances to the next step if applicable. + * walk->nbytes is set to the number of bytes available in the next step, + * walk->total is set to the new total number of bytes remaining, and + * walk->{src,dst}.virt.addr is set to the next pair of data pointers. If there + * is no more data, or if an error occurred (i.e. -errno return), then + * walk->nbytes and walk->total are set to 0 and all resources owned by the + * skcipher_walk are freed. + * + * Return: 0 or a -errno value. If @res was a -errno value then it will be + * returned, but other errors may occur too. + */ +int skcipher_walk_done(struct skcipher_walk *walk, int res) { - unsigned int n = walk->nbytes; - unsigned int nbytes = 0; + unsigned int n = walk->nbytes; /* num bytes processed this step */ + unsigned int total = 0; /* new total remaining */ if (!n) goto finish; - if (likely(err >= 0)) { - n -= err; - nbytes = walk->total - n; + if (likely(res >= 0)) { + n -= res; /* subtract num bytes *not* processed */ + total = walk->total - n; } if (likely(!(walk->flags & (SKCIPHER_WALK_SLOW | SKCIPHER_WALK_COPY | SKCIPHER_WALK_DIFF)))) { @@ -113,35 +131,35 @@ int skcipher_walk_done(struct skcipher_walk *walk, int err) } else if (walk->flags & SKCIPHER_WALK_COPY) { skcipher_map_dst(walk); memcpy(walk->dst.virt.addr, walk->page, n); skcipher_unmap_dst(walk); } else if (unlikely(walk->flags & SKCIPHER_WALK_SLOW)) { - if (err > 0) { + if (res > 0) { /* * Didn't process all bytes. Either the algorithm is * broken, or this was the last step and it turned out * the message wasn't evenly divisible into blocks but * the algorithm requires it. */ - err = -EINVAL; - nbytes = 0; + res = -EINVAL; + total = 0; } else n = skcipher_done_slow(walk, n); } - if (err > 0) - err = 0; + if (res > 0) + res = 0; - walk->total = nbytes; + walk->total = total; walk->nbytes = 0; scatterwalk_advance(&walk->in, n); scatterwalk_advance(&walk->out, n); - scatterwalk_done(&walk->in, 0, nbytes); - scatterwalk_done(&walk->out, 1, nbytes); + scatterwalk_done(&walk->in, 0, total); + scatterwalk_done(&walk->out, 1, total); - if (nbytes) { + if (total) { crypto_yield(walk->flags & SKCIPHER_WALK_SLEEP ? CRYPTO_TFM_REQ_MAY_SLEEP : 0); return skcipher_walk_next(walk); } @@ -156,11 +174,11 @@ int skcipher_walk_done(struct skcipher_walk *walk, int err) kfree(walk->buffer); if (walk->page) free_page((unsigned long)walk->page); out: - return err; + return res; } EXPORT_SYMBOL_GPL(skcipher_walk_done); static int skcipher_next_slow(struct skcipher_walk *walk, unsigned int bsize) { diff --git a/include/crypto/internal/skcipher.h b/include/crypto/internal/skcipher.h index 08d1e8c63afc..4f49621d3eb6 100644 --- a/include/crypto/internal/skcipher.h +++ b/include/crypto/internal/skcipher.h @@ -194,11 +194,11 @@ void crypto_unregister_lskcipher(struct lskcipher_alg *alg); int crypto_register_lskciphers(struct lskcipher_alg *algs, int count); void crypto_unregister_lskciphers(struct lskcipher_alg *algs, int count); int lskcipher_register_instance(struct crypto_template *tmpl, struct lskcipher_instance *inst); -int skcipher_walk_done(struct skcipher_walk *walk, int err); +int skcipher_walk_done(struct skcipher_walk *walk, int res); int skcipher_walk_virt(struct skcipher_walk *walk, struct skcipher_request *req, bool atomic); int skcipher_walk_aead_encrypt(struct skcipher_walk *walk, struct aead_request *req, bool atomic); From patchwork Mon Dec 30 00:13:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922905 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BA86259481; Mon, 30 Dec 2024 00:15:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; cv=none; b=r4Uwxtc4AqKgA8PG681+/fL3bGpKI0Q8bxl/QVtcRlOdqAAR934yjYEXC0fe75u/WgAabIixz+KIoPunIno8J4rYmpF6rD+RGzDNX1Dunq91eksbceuEiKedT/q3+obZe5ygIVRDTgrJ1CwTZARTQCdKLz4YG+LPi+Zipsosjs0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; c=relaxed/simple; bh=J5FzviwCD+n8p+5rLyvIFWAP97F/jKX49Nuk3AnfzpU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CqDufiVMlPtNtSz2B/9DMIAUET0RAvEywqMrYiqU0kr79KwMFWRdagk/ckj7LxxNupXA+urray4fiL2V7uij+kA7zQfqC28m5J0CtZHHTV559MkeYWNfSr9PNS8qB7QMnTl1DU+hnkUSFNGpcyP22jr9wRFiGI26kQfun/aLVPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=n/HpaXI5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="n/HpaXI5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 97E2CC4CED4; Mon, 30 Dec 2024 00:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517759; bh=J5FzviwCD+n8p+5rLyvIFWAP97F/jKX49Nuk3AnfzpU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n/HpaXI5IGM2bT2b3rlNIJtS96jx+szt27vQd/3l1Vm3/4Nm85/XQsNY1RDbdtv8z 9M4alsvzIy89ecSly9DTTsgc+2crthiG1KsIEKJl3+/XZoOLw3FK8toGbXFvT2DSh8 /7uoqlCaw39PKwt50yEzep3KC3LomwCc8+61kwRL2aO+a0DURUTvu8gSXC2LWB/KjR S+AIMSzVI9c/1kuTPXb3rzlMoIqRx17BwGfg3rwdi4pfD1gJBtAxBYOk4lrTce53um 98580FImzduySG8GJypA6eciUewgJPi5vuaKvaCx/7OPKulnzYSQKvCkXCBAYG7C/H kdcVv6lMjrX8A== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 02/29] crypto: skcipher - remove unnecessary page alignment of bounce buffer Date: Sun, 29 Dec 2024 16:13:51 -0800 Message-ID: <20241230001418.74739-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In the slow path of skcipher_walk where it uses a slab bounce buffer for the data and/or IV, do not bother to avoid crossing a page boundary in the part(s) of this buffer that are used, and do not bother to allocate extra space in the buffer for that purpose. The buffer is accessed only by virtual address, so pages are irrelevant for it. This logic may have been present due to the physical address support in skcipher_walk, but that has now been removed. Or it may have been present to be consistent with the fast path that currently does not hand back addresses that span pages, but that behavior is a side effect of the pages being "mapped" one by one and is not actually a requirement. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 62 ++++++++++++----------------------------------- 1 file changed, 15 insertions(+), 47 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 8749c44f98a2..887cbce8f78d 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -61,32 +61,20 @@ static inline void skcipher_unmap_dst(struct skcipher_walk *walk) static inline gfp_t skcipher_walk_gfp(struct skcipher_walk *walk) { return walk->flags & SKCIPHER_WALK_SLEEP ? GFP_KERNEL : GFP_ATOMIC; } -/* Get a spot of the specified length that does not straddle a page. - * The caller needs to ensure that there is enough space for this operation. - */ -static inline u8 *skcipher_get_spot(u8 *start, unsigned int len) -{ - u8 *end_page = (u8 *)(((unsigned long)(start + len - 1)) & PAGE_MASK); - - return max(start, end_page); -} - static inline struct skcipher_alg *__crypto_skcipher_alg( struct crypto_alg *alg) { return container_of(alg, struct skcipher_alg, base); } static int skcipher_done_slow(struct skcipher_walk *walk, unsigned int bsize) { - u8 *addr; + u8 *addr = PTR_ALIGN(walk->buffer, walk->alignmask + 1); - addr = (u8 *)ALIGN((unsigned long)walk->buffer, walk->alignmask + 1); - addr = skcipher_get_spot(addr, bsize); scatterwalk_copychunks(addr, &walk->out, bsize, 1); return 0; } /** @@ -181,37 +169,26 @@ int skcipher_walk_done(struct skcipher_walk *walk, int res) EXPORT_SYMBOL_GPL(skcipher_walk_done); static int skcipher_next_slow(struct skcipher_walk *walk, unsigned int bsize) { unsigned alignmask = walk->alignmask; - unsigned a; unsigned n; u8 *buffer; if (!walk->buffer) walk->buffer = walk->page; buffer = walk->buffer; - if (buffer) - goto ok; - - /* Start with the minimum alignment of kmalloc. */ - a = crypto_tfm_ctx_alignment() - 1; - n = bsize; - - /* Minimum size to align buffer by alignmask. */ - n += alignmask & ~a; - - /* Minimum size to ensure buffer does not straddle a page. */ - n += (bsize - 1) & ~(alignmask | a); - - buffer = kzalloc(n, skcipher_walk_gfp(walk)); - if (!buffer) - return skcipher_walk_done(walk, -ENOMEM); - walk->buffer = buffer; -ok: + if (!buffer) { + /* Min size for a buffer of bsize bytes aligned to alignmask */ + n = bsize + (alignmask & ~(crypto_tfm_ctx_alignment() - 1)); + + buffer = kzalloc(n, skcipher_walk_gfp(walk)); + if (!buffer) + return skcipher_walk_done(walk, -ENOMEM); + walk->buffer = buffer; + } walk->dst.virt.addr = PTR_ALIGN(buffer, alignmask + 1); - walk->dst.virt.addr = skcipher_get_spot(walk->dst.virt.addr, bsize); walk->src.virt.addr = walk->dst.virt.addr; scatterwalk_copychunks(walk->src.virt.addr, &walk->in, bsize, 0); walk->nbytes = bsize; @@ -294,34 +271,25 @@ static int skcipher_walk_next(struct skcipher_walk *walk) return skcipher_next_fast(walk); } static int skcipher_copy_iv(struct skcipher_walk *walk) { - unsigned a = crypto_tfm_ctx_alignment() - 1; unsigned alignmask = walk->alignmask; unsigned ivsize = walk->ivsize; - unsigned bs = walk->stride; - unsigned aligned_bs; + unsigned aligned_stride = ALIGN(walk->stride, alignmask + 1); unsigned size; u8 *iv; - aligned_bs = ALIGN(bs, alignmask + 1); - - /* Minimum size to align buffer by alignmask. */ - size = alignmask & ~a; - - size += aligned_bs + ivsize; - - /* Minimum size to ensure buffer does not straddle a page. */ - size += (bs - 1) & ~(alignmask | a); + /* Min size for a buffer of stride + ivsize, aligned to alignmask */ + size = aligned_stride + ivsize + + (alignmask & ~(crypto_tfm_ctx_alignment() - 1)); walk->buffer = kmalloc(size, skcipher_walk_gfp(walk)); if (!walk->buffer) return -ENOMEM; - iv = PTR_ALIGN(walk->buffer, alignmask + 1); - iv = skcipher_get_spot(iv, bs) + aligned_bs; + iv = PTR_ALIGN(walk->buffer, alignmask + 1) + aligned_stride; walk->iv = memcpy(iv, walk->iv, walk->ivsize); return 0; } From patchwork Mon Dec 30 00:13:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922908 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EAC91BDDF; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; cv=none; b=Vs5KURalkVpe3WibTQVXds7ALK7onBfhSE5vxIHRuFKGf8uX4k0DMD4sam7SisEy/rJoDgxC6Jkfe2Gzg6pcVYETBzuB7zzKD6mrzNGxtTO37JTXcBFVHpNUGUrrqYd8xmC6TV4oZqNY+q5zBl5+T/wrL7EhpDg4ohFu8Bypm2o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; c=relaxed/simple; bh=l9HLRzTyaTzJjaI3UCqrNcL/jvjXdUpirL97KFLMmiQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LF3efJu+bOxsZyHDVNpkBeHv9kSvoJE5KOO/qwcCXC03QMA7qNdqUxqHttSx1P/R1xcRbMAI5UR0aS5LFV9f6nRCCUV8wzZHOrebaJh7IdzFjzII6ubD7RkRODUqUwi5cg3sFq0dSPFjDbFXHyqXKQ5Yn6emeVakGyalGTOG9SI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=B7Rk1Q3U; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="B7Rk1Q3U" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D84A3C4CEDC; Mon, 30 Dec 2024 00:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517760; bh=l9HLRzTyaTzJjaI3UCqrNcL/jvjXdUpirL97KFLMmiQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B7Rk1Q3U5IxG2wqwIkpH2UiBhk8uFIf4ZAU3VhFPR+q0GcuAx94/MQ0Op5pc5p9Ze DZeWemTcHgeheZrIxk5238wUwh47YKIQ6rbqIaReCxsztChlx5ZGcCGcr70VERevcS +kJn0AYqY1e9SFFyiDW0VN76GVxb62YnWCchb7QzbFd87b9NtzkWZpNzbEZPVp9NqE zfenIVT6st7zZDKQLqDXbZaGGhzfibUbYTojkOinu301+3mLNqYyjPqXrxDGtE06g+ yIKkoY7CwxKp8pjvoH8J9CdUWci6S2VCTfrOZDhu+gWkmRqIw69MJ1TkgWRm7cDfYV U8c/ViwzEK+tg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 03/29] crypto: skcipher - remove redundant clamping to page size Date: Sun, 29 Dec 2024 16:13:52 -0800 Message-ID: <20241230001418.74739-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In the case where skcipher_walk_next() allocates a bounce page, that page by definition has size PAGE_SIZE. The number of bytes to copy 'n' is guaranteed to fit in it, since earlier in the function it was clamped to be at most a page. Therefore remove the unnecessary logic that tried to clamp 'n' again to fit in the bounce page. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 887cbce8f78d..c627e267b125 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -248,28 +248,24 @@ static int skcipher_walk_next(struct skcipher_walk *walk) return skcipher_walk_done(walk, -EINVAL); slow_path: return skcipher_next_slow(walk, bsize); } + walk->nbytes = n; if (unlikely((walk->in.offset | walk->out.offset) & walk->alignmask)) { if (!walk->page) { gfp_t gfp = skcipher_walk_gfp(walk); walk->page = (void *)__get_free_page(gfp); if (!walk->page) goto slow_path; } - - walk->nbytes = min_t(unsigned, n, - PAGE_SIZE - offset_in_page(walk->page)); walk->flags |= SKCIPHER_WALK_COPY; return skcipher_next_copy(walk); } - walk->nbytes = n; - return skcipher_next_fast(walk); } static int skcipher_copy_iv(struct skcipher_walk *walk) { From patchwork Mon Dec 30 00:13:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922907 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EA9518EB0; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; cv=none; b=i7PZaUsd6EV60wDV6kUX7+rTBty/myUFvlGAslSxvG3hJ+xIF5MGdygMRhT0sCLm5Il2SkESQsPR38wLD2PvYG6iiagMRJ4xJSrEIbmOtQqIR46qKMAaJbcxnOc3wmfjjSCBi9uESY/JDIknToZajDm7adA/Q4oBBZHv9bwfqEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; c=relaxed/simple; bh=7uVm3CYoqa7+cTgqCe8InwCLob5UXrzsedA8gOn7b/Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UMzVNbetNR8/FuUolU4Q70VK6Vgboi3LKgZ4eABWswtSO2BiIFbqqQChcpdT9lYtm37AIE61ML90LVwnSdzJPo1vBIVP76eJNpPx1O4oqH4srSFlpWw2N1cI6IblKGgriOKIJpYvcEWbw886s+meMWEQPf8wCcVGipkc+Ht2mqI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=emALD3YB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="emALD3YB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26C31C4CEDD; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517760; bh=7uVm3CYoqa7+cTgqCe8InwCLob5UXrzsedA8gOn7b/Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=emALD3YB1G99ePmpruIuIykcrGHRoLeNnePWIthGG5XfZMHencEMweAYsuv1Eb/VY kRX+xXxkrhmAtDE6+HzyZdxDsSvhCg/Og3egnfeQtOihHODmxJap8ay0N9EOfJl+xK X6v5gi3tYb40L6uwfT64na66vsa0fODtzsVadrNaWJGzEU2w1S4yL8CY3wVAgv5UGN gb0AiJ3NTeZ5XHOzkhca9bkcZ12dgVEQY621eSd9YUpIO8Gx30Dfih6iFNsbATaU56 JblOuvnIG2ppcNxlLnQLzi+V12CTkxJKNBpHDnoqkgvrFW/lURjjc5JPi+l9OjzEuB 7krS+qPes5mCA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 04/29] crypto: skcipher - remove redundant check for SKCIPHER_WALK_SLOW Date: Sun, 29 Dec 2024 16:13:53 -0800 Message-ID: <20241230001418.74739-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In skcipher_walk_done(), remove the check for SKCIPHER_WALK_SLOW because it is always true. All other flags (and lack thereof) were checked earlier in the function, leaving SKCIPHER_WALK_SLOW as the only remaining possibility. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index c627e267b125..98606def1bf9 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -118,11 +118,11 @@ int skcipher_walk_done(struct skcipher_walk *walk, int res) goto unmap_src; } else if (walk->flags & SKCIPHER_WALK_COPY) { skcipher_map_dst(walk); memcpy(walk->dst.virt.addr, walk->page, n); skcipher_unmap_dst(walk); - } else if (unlikely(walk->flags & SKCIPHER_WALK_SLOW)) { + } else { /* SKCIPHER_WALK_SLOW */ if (res > 0) { /* * Didn't process all bytes. Either the algorithm is * broken, or this was the last step and it turned out * the message wasn't evenly divisible into blocks but From patchwork Mon Dec 30 00:13:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922909 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE9F71C6BE; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; cv=none; b=h8EPMTux2OhWAfkjIR3ug0S/7SxIOwKVadQ8MJqre90Tk8VmDDu5UKz7KkYkuxB1M8+UgmZ103DySWT6yU0vf74RxHBDIWKGZ26Q7Q2/grBBj5F19b3B2G0+3viUkaASKL5wQDXLxsMjaDyh+6Zbd1Ci55UX1mC4aKdCo98pRTg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517760; c=relaxed/simple; bh=+GPayxh5RqGj7KlYP7lL3+3fPmB0J8TTLnx1xFmFjIM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TRXEYUNT59oAN9wwc/sjiYBXjnNHtnceRsVAbpATaET/j2hGGleDFPhesz/PiHOuDQhScCPRGpQBvJp0I64jp+MPev46cJB33xWEfLSnTw02SR5pu+1qPEHD6NlJaqF8Eejiw2ByMksglh09ll0s9zx7uZ8oiK0NbRAYsvFQayw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dZNsQSLB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dZNsQSLB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6B4C8C4AF09; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517760; bh=+GPayxh5RqGj7KlYP7lL3+3fPmB0J8TTLnx1xFmFjIM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dZNsQSLBEhtVtXsdQNpvzY+pSL2sa2YCHygrnL583zTqrCQ6hNp7S7db+DOeNoTGW QE+9H4IHjMVIFjsTUO0xTHpUU8ycs8XKQQiW36cnClsoOVNw4YMYmJUN3LHwAcGzjO NpwpWVxnRZzoe5H4JE1rZ2gjMjeGVGdl0oaZde8R/nfvfHip+clVSK7Hcjkbtfiy6H eaY25PT/Li0So3MOPPII+/V5RbTVoGEadBXOM5tsjSsKXI/eclSk7EItGGSW0PK9Al NgZszWkFaMu4ecH7gmTJfJlkWvGRmsof5OkG/8hsRCThwkQsG47WIV9/9w/xKCNg1H uwaMg10ij62cA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 05/29] crypto: skcipher - fold skcipher_walk_skcipher() into skcipher_walk_virt() Date: Sun, 29 Dec 2024 16:13:54 -0800 Message-ID: <20241230001418.74739-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Fold skcipher_walk_skcipher() into skcipher_walk_virt() which is its only remaining caller. No change in behavior. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 23 ++++++++--------------- 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 98606def1bf9..17f4bc79ca8b 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -304,23 +304,26 @@ static int skcipher_walk_first(struct skcipher_walk *walk) walk->page = NULL; return skcipher_walk_next(walk); } -static int skcipher_walk_skcipher(struct skcipher_walk *walk, - struct skcipher_request *req) +int skcipher_walk_virt(struct skcipher_walk *walk, + struct skcipher_request *req, bool atomic) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct skcipher_alg *alg = crypto_skcipher_alg(tfm); + int err = 0; + + might_sleep_if(req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP); walk->total = req->cryptlen; walk->nbytes = 0; walk->iv = req->iv; walk->oiv = req->iv; if (unlikely(!walk->total)) - return 0; + goto out; scatterwalk_start(&walk->in, req->src); scatterwalk_start(&walk->out, req->dst); walk->flags &= ~SKCIPHER_WALK_SLEEP; @@ -334,22 +337,12 @@ static int skcipher_walk_skcipher(struct skcipher_walk *walk, if (alg->co.base.cra_type != &crypto_skcipher_type) walk->stride = alg->co.chunksize; else walk->stride = alg->walksize; - return skcipher_walk_first(walk); -} - -int skcipher_walk_virt(struct skcipher_walk *walk, - struct skcipher_request *req, bool atomic) -{ - int err; - - might_sleep_if(req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP); - - err = skcipher_walk_skcipher(walk, req); - + err = skcipher_walk_first(walk); +out: walk->flags &= atomic ? ~SKCIPHER_WALK_SLEEP : ~0; return err; } EXPORT_SYMBOL_GPL(skcipher_walk_virt); From patchwork Mon Dec 30 00:13:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922912 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45B1570828; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517761; cv=none; b=EgJlmFcJ51cyrat0ScIR4842Fv3eO+T5PeBSDEYjOBzfubcchTzj1V7ERBe0ZFtG4US6qjOv9s4j6GJ16x8ZxQGjP9vJHjoQ5Encs6f/yKlB5ElAOYyBAoKb3+U6rFEKW58qT/8C7C0bIk3ZJ4TvlV3VfJECmYPKw9W7fs/e3bM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517761; c=relaxed/simple; bh=VsaYFM7nf9+iC1YnzN8b1pGbhUrUR9GOM+Yzo4COmtY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y+poRGU6G/UD9MAW/a9QD0VwV+KMLFhjgRw01xuxCVdmaL3qQabU+tzcAXeYqz0WmqcNBhQe02PdbnbNhjp2FYEXr9yflkSeE7WWGwUTTtgohirnUzQZBQUNuPdOaVFR4KcnHSxC18ALUvPFXGlKvONo7sETEmQn6KBwyNS+LgI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=INxUhhS5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="INxUhhS5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id ADD05C4CEE1; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517760; bh=VsaYFM7nf9+iC1YnzN8b1pGbhUrUR9GOM+Yzo4COmtY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=INxUhhS5RJfwKM17mEByCo+UysJ62eZJvQ7A1sHZUlBevaCFcsOEi8wvAKx1/fQr6 oYyRR+8ij+ngOvU7DcTMo2qX/9K6gtFHGnagnQsfuOjnxHr/y2WlZi45Hu6XscO6gJ pNuUHLLY+Za8MFrMibOvddFEJGZWdIoqimeky0aKAr6Gn1vaeHk3rZlY4rBAgRQ+ff xGkn2nNm1iRADKTLjxTXIG2g9aBqFENu+1tp2fFO+Ls77nt4FOBND4yNstWYMU59tF A7DK8DXqpgAo7NA1JF27rmcC3iPtCwkJ3yhLrwN0GlDriIdMH0uKtgPT+Tz6WamOVL M4jjsRUWzl16w== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 06/29] crypto: skcipher - clean up initialization of skcipher_walk::flags Date: Sun, 29 Dec 2024 16:13:55 -0800 Message-ID: <20241230001418.74739-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers - Initialize SKCIPHER_WALK_SLEEP in a consistent way, and check for atomic=true at the same time as CRYPTO_TFM_REQ_MAY_SLEEP. Technically atomic=true only needs to apply after the first step, but it is very rarely used. We should optimize for the common case. So, check 'atomic' alongside CRYPTO_TFM_REQ_MAY_SLEEP. This is more efficient. - Initialize flags other than SKCIPHER_WALK_SLEEP to 0 rather than preserving them. No caller actually initializes the flags, which makes it impossible to use their original values for anything. Indeed, that does not happen and all meaningful flags get overridden anyway. It may have been thought that just clearing one flag would be faster than clearing all flags, but that's not the case as the former is a read-write operation whereas the latter is just a write. - Move the explicit clearing of SKCIPHER_WALK_SLOW, SKCIPHER_WALK_COPY, and SKCIPHER_WALK_DIFF into skcipher_walk_done(), since it is now only needed on non-first steps. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 39 +++++++++++++-------------------------- 1 file changed, 13 insertions(+), 26 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 17f4bc79ca8b..e54d1ad46566 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -146,10 +146,12 @@ int skcipher_walk_done(struct skcipher_walk *walk, int res) scatterwalk_done(&walk->out, 1, total); if (total) { crypto_yield(walk->flags & SKCIPHER_WALK_SLEEP ? CRYPTO_TFM_REQ_MAY_SLEEP : 0); + walk->flags &= ~(SKCIPHER_WALK_SLOW | SKCIPHER_WALK_COPY | + SKCIPHER_WALK_DIFF); return skcipher_walk_next(walk); } finish: /* Short-circuit for the common/fast path. */ @@ -233,13 +235,10 @@ static int skcipher_next_fast(struct skcipher_walk *walk) static int skcipher_walk_next(struct skcipher_walk *walk) { unsigned int bsize; unsigned int n; - walk->flags &= ~(SKCIPHER_WALK_SLOW | SKCIPHER_WALK_COPY | - SKCIPHER_WALK_DIFF); - n = walk->total; bsize = min(walk->stride, max(n, walk->blocksize)); n = scatterwalk_clamp(&walk->in, n); n = scatterwalk_clamp(&walk->out, n); @@ -309,55 +308,53 @@ static int skcipher_walk_first(struct skcipher_walk *walk) int skcipher_walk_virt(struct skcipher_walk *walk, struct skcipher_request *req, bool atomic) { struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct skcipher_alg *alg = crypto_skcipher_alg(tfm); - int err = 0; might_sleep_if(req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP); walk->total = req->cryptlen; walk->nbytes = 0; walk->iv = req->iv; walk->oiv = req->iv; + if ((req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) && !atomic) + walk->flags = SKCIPHER_WALK_SLEEP; + else + walk->flags = 0; if (unlikely(!walk->total)) - goto out; + return 0; scatterwalk_start(&walk->in, req->src); scatterwalk_start(&walk->out, req->dst); - walk->flags &= ~SKCIPHER_WALK_SLEEP; - walk->flags |= req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ? - SKCIPHER_WALK_SLEEP : 0; - walk->blocksize = crypto_skcipher_blocksize(tfm); walk->ivsize = crypto_skcipher_ivsize(tfm); walk->alignmask = crypto_skcipher_alignmask(tfm); if (alg->co.base.cra_type != &crypto_skcipher_type) walk->stride = alg->co.chunksize; else walk->stride = alg->walksize; - err = skcipher_walk_first(walk); -out: - walk->flags &= atomic ? ~SKCIPHER_WALK_SLEEP : ~0; - - return err; + return skcipher_walk_first(walk); } EXPORT_SYMBOL_GPL(skcipher_walk_virt); static int skcipher_walk_aead_common(struct skcipher_walk *walk, struct aead_request *req, bool atomic) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); - int err; walk->nbytes = 0; walk->iv = req->iv; walk->oiv = req->iv; + if ((req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) && !atomic) + walk->flags = SKCIPHER_WALK_SLEEP; + else + walk->flags = 0; if (unlikely(!walk->total)) return 0; scatterwalk_start(&walk->in, req->src); @@ -367,26 +364,16 @@ static int skcipher_walk_aead_common(struct skcipher_walk *walk, scatterwalk_copychunks(NULL, &walk->out, req->assoclen, 2); scatterwalk_done(&walk->in, 0, walk->total); scatterwalk_done(&walk->out, 0, walk->total); - if (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) - walk->flags |= SKCIPHER_WALK_SLEEP; - else - walk->flags &= ~SKCIPHER_WALK_SLEEP; - walk->blocksize = crypto_aead_blocksize(tfm); walk->stride = crypto_aead_chunksize(tfm); walk->ivsize = crypto_aead_ivsize(tfm); walk->alignmask = crypto_aead_alignmask(tfm); - err = skcipher_walk_first(walk); - - if (atomic) - walk->flags &= ~SKCIPHER_WALK_SLEEP; - - return err; + return skcipher_walk_first(walk); } int skcipher_walk_aead_encrypt(struct skcipher_walk *walk, struct aead_request *req, bool atomic) { From patchwork Mon Dec 30 00:13:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922910 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A73361369A8; Mon, 30 Dec 2024 00:16:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517761; cv=none; b=iTbaacasC0sXjPrLg2ZtpHrZKq9c79YvBfmwK7eRsnVFEKxSPBvH+zFfvzsNrWw+dwRkxI2feqzAQxjxWsMAJodD6lHf8w7L6Ctr5Rm3Ok/S6gWuktimwMp/txax1zXHLU5pMV80QdFifkEmsTlkgLwbJ/Ct5MdKpoBtwv9VlcU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517761; c=relaxed/simple; bh=6EPGGWMvq5sU1DZayuB4lsmsIXL+PbHsRHle3o9unzc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=toxzF+NZv+A2+ZI71TwHy1e4LfzW2gCF9apafy/y5Klw4sRwnr1hDUrWbsaiYCxU/Q74wQbWTLOYPawL3SBaKdmrUrR+JepvrcQLKoxgSlR9NwXBOxGWgfmeEuFA9cOjck4Aj6GIyIYAqqsqIvxD0VtkIWZHaVEbP/8a7m91y+8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=CJT7yIN/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="CJT7yIN/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F09F7C4CED7; Mon, 30 Dec 2024 00:16:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517761; bh=6EPGGWMvq5sU1DZayuB4lsmsIXL+PbHsRHle3o9unzc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CJT7yIN/x8Vbgl7vzKa9DUFxpKxXjlDYqBOoweuzubcg++adAVauicPe2irkETNiy N9V9A9vWCBKLle2tPEBRkbJDUoHnOWyELog2V+VF7GXYDTJBXlvMM5LqI3Y/YVVPdZ 1pgUPOSEngfZ7gg4Rxu/QCdGCMfWuyODgVEgfQMU/g89XOhx2ytO0wvQR6RiT0lYlX O11YiZT1b737V4FIn7qc64mgHq42Uk5CBY12oOH9kld/VUC/pjUiG2QZa0S2HuAOz/ Ajefs/fJHAXmfUgBLX81s4v3i7n8gYaYIvjz/H3vw/jAJ0VNgIN1ESbeyR8zS3SrZl OS2Qu2BGpTSnw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 07/29] crypto: skcipher - optimize initializing skcipher_walk fields Date: Sun, 29 Dec 2024 16:13:56 -0800 Message-ID: <20241230001418.74739-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers The helper functions like crypto_skcipher_blocksize() take in a pointer to a tfm object, but they actually return properties of the algorithm. As the Linux kernel is compiled with -fno-strict-aliasing, the compiler has to assume that the writes to struct skcipher_walk could clobber the tfm's pointer to its algorithm. Thus it gets repeatedly reloaded in the generated code. Therefore, replace the use of these helper functions with staightforward accesses to the struct fields. Note that while *users* of the skcipher and aead APIs are supposed to use the helper functions, this particular code is part of the API *implementation* in crypto/skcipher.c, which already accesses the algorithm struct directly in many cases. So there is no reason to prefer the helper functions here. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index e54d1ad46566..7ef2e4ddf07a 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -306,12 +306,12 @@ static int skcipher_walk_first(struct skcipher_walk *walk) } int skcipher_walk_virt(struct skcipher_walk *walk, struct skcipher_request *req, bool atomic) { - struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); - struct skcipher_alg *alg = crypto_skcipher_alg(tfm); + const struct skcipher_alg *alg = + crypto_skcipher_alg(crypto_skcipher_reqtfm(req)); might_sleep_if(req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP); walk->total = req->cryptlen; walk->nbytes = 0; @@ -326,13 +326,13 @@ int skcipher_walk_virt(struct skcipher_walk *walk, return 0; scatterwalk_start(&walk->in, req->src); scatterwalk_start(&walk->out, req->dst); - walk->blocksize = crypto_skcipher_blocksize(tfm); - walk->ivsize = crypto_skcipher_ivsize(tfm); - walk->alignmask = crypto_skcipher_alignmask(tfm); + walk->blocksize = alg->base.cra_blocksize; + walk->ivsize = alg->co.ivsize; + walk->alignmask = alg->base.cra_alignmask; if (alg->co.base.cra_type != &crypto_skcipher_type) walk->stride = alg->co.chunksize; else walk->stride = alg->walksize; @@ -342,11 +342,11 @@ int skcipher_walk_virt(struct skcipher_walk *walk, EXPORT_SYMBOL_GPL(skcipher_walk_virt); static int skcipher_walk_aead_common(struct skcipher_walk *walk, struct aead_request *req, bool atomic) { - struct crypto_aead *tfm = crypto_aead_reqtfm(req); + const struct aead_alg *alg = crypto_aead_alg(crypto_aead_reqtfm(req)); walk->nbytes = 0; walk->iv = req->iv; walk->oiv = req->iv; if ((req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) && !atomic) @@ -364,14 +364,14 @@ static int skcipher_walk_aead_common(struct skcipher_walk *walk, scatterwalk_copychunks(NULL, &walk->out, req->assoclen, 2); scatterwalk_done(&walk->in, 0, walk->total); scatterwalk_done(&walk->out, 0, walk->total); - walk->blocksize = crypto_aead_blocksize(tfm); - walk->stride = crypto_aead_chunksize(tfm); - walk->ivsize = crypto_aead_ivsize(tfm); - walk->alignmask = crypto_aead_alignmask(tfm); + walk->blocksize = alg->base.cra_blocksize; + walk->stride = alg->chunksize; + walk->ivsize = alg->ivsize; + walk->alignmask = alg->base.cra_alignmask; return skcipher_walk_first(walk); } int skcipher_walk_aead_encrypt(struct skcipher_walk *walk, From patchwork Mon Dec 30 00:13:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922911 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AEC486AE3; Mon, 30 Dec 2024 00:16:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517761; cv=none; b=k2/2d+t6pc3yynNqH4iwp9xDzhxhhDP5CGqIcn0RZf2bo5KbDsY6HnUrIx0hAcTX/zlyEWNvFnHlKVz1+sFru7VpUTjYUMK/xaOo3mZ9aQeoqqveV7JQnWvFh7yQlhZHubKUTtbnQsx9KdQVcd6uNnGeHwV61ZKRTTHQU9tSWek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517761; c=relaxed/simple; bh=2nhmId8pbA4hd6z9AgDYhLqfIP1ELc8P6LVbb7WGcqw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p8ZCqVbmpDsERb0tRi/b2ocDDVuTGwxamtKJfBcZQpXVgkt4D2HBoJRmEAC7ZTYWo3i6grtsLJQT9Jltrc5sPzzNzjEhgWjjt4gsducqrIddEtwUGzhRhlmuRyIcaUBW1XKazpHUejnehRr6i1QajHvGi9pBOQGCIGxNreFDLAY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZMFkLnqR; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZMFkLnqR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B3CEC4CEF1; Mon, 30 Dec 2024 00:16:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517761; bh=2nhmId8pbA4hd6z9AgDYhLqfIP1ELc8P6LVbb7WGcqw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZMFkLnqRtA04wVaHvzFSb6Gmt7tuCWYk6txsA7zvAlBLdQWt7M/lTNI1Pd1JSpzIx bleUScTLiFxDqucG1nRoqFi6fskuYKpxUXtuyv2GqmEzg6zZMtAlO4Gtd0qH/V9/S0 UumW5V9vZ7lj0Zyf8pker73f1dpmrFVfnCtKKVk1YMBrbFfTMTRDTgFdZgD9ev3pvg BtNRFoxVJljicnRWnljKoWn4bK/9zmszf6hSUZpEqbH2LNgnQZsJEWkaiHMu0v/kGW ocS08eH6t3kRx75WwOyCTS74IQdSlPJXp/CS7u+ljqaaL0vVCHELZQbuBCIKhz82PE KKsxc/tFOY5QA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 08/29] crypto: skcipher - call cond_resched() directly Date: Sun, 29 Dec 2024 16:13:57 -0800 Message-ID: <20241230001418.74739-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In skcipher_walk_done(), instead of calling crypto_yield() which requires a translation between flags, just call cond_resched() directly. This has the same effect. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 7ef2e4ddf07a..441e1d254d36 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -144,12 +144,12 @@ int skcipher_walk_done(struct skcipher_walk *walk, int res) scatterwalk_advance(&walk->out, n); scatterwalk_done(&walk->in, 0, total); scatterwalk_done(&walk->out, 1, total); if (total) { - crypto_yield(walk->flags & SKCIPHER_WALK_SLEEP ? - CRYPTO_TFM_REQ_MAY_SLEEP : 0); + if (walk->flags & SKCIPHER_WALK_SLEEP) + cond_resched(); walk->flags &= ~(SKCIPHER_WALK_SLOW | SKCIPHER_WALK_COPY | SKCIPHER_WALK_DIFF); return skcipher_walk_next(walk); } From patchwork Mon Dec 30 00:13:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922913 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00B9514386D; Mon, 30 Dec 2024 00:16:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517762; cv=none; b=aJvW6HmgVjXpYmD4qqG1DDCM35JpK/6+HkKlaM0CGyU+LL0t28oK6x5+DGismVkEYguZojBuF9L5diszg9dCK5mX5f5e4y99bmOkUXBLjN4eNWB82i59/qhWcCG45ygTV6EZFL0CahdojTy2XYqQqvEtTXmvc9DJQap6KIuQuFM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517762; c=relaxed/simple; bh=hjTDNu4OaDV1N4uaEc/+cpuTwHMmIUw/rjZOhyi9fg8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZlWhZeQx2cfRam7k2X3gyEAOGFPhnJzLmr5EY/1ueo7vCD3Yt7ysY+4sKET8sai/AlV67gz6Gx7/+QyKC66zOYhSzFiE6OzHnn4YbyBYsU21xOR2YxfEiJwsFtOSXhtef/QNpD+TBERJozvoI6BSHf8T5S+pUifXpU772TUWzxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=f/DkDDcK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="f/DkDDcK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5C12C4CED1; Mon, 30 Dec 2024 00:16:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517761; bh=hjTDNu4OaDV1N4uaEc/+cpuTwHMmIUw/rjZOhyi9fg8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f/DkDDcKcQOda246cWeezCWHmdGidIBA0Df8MW1b5geBRMLGW6n+ho1oRQ9PqOumb zeQrF13BhWBbITct2k7UzINIsz6VBoUgdztgDgJu0qHXTamGdWTdG6uW2d1J6RBH3I VRPXPACRikyjQNczIYhh3rRsBqtglGeLZkJrs2vcHOv9JJZyI3JwwbVc3PIP0UbmqL iVRlzofzk7ANGDoeGZYN9Cn3Qa4cAIIbWVS6BtMQ76LxmPH3xdOmfLZpQTibn6GgUB EVMnsO37pAdTcjbg8dHl+q6clXS4kYpbVQL1LatH225i7tmsYsDvWpBVtzEUCLSj+j UQDsOq5zwPIrw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 09/29] crypto: omap - switch from scatter_walk to plain offset Date: Sun, 29 Dec 2024 16:13:58 -0800 Message-ID: <20241230001418.74739-10-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers The omap driver was using struct scatter_walk, but only to maintain an offset, rather than iterating through the virtual addresses of the data contained in the scatterlist which is what scatter_walk is intended for. Make it just use a plain offset instead. This is simpler and avoids using struct scatter_walk in a way that is not well supported. Signed-off-by: Eric Biggers --- drivers/crypto/omap-aes.c | 34 ++++++++++++++------------------- drivers/crypto/omap-aes.h | 6 ++---- drivers/crypto/omap-des.c | 40 ++++++++++++++++----------------------- 3 files changed, 32 insertions(+), 48 deletions(-) diff --git a/drivers/crypto/omap-aes.c b/drivers/crypto/omap-aes.c index e27b84616743..551dd32a8db0 100644 --- a/drivers/crypto/omap-aes.c +++ b/drivers/crypto/omap-aes.c @@ -16,11 +16,10 @@ #include #include #include #include #include -#include #include #include #include #include #include @@ -270,13 +269,13 @@ static int omap_aes_crypt_dma(struct omap_aes_dev *dd, struct dma_async_tx_descriptor *tx_in, *tx_out = NULL, *cb_desc; struct dma_slave_config cfg; int ret; if (dd->pio_only) { - scatterwalk_start(&dd->in_walk, dd->in_sg); + dd->in_sg_offset = 0; if (out_sg_len) - scatterwalk_start(&dd->out_walk, dd->out_sg); + dd->out_sg_offset = 0; /* Enable DATAIN interrupt and let it take care of the rest */ omap_aes_write(dd, AES_REG_IRQ_ENABLE(dd), 0x2); return 0; @@ -869,25 +868,22 @@ static irqreturn_t omap_aes_irq(int irq, void *dev_id) if (status & AES_REG_IRQ_DATA_IN) { omap_aes_write(dd, AES_REG_IRQ_ENABLE(dd), 0x0); BUG_ON(!dd->in_sg); - BUG_ON(_calc_walked(in) > dd->in_sg->length); + BUG_ON(dd->in_sg_offset > dd->in_sg->length); - src = sg_virt(dd->in_sg) + _calc_walked(in); + src = sg_virt(dd->in_sg) + dd->in_sg_offset; for (i = 0; i < AES_BLOCK_WORDS; i++) { omap_aes_write(dd, AES_REG_DATA_N(dd, i), *src); - - scatterwalk_advance(&dd->in_walk, 4); - if (dd->in_sg->length == _calc_walked(in)) { + dd->in_sg_offset += 4; + if (dd->in_sg_offset == dd->in_sg->length) { dd->in_sg = sg_next(dd->in_sg); if (dd->in_sg) { - scatterwalk_start(&dd->in_walk, - dd->in_sg); - src = sg_virt(dd->in_sg) + - _calc_walked(in); + dd->in_sg_offset = 0; + src = sg_virt(dd->in_sg); } } else { src++; } } @@ -902,24 +898,22 @@ static irqreturn_t omap_aes_irq(int irq, void *dev_id) } else if (status & AES_REG_IRQ_DATA_OUT) { omap_aes_write(dd, AES_REG_IRQ_ENABLE(dd), 0x0); BUG_ON(!dd->out_sg); - BUG_ON(_calc_walked(out) > dd->out_sg->length); + BUG_ON(dd->out_sg_offset > dd->out_sg->length); - dst = sg_virt(dd->out_sg) + _calc_walked(out); + dst = sg_virt(dd->out_sg) + dd->out_sg_offset; for (i = 0; i < AES_BLOCK_WORDS; i++) { *dst = omap_aes_read(dd, AES_REG_DATA_N(dd, i)); - scatterwalk_advance(&dd->out_walk, 4); - if (dd->out_sg->length == _calc_walked(out)) { + dd->out_sg_offset += 4; + if (dd->out_sg_offset == dd->out_sg->length) { dd->out_sg = sg_next(dd->out_sg); if (dd->out_sg) { - scatterwalk_start(&dd->out_walk, - dd->out_sg); - dst = sg_virt(dd->out_sg) + - _calc_walked(out); + dd->out_sg_offset = 0; + dst = sg_virt(dd->out_sg); } } else { dst++; } } diff --git a/drivers/crypto/omap-aes.h b/drivers/crypto/omap-aes.h index 0f35c9164764..41d67780fd45 100644 --- a/drivers/crypto/omap-aes.h +++ b/drivers/crypto/omap-aes.h @@ -12,12 +12,10 @@ #include #define DST_MAXBURST 4 #define DMA_MIN (DST_MAXBURST * sizeof(u32)) -#define _calc_walked(inout) (dd->inout##_walk.offset - dd->inout##_sg->offset) - /* * OMAP TRM gives bitfields as start:end, where start is the higher bit * number. For example 7:0 */ #define FLD_MASK(start, end) (((1 << ((start) - (end) + 1)) - 1) << (end)) @@ -184,12 +182,12 @@ struct omap_aes_dev { /* Buffers for copying for unaligned cases */ struct scatterlist in_sgl[2]; struct scatterlist out_sgl; struct scatterlist *orig_out; - struct scatter_walk in_walk; - struct scatter_walk out_walk; + unsigned int in_sg_offset; + unsigned int out_sg_offset; struct dma_chan *dma_lch_in; struct dma_chan *dma_lch_out; int in_sg_len; int out_sg_len; int pio_only; diff --git a/drivers/crypto/omap-des.c b/drivers/crypto/omap-des.c index 498cbd585ed1..a099460d5f21 100644 --- a/drivers/crypto/omap-des.c +++ b/drivers/crypto/omap-des.c @@ -17,11 +17,10 @@ #endif #include #include #include -#include #include #include #include #include #include @@ -38,12 +37,10 @@ #define DST_MAXBURST 2 #define DES_BLOCK_WORDS (DES_BLOCK_SIZE >> 2) -#define _calc_walked(inout) (dd->inout##_walk.offset - dd->inout##_sg->offset) - #define DES_REG_KEY(dd, x) ((dd)->pdata->key_ofs - \ ((x ^ 0x01) * 0x04)) #define DES_REG_IV(dd, x) ((dd)->pdata->iv_ofs + ((x) * 0x04)) @@ -150,12 +147,12 @@ struct omap_des_dev { /* Buffers for copying for unaligned cases */ struct scatterlist in_sgl; struct scatterlist out_sgl; struct scatterlist *orig_out; - struct scatter_walk in_walk; - struct scatter_walk out_walk; + unsigned int in_sg_offset; + unsigned int out_sg_offset; struct dma_chan *dma_lch_in; struct dma_chan *dma_lch_out; int in_sg_len; int out_sg_len; int pio_only; @@ -377,12 +374,12 @@ static int omap_des_crypt_dma(struct crypto_tfm *tfm, struct dma_async_tx_descriptor *tx_in, *tx_out; struct dma_slave_config cfg; int ret; if (dd->pio_only) { - scatterwalk_start(&dd->in_walk, dd->in_sg); - scatterwalk_start(&dd->out_walk, dd->out_sg); + dd->in_sg_offset = 0; + dd->out_sg_offset = 0; /* Enable DATAIN interrupt and let it take care of the rest */ omap_des_write(dd, DES_REG_IRQ_ENABLE(dd), 0x2); return 0; @@ -834,25 +831,22 @@ static irqreturn_t omap_des_irq(int irq, void *dev_id) if (status & DES_REG_IRQ_DATA_IN) { omap_des_write(dd, DES_REG_IRQ_ENABLE(dd), 0x0); BUG_ON(!dd->in_sg); - BUG_ON(_calc_walked(in) > dd->in_sg->length); + BUG_ON(dd->in_sg_offset > dd->in_sg->length); - src = sg_virt(dd->in_sg) + _calc_walked(in); + src = sg_virt(dd->in_sg) + dd->in_sg_offset; for (i = 0; i < DES_BLOCK_WORDS; i++) { omap_des_write(dd, DES_REG_DATA_N(dd, i), *src); - - scatterwalk_advance(&dd->in_walk, 4); - if (dd->in_sg->length == _calc_walked(in)) { + dd->in_sg_offset += 4; + if (dd->in_sg_offset == dd->in_sg->length) { dd->in_sg = sg_next(dd->in_sg); if (dd->in_sg) { - scatterwalk_start(&dd->in_walk, - dd->in_sg); - src = sg_virt(dd->in_sg) + - _calc_walked(in); + dd->in_sg_offset = 0; + src = sg_virt(dd->in_sg); } } else { src++; } } @@ -867,24 +861,22 @@ static irqreturn_t omap_des_irq(int irq, void *dev_id) } else if (status & DES_REG_IRQ_DATA_OUT) { omap_des_write(dd, DES_REG_IRQ_ENABLE(dd), 0x0); BUG_ON(!dd->out_sg); - BUG_ON(_calc_walked(out) > dd->out_sg->length); + BUG_ON(dd->out_sg_offset > dd->out_sg->length); - dst = sg_virt(dd->out_sg) + _calc_walked(out); + dst = sg_virt(dd->out_sg) + dd->out_sg_offset; for (i = 0; i < DES_BLOCK_WORDS; i++) { *dst = omap_des_read(dd, DES_REG_DATA_N(dd, i)); - scatterwalk_advance(&dd->out_walk, 4); - if (dd->out_sg->length == _calc_walked(out)) { + dd->out_sg_offset += 4; + if (dd->out_sg_offset == dd->out_sg->length) { dd->out_sg = sg_next(dd->out_sg); if (dd->out_sg) { - scatterwalk_start(&dd->out_walk, - dd->out_sg); - dst = sg_virt(dd->out_sg) + - _calc_walked(out); + dd->out_sg_offset = 0; + dst = sg_virt(dd->out_sg); } } else { dst++; } } From patchwork Mon Dec 30 00:13:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922914 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C218A183CD9; Mon, 30 Dec 2024 00:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517762; cv=none; b=gLtSYpD3qoGi0F+FatoTIKfnYdR69gZFHPX9r/Yno0Un0mOevaeZtGUYSQ4uG5+aUPg5Ae2iJPRgZdrS32vprKlc+RdxxjGTNmsoHoJrjeiEGq0nZDF/ahtvnUHvEKQaq+ypbTl6AHMFWkrGA83NVvbKP9Zl42H6F6qaf+2phT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517762; c=relaxed/simple; bh=l3mcIF8SEtYBua+ussMVqyNpUvcPBoNrQm89sO2KshY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BcXI3vFRhKAm25FoiZ1hUL6lkuDv8uEcJyQjXZrjJGazDqAdAUSDQVeSbMEms/LD4Bu58ocQkwt7CKSbLA2FU8TvyaMtQCHuE4PX/p/nVrXPKgsXWxqG/HLcWc/hnraW8AhZlwRdWPlF7eE9VGHj4AX+62suVojXtHE7X1FnBHI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=O5il+uw4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="O5il+uw4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F3560C4CEE1; Mon, 30 Dec 2024 00:16:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517762; bh=l3mcIF8SEtYBua+ussMVqyNpUvcPBoNrQm89sO2KshY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=O5il+uw47yUf4bNQKXq32psvRn/OjXkiWSchhO7EH6YLcHiQWhfzIVuy2natJvgp3 coB7MaqzMUG6J7buiQVoj5N/OY8tkKYlmOXu5jF8SiHshrlMngtjET2Ep0xcwH4K+T hcsuUbcDsZHW43Rw7UlkwXRvKFsEQ44k/RMVggp8D42zCvK34maSXPPf+NnMdKQDE+ f+61fYWvEW9BxwsGNqpK5feRzlwLXjtMFv7upZE5CIXw3nYJ0egqn+Zzl+pOmC28x6 Urz2j59NsldJo5BjW3a94bf5OSSPryQ03rKqfh5pG4OxZdLUQE/tj7SQukbbY7vqIb vM4xU3YqhtSHQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Christophe Leroy , Danny Tsen , Michael Ellerman , Naveen N Rao , Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 10/29] crypto: powerpc/p10-aes-gcm - simplify handling of linear associated data Date: Sun, 29 Dec 2024 16:13:59 -0800 Message-ID: <20241230001418.74739-11-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers p10_aes_gcm_crypt() is abusing the scatter_walk API to get the virtual address for the first source scatterlist element. But this code is only built for PPC64 which is a !HIGHMEM platform, and it can read past a page boundary from the address returned by scatterwalk_map() which means it already assumes the address is from the kernel's direct map. Thus, just use sg_virt() instead to get the same result in a simpler way. Cc: Christophe Leroy Cc: Danny Tsen Cc: Michael Ellerman Cc: Naveen N Rao Cc: Nicholas Piggin Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. arch/powerpc/crypto/aes-gcm-p10-glue.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/arch/powerpc/crypto/aes-gcm-p10-glue.c b/arch/powerpc/crypto/aes-gcm-p10-glue.c index f37b3d13fc53..2862c3cf8e41 100644 --- a/arch/powerpc/crypto/aes-gcm-p10-glue.c +++ b/arch/powerpc/crypto/aes-gcm-p10-glue.c @@ -212,11 +212,10 @@ static int p10_aes_gcm_crypt(struct aead_request *req, u8 *riv, struct p10_aes_gcm_ctx *ctx = crypto_tfm_ctx(tfm); u8 databuf[sizeof(struct gcm_ctx) + PPC_ALIGN]; struct gcm_ctx *gctx = PTR_ALIGN((void *)databuf, PPC_ALIGN); u8 hashbuf[sizeof(struct Hash_ctx) + PPC_ALIGN]; struct Hash_ctx *hash = PTR_ALIGN((void *)hashbuf, PPC_ALIGN); - struct scatter_walk assoc_sg_walk; struct skcipher_walk walk; u8 *assocmem = NULL; u8 *assoc; unsigned int cryptlen = req->cryptlen; unsigned char ivbuf[AES_BLOCK_SIZE+PPC_ALIGN]; @@ -232,12 +231,11 @@ static int p10_aes_gcm_crypt(struct aead_request *req, u8 *riv, memset(ivbuf, 0, sizeof(ivbuf)); memcpy(iv, riv, GCM_IV_SIZE); /* Linearize assoc, if not already linear */ if (req->src->length >= assoclen && req->src->length) { - scatterwalk_start(&assoc_sg_walk, req->src); - assoc = scatterwalk_map(&assoc_sg_walk); + assoc = sg_virt(req->src); /* ppc64 is !HIGHMEM */ } else { gfp_t flags = (req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP) ? GFP_KERNEL : GFP_ATOMIC; /* assoc can be any length, so must be on heap */ @@ -251,13 +249,11 @@ static int p10_aes_gcm_crypt(struct aead_request *req, u8 *riv, vsx_begin(); gcmp10_init(gctx, iv, (unsigned char *) &ctx->enc_key, hash, assoc, assoclen); vsx_end(); - if (!assocmem) - scatterwalk_unmap(assoc); - else + if (assocmem) kfree(assocmem); if (enc) ret = skcipher_walk_aead_encrypt(&walk, req, false); else From patchwork Mon Dec 30 00:14:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922915 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 029AB18991E; Mon, 30 Dec 2024 00:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517763; cv=none; b=TEF1y2mDinncO7jr0ZUKGEKx4ilqMg56h0mSQBmBGMYEZXy7x52htaX1TpePqiwtQ6RgvHuM9Qlqyr7dGSaIzHV8ACb5SC2pybiYDvIR8R64JtybqnSWqGiR1yy7mBed08QVr0yQKbzTRYvpPPy7+lpzIFjm69Tu5kGcTbMOObo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517763; c=relaxed/simple; bh=IpLvn2XtgphqyEr3C9o4IvzLjuZxaBrxGrUoOcvifmA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YMgi4DYSRVbEM99G4gDt47BS5pKKeSqNnUajltlNo3yRiJM34AtvxaNmCi6FDtXEacUSQj6XsyY/Lvcf+yPw5M0RzjvFAvD74ykjtkoRRPo2LuNeW6hcy/LRrf9Ix3U2fO2/KdOL7LZabrY3n7ri+z0+yLlGH9JHSLh+IHv6PxI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JGKclWey; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JGKclWey" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 72356C4CEE5; Mon, 30 Dec 2024 00:16:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517762; bh=IpLvn2XtgphqyEr3C9o4IvzLjuZxaBrxGrUoOcvifmA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JGKclWeygDuiTSJhXU6d/wBYsCbxpUiMSCrVqgigcPIyzmlqEImT78huP0qfJ6gWM aVlg2a/aqw/a8eqW5vkWod7VxLsN+GEGj+VQbK72d7Q7oqbNn6RaesM545RZDZK3Or T2dyyon1XLidW7e0VunKESBOePbl+Y2CI21uXkGT2r/UK5MlhIk0u/NzZ8HavvnSs5 Ah2EwAJmmlDh4eibTzN9dqX2HKUegN/hHWBlpt1ygRbFv7RSdvfAMDi0Sv2E7P+SQ4 EA0F/3vLftFgv0T6Q0gFjXTKOAIon3UmLEllJjpBtDRDTS0Af8MEOO5uK1YYkpXVrm UaevCQEGAcwtw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 11/29] crypto: scatterwalk - move to next sg entry just in time Date: Sun, 29 Dec 2024 16:14:00 -0800 Message-ID: <20241230001418.74739-12-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers The scatterwalk_* functions are designed to advance to the next sg entry only when there is more data from the request to process. Compared to the alternative of advancing after each step if !sg_is_last(sg), this has the advantage that it doesn't cause problems if users accidentally don't terminate their scatterlist with the end marker (which is an easy mistake to make, and there are examples of this). Currently, the advance to the next sg entry happens in scatterwalk_done(), which is called after each "step" of the walk. It requires the caller to pass in a boolean 'more' that indicates whether there is more data. This works when the caller immediately knows whether there is more data, though it adds some complexity. However in the case of scatterwalk_copychunks() it's not immediately known whether there is more data, so the call to scatterwalk_done() has to happen higher up the stack. This is error-prone, and indeed the needed call to scatterwalk_done() is not always made, e.g. scatterwalk_copychunks() is sometimes called multiple times in a row. This causes a zero-length step to get added in some cases, which is unexpected and seems to work only by accident. This patch begins the switch to a less error-prone approach where the advance to the next sg entry happens just in time instead. For now, that means just doing the advance in scatterwalk_clamp() if it's needed there. Initially this is redundant, but it's needed to keep the tree in a working state as later patches change things to the final state. Later patches will similarly move the dcache flushing logic out of scatterwalk_done() and then remove scatterwalk_done() entirely. Signed-off-by: Eric Biggers --- include/crypto/scatterwalk.h | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index 32fc4473175b..924efbaefe67 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -24,22 +24,30 @@ static inline void scatterwalk_crypto_chain(struct scatterlist *head, sg_chain(head, num, sg); else sg_mark_end(head); } +static inline void scatterwalk_start(struct scatter_walk *walk, + struct scatterlist *sg) +{ + walk->sg = sg; + walk->offset = sg->offset; +} + static inline unsigned int scatterwalk_pagelen(struct scatter_walk *walk) { unsigned int len = walk->sg->offset + walk->sg->length - walk->offset; unsigned int len_this_page = offset_in_page(~walk->offset) + 1; return len_this_page > len ? len : len_this_page; } static inline unsigned int scatterwalk_clamp(struct scatter_walk *walk, unsigned int nbytes) { - unsigned int len_this_page = scatterwalk_pagelen(walk); - return nbytes > len_this_page ? len_this_page : nbytes; + if (walk->offset >= walk->sg->offset + walk->sg->length) + scatterwalk_start(walk, sg_next(walk->sg)); + return min(nbytes, scatterwalk_pagelen(walk)); } static inline void scatterwalk_advance(struct scatter_walk *walk, unsigned int nbytes) { @@ -54,17 +62,10 @@ static inline struct page *scatterwalk_page(struct scatter_walk *walk) static inline void scatterwalk_unmap(void *vaddr) { kunmap_local(vaddr); } -static inline void scatterwalk_start(struct scatter_walk *walk, - struct scatterlist *sg) -{ - walk->sg = sg; - walk->offset = sg->offset; -} - static inline void *scatterwalk_map(struct scatter_walk *walk) { return kmap_local_page(scatterwalk_page(walk)) + offset_in_page(walk->offset); } From patchwork Mon Dec 30 00:14:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922916 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 399C719882F; Mon, 30 Dec 2024 00:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517763; cv=none; b=j77ioStkYkWm2d35FJr6R2BJ36C6DDhWhMk34tSg1A8jpoUthV4f5GuBW+VtpNkk5Ls819h7UisqY/uUjUK4w4dJpys0nVRseGrMDnzQdCDxIm7SOkHkgukAoPEUv74UL9x6QI2FMiZsjzjZWBpyoLl7iy+kL0j4jryDmJCrWgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517763; c=relaxed/simple; bh=Aca6uPsUCVLdX4GRY5oc+9eRqWCqEIgCJje21Fl6qos=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gAmDrLfs79xjxS97qe8R8h345NV+M/V2hGrshgwSb1UJwteoG4xnmJnccJh+SAxAFOJiVwb2pw+BlycJWwmyzIYk4GvDw8CUlkromPpmCmrqVZr7YqkVViQFwByUiPWmUI8kaRiQQVuCUYePOpirg9V75V4nDTzFBoLvduOxYPc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IaqQTqpr; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IaqQTqpr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B224CC4CED7; Mon, 30 Dec 2024 00:16:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517762; bh=Aca6uPsUCVLdX4GRY5oc+9eRqWCqEIgCJje21Fl6qos=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IaqQTqprjEebzD3/APWUfCv730v52GNpEKc3mhF1u2/UbChVtIT3twHh5P+t8zDVI nk5lCqsKkOxkxvrIDUDsDK3MRuwchANaeymOiIFYGcebqYDCS5UwNkaZAcH1x4Anvw owynb3sn7ml9iLVL8FydhgWL8AFLPAvv7TmroPwRaDChh6kRhP25514uIn/kNubFBc VzFPfOdubPh7brwHWto3LPUHizBH70XlcQv4R9n4SLzBtX3XJU+00PS6Nxb6FwIdWl KPhqu4RL5LLq0w84CvcSHVbfki6lt+RXRlAPuYyAlgVOwQchtItNKG4YUedWLN/WLe zRta/MpzNMuOQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 12/29] crypto: scatterwalk - add new functions for skipping data Date: Sun, 29 Dec 2024 16:14:01 -0800 Message-ID: <20241230001418.74739-13-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Add scatterwalk_skip() to skip the given number of bytes in a scatter_walk. Previously support for skipping was provided through scatterwalk_copychunks(..., 2) followed by scatterwalk_done(), which was confusing and less efficient. Also add scatterwalk_start_at_pos() which starts a scatter_walk at the given position, equivalent to scatterwalk_start() + scatterwalk_skip(). This addresses another common need in a more streamlined way. Later patches will convert various users to use these functions. Signed-off-by: Eric Biggers --- crypto/scatterwalk.c | 15 +++++++++++++++ include/crypto/scatterwalk.h | 18 ++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/crypto/scatterwalk.c b/crypto/scatterwalk.c index 16f6ba896fb6..af436ad02e3f 100644 --- a/crypto/scatterwalk.c +++ b/crypto/scatterwalk.c @@ -13,10 +13,25 @@ #include #include #include #include +void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes) +{ + struct scatterlist *sg = walk->sg; + + nbytes += walk->offset - sg->offset; + + while (nbytes > sg->length) { + nbytes -= sg->length; + sg = sg_next(sg); + } + walk->sg = sg; + walk->offset = sg->offset + nbytes; +} +EXPORT_SYMBOL_GPL(scatterwalk_skip); + static inline void memcpy_dir(void *buf, void *sgdata, size_t nbytes, int out) { void *src = out ? buf : sgdata; void *dst = out ? sgdata : buf; diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index 924efbaefe67..5c7765f601e0 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -31,10 +31,26 @@ static inline void scatterwalk_start(struct scatter_walk *walk, { walk->sg = sg; walk->offset = sg->offset; } +/* + * This is equivalent to scatterwalk_start(walk, sg) followed by + * scatterwalk_skip(walk, pos). + */ +static inline void scatterwalk_start_at_pos(struct scatter_walk *walk, + struct scatterlist *sg, + unsigned int pos) +{ + while (pos > sg->length) { + pos -= sg->length; + sg = sg_next(sg); + } + walk->sg = sg; + walk->offset = sg->offset + pos; +} + static inline unsigned int scatterwalk_pagelen(struct scatter_walk *walk) { unsigned int len = walk->sg->offset + walk->sg->length - walk->offset; unsigned int len_this_page = offset_in_page(~walk->offset) + 1; return len_this_page > len ? len : len_this_page; @@ -90,10 +106,12 @@ static inline void scatterwalk_done(struct scatter_walk *walk, int out, if (!more || walk->offset >= walk->sg->offset + walk->sg->length || !(walk->offset & (PAGE_SIZE - 1))) scatterwalk_pagedone(walk, out, more); } +void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes); + void scatterwalk_copychunks(void *buf, struct scatter_walk *walk, size_t nbytes, int out); void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg, unsigned int start, unsigned int nbytes, int out); From patchwork Mon Dec 30 00:14:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922917 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4B7819AD93; Mon, 30 Dec 2024 00:16:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517763; cv=none; b=a+4+y4vF5RLgkPdrl1vWzJmbQO2aU4fvyUnL8WhfIXmXlxZ0lPBqfGOTYm9K51uwnX5L7tBW3lXhMQ5znjOmd8xuBSgi0cdGtfgpHgE4Uzno4BYAxjhiMRzikCaUEfMiMsc3sdhZ7BlRc7tN8B7b2Kig7aLFfzxMYiqv2MYtaEE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517763; c=relaxed/simple; bh=gxWz7ItVuXZh7ZMr8mOrMUU9vut7iYem6KQDBMBsI2Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=m3QT+BrGGPMwwfFV2xo9ea84kkFB7RzdR99OBHb3RbwtMMSV0mYIlcnzirCxxTiNfiV9mQMKKLXpUlItaMDmQRfAdwRN+w1kNvc0MefyFU/ABvc2Q+3y+tvNh+EYh6ZEInuLQdTdublOAFKSwSwEg86fSwCsxwJB392jjDjQn98= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mqsgrDXN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mqsgrDXN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F120EC4CED1; Mon, 30 Dec 2024 00:16:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517763; bh=gxWz7ItVuXZh7ZMr8mOrMUU9vut7iYem6KQDBMBsI2Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mqsgrDXN20s8Fr17gpRjswtOcmwmPwp48qmCUjmftKxDXucDgA7zMbDojvTfGePJR 2ulGDs/k/VXAZpxixAvCBo3i72LbT03V+pzzqSEylr5W4cypx9R1T+RcbRKVYN44oO mlKjj64YSrwJCZK0gwcnutdv3DmtOTSwrshOXupGc0IHxV4qjw3n10rYTVD8wlr+fO IZWkuCbfZR/fOIeUNZ/cTrqZ+Tqo0D+yNFtSJF3RvGCGwDKe+ENnM3KtnVqGNvZY8U H070vgD/IS6rJkxkWAWwSltDR9uiPMoN3xTTOE93BnDJ+/8kXhAOQR0tqsFEqMJKzU TKd/OwN29+92g== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 13/29] crypto: scatterwalk - add new functions for iterating through data Date: Sun, 29 Dec 2024 16:14:02 -0800 Message-ID: <20241230001418.74739-14-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Add scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(). Also add scatterwalk_done_src() and scatterwalk_done_dst() which consolidate scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done() or scatterwalk_pagedone(). A later patch will remove scatterwalk_done() and scatterwalk_pagedone(). The new code eliminates the error-prone 'more' parameter. Advancing to the next sg entry now only happens just-in-time in scatterwalk_next(). The new code also pairs the dcache flush more closely with the actual write, similar to memcpy_to_page(). Previously it was paired with advancing to the next page. This is currently causing bugs where the dcache flush is incorrectly being skipped, usually due to scatterwalk_copychunks() being called without a following scatterwalk_done(). The dcache flush may have been placed where it was in order to not call flush_dcache_page() redundantly when visiting a page more than once. However, that case is rare in practice, and most architectures either do not implement flush_dcache_page() anyway or implement it lazily where it just clears a page flag. Another limitation of the old code was that by the time the flush happened, there was no way to tell if more than one page needed to be flushed. That has been sufficient because the code goes page by page, but I would like to optimize that on !HIGHMEM platforms. The new code makes this possible, and a later patch will implement this optimization. Signed-off-by: Eric Biggers --- include/crypto/scatterwalk.h | 69 ++++++++++++++++++++++++++++++++---- 1 file changed, 63 insertions(+), 6 deletions(-) diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index 5c7765f601e0..8e83c43016c9 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -62,16 +62,10 @@ static inline unsigned int scatterwalk_clamp(struct scatter_walk *walk, if (walk->offset >= walk->sg->offset + walk->sg->length) scatterwalk_start(walk, sg_next(walk->sg)); return min(nbytes, scatterwalk_pagelen(walk)); } -static inline void scatterwalk_advance(struct scatter_walk *walk, - unsigned int nbytes) -{ - walk->offset += nbytes; -} - static inline struct page *scatterwalk_page(struct scatter_walk *walk) { return sg_page(walk->sg) + (walk->offset >> PAGE_SHIFT); } @@ -84,10 +78,28 @@ static inline void *scatterwalk_map(struct scatter_walk *walk) { return kmap_local_page(scatterwalk_page(walk)) + offset_in_page(walk->offset); } +/** + * scatterwalk_next() - Get the next data buffer in a scatterlist walk + * @walk: the scatter_walk + * @total: the total number of bytes remaining, > 0 + * @nbytes_ret: (out) the next number of bytes available, <= @total + * + * Return: A virtual address for the next segment of data from the scatterlist. + * The caller must call scatterwalk_done_src() or scatterwalk_done_dst() + * when it is done using this virtual address. + */ +static inline void *scatterwalk_next(struct scatter_walk *walk, + unsigned int total, + unsigned int *nbytes_ret) +{ + *nbytes_ret = scatterwalk_clamp(walk, total); + return scatterwalk_map(walk); +} + static inline void scatterwalk_pagedone(struct scatter_walk *walk, int out, unsigned int more) { if (out) { struct page *page; @@ -106,10 +118,55 @@ static inline void scatterwalk_done(struct scatter_walk *walk, int out, if (!more || walk->offset >= walk->sg->offset + walk->sg->length || !(walk->offset & (PAGE_SIZE - 1))) scatterwalk_pagedone(walk, out, more); } +static inline void scatterwalk_advance(struct scatter_walk *walk, + unsigned int nbytes) +{ + walk->offset += nbytes; +} + +/** + * scatterwalk_done_src() - Finish one step of a walk of source scatterlist + * @walk: the scatter_walk + * @vaddr: the address returned by scatterwalk_next() + * @nbytes: the number of bytes processed this step, less than or equal to the + * number of bytes that scatterwalk_next() returned. + * + * Use this if the @vaddr was not written to, i.e. it is source data. + */ +static inline void scatterwalk_done_src(struct scatter_walk *walk, + const void *vaddr, unsigned int nbytes) +{ + scatterwalk_unmap((void *)vaddr); + scatterwalk_advance(walk, nbytes); +} + +/** + * scatterwalk_done_dst() - Finish one step of a walk of destination scatterlist + * @walk: the scatter_walk + * @vaddr: the address returned by scatterwalk_next() + * @nbytes: the number of bytes processed this step, less than or equal to the + * number of bytes that scatterwalk_next() returned. + * + * Use this if the @vaddr may have been written to, i.e. it is destination data. + */ +static inline void scatterwalk_done_dst(struct scatter_walk *walk, + void *vaddr, unsigned int nbytes) +{ + scatterwalk_unmap(vaddr); + /* + * Explicitly check ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE instead of just + * relying on flush_dcache_page() being a no-op when not implemented, + * since otherwise the BUG_ON in sg_page() does not get optimized out. + */ + if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE) + flush_dcache_page(scatterwalk_page(walk)); + scatterwalk_advance(walk, nbytes); +} + void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes); void scatterwalk_copychunks(void *buf, struct scatter_walk *walk, size_t nbytes, int out); From patchwork Mon Dec 30 00:14:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922921 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECC7F19D8A3; Mon, 30 Dec 2024 00:16:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; cv=none; b=n5GWdFZUxVjMMwKZHcdVDDqYn4czffmMuhQM+M7KgQtpIgGoRTlxeHIYx5iX9CW0rHLsciDjRgvQ3RxpAMF1aTv/D+Sb+l0cAlDnaEkuoRnsA0VBLnw4OZ129ArGh47wCZ8hTFo66YRkPOeqGSkpAL78OAP8pWxQ5m83hXuWNe4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; c=relaxed/simple; bh=lWRQiUFYe2vQoQj3axJpxeip5NuLznrPRnr+KRPRjGQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YUpH7A264XNlR8MA7dDBLUyB1SXLBGowAQy+yP2fJWsOAtB7RzfyikagLk6p37NX4EZ53l7qOdPvymP82GuVBl2QUMQQi09BihSDi1r+PfiDhDgLCJQyz+N29d0HZQjmpIP/gxKOyGT99oxDdR7vOKvmfXx17FIYdDNAui/0JgE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HN3jGOPl; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HN3jGOPl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 72157C4CED4; Mon, 30 Dec 2024 00:16:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517763; bh=lWRQiUFYe2vQoQj3axJpxeip5NuLznrPRnr+KRPRjGQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HN3jGOPlJ/NCbL/SElFId+z7QwSQKWcfJvmSbQRczmZLUhAPg4lKgF6qs4ZfAchj+ QxOXj/ptURI6s5xbNFI7P7Hmtx57C5wj0dgZOHRc3DOm2MCrcJht1TNqBHx00JNXct huT1kYYDIqQscFvgTvXp+pxJGytlFCsqiNJsfb3ZaLBYj6N/+BKo64aIjPYSIjf3x6 eA//STf+IqeX7T4Qpb4su767HiP+ft/gHMb8oyb8yqP1qHwVM1M9Q5skgs479Cb+/R sfn0UmARj6Ukie9SWo7FFNWgP0OqZV6Cvzp44MAf4TxfNtmosthoiKs7I86wMc//dR nmz27sytmV0bA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 14/29] crypto: scatterwalk - add new functions for copying data Date: Sun, 29 Dec 2024 16:14:03 -0800 Message-ID: <20241230001418.74739-15-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Add memcpy_from_sglist() and memcpy_to_sglist() which are more readable versions of scatterwalk_map_and_copy() with the 'out' argument 0 and 1 respectively. They follow the same argument order as memcpy_from_page() and memcpy_to_page() from . Note that in the case of memcpy_from_sglist(), this also happens to be the same argument order that scatterwalk_map_and_copy() uses. The new code is also faster, mainly because it builds the scatter_walk directly without creating a temporary scatterlist. E.g., a 20% performance improvement is seen for copying the AES-GCM auth tag. Make scatterwalk_map_and_copy() be a wrapper around memcpy_from_sglist() and memcpy_to_sglist(). Callers of scatterwalk_map_and_copy() should be updated to call memcpy_from_sglist() or memcpy_to_sglist() directly, but there are a lot of them so they aren't all being updated right away. Also add functions memcpy_from_scatterwalk() and memcpy_to_scatterwalk() which are similar but operate on a scatter_walk instead of a scatterlist. These will replace scatterwalk_copychunks() with the 'out' argument 0 and 1 respectively. Their behavior differs slightly from scatterwalk_copychunks() in that they automatically take care of flushing the dcache when needed, making them easier to use. scatterwalk_copychunks() itself is left unchanged for now. It will be removed after its callers are updated to use other functions instead. Signed-off-by: Eric Biggers --- crypto/scatterwalk.c | 59 ++++++++++++++++++++++++++++++------ include/crypto/scatterwalk.h | 24 +++++++++++++-- 2 files changed, 72 insertions(+), 11 deletions(-) diff --git a/crypto/scatterwalk.c b/crypto/scatterwalk.c index af436ad02e3f..2e7a532152d6 100644 --- a/crypto/scatterwalk.c +++ b/crypto/scatterwalk.c @@ -65,26 +65,67 @@ void scatterwalk_copychunks(void *buf, struct scatter_walk *walk, scatterwalk_pagedone(walk, out & 1, 1); } } EXPORT_SYMBOL_GPL(scatterwalk_copychunks); -void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg, - unsigned int start, unsigned int nbytes, int out) +inline void memcpy_from_scatterwalk(void *buf, struct scatter_walk *walk, + unsigned int nbytes) +{ + do { + const void *src_addr; + unsigned int to_copy; + + src_addr = scatterwalk_next(walk, nbytes, &to_copy); + memcpy(buf, src_addr, to_copy); + scatterwalk_done_src(walk, src_addr, to_copy); + buf += to_copy; + nbytes -= to_copy; + } while (nbytes); +} +EXPORT_SYMBOL_GPL(memcpy_from_scatterwalk); + +inline void memcpy_to_scatterwalk(struct scatter_walk *walk, const void *buf, + unsigned int nbytes) +{ + do { + void *dst_addr; + unsigned int to_copy; + + dst_addr = scatterwalk_next(walk, nbytes, &to_copy); + memcpy(dst_addr, buf, to_copy); + scatterwalk_done_dst(walk, dst_addr, to_copy); + buf += to_copy; + nbytes -= to_copy; + } while (nbytes); +} +EXPORT_SYMBOL_GPL(memcpy_to_scatterwalk); + +void memcpy_from_sglist(void *buf, struct scatterlist *sg, + unsigned int start, unsigned int nbytes) { struct scatter_walk walk; - struct scatterlist tmp[2]; - if (!nbytes) + if (unlikely(nbytes == 0)) /* in case sg == NULL */ return; - sg = scatterwalk_ffwd(tmp, sg, start); + scatterwalk_start_at_pos(&walk, sg, start); + memcpy_from_scatterwalk(buf, &walk, nbytes); +} +EXPORT_SYMBOL_GPL(memcpy_from_sglist); + +void memcpy_to_sglist(struct scatterlist *sg, unsigned int start, + const void *buf, unsigned int nbytes) +{ + struct scatter_walk walk; + + if (unlikely(nbytes == 0)) /* in case sg == NULL */ + return; - scatterwalk_start(&walk, sg); - scatterwalk_copychunks(buf, &walk, nbytes, out); - scatterwalk_done(&walk, out, 0); + scatterwalk_start_at_pos(&walk, sg, start); + memcpy_to_scatterwalk(&walk, buf, nbytes); } -EXPORT_SYMBOL_GPL(scatterwalk_map_and_copy); +EXPORT_SYMBOL_GPL(memcpy_to_sglist); struct scatterlist *scatterwalk_ffwd(struct scatterlist dst[2], struct scatterlist *src, unsigned int len) { diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index 8e83c43016c9..1689ecd7ddaf 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -168,12 +168,32 @@ static inline void scatterwalk_done_dst(struct scatter_walk *walk, void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes); void scatterwalk_copychunks(void *buf, struct scatter_walk *walk, size_t nbytes, int out); -void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg, - unsigned int start, unsigned int nbytes, int out); +void memcpy_from_scatterwalk(void *buf, struct scatter_walk *walk, + unsigned int nbytes); + +void memcpy_to_scatterwalk(struct scatter_walk *walk, const void *buf, + unsigned int nbytes); + +void memcpy_from_sglist(void *buf, struct scatterlist *sg, + unsigned int start, unsigned int nbytes); + +void memcpy_to_sglist(struct scatterlist *sg, unsigned int start, + const void *buf, unsigned int nbytes); + +/* In new code, please use memcpy_{from,to}_sglist() directly instead. */ +static inline void scatterwalk_map_and_copy(void *buf, struct scatterlist *sg, + unsigned int start, + unsigned int nbytes, int out) +{ + if (out) + memcpy_to_sglist(sg, start, buf, nbytes); + else + memcpy_from_sglist(buf, sg, start, nbytes); +} struct scatterlist *scatterwalk_ffwd(struct scatterlist dst[2], struct scatterlist *src, unsigned int len); From patchwork Mon Dec 30 00:14:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922918 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EC0A19CC34; Mon, 30 Dec 2024 00:16:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517764; cv=none; b=GQrncbUXID7CBrF1ZGVV8sqvYzsEFCRygHNbfhXVhrO7dhfYX03P8naLI4pM3Pn+mFlcY+xdiN0JuAX0bYiQG/EtXqfl2XLUifhFfXkEMYr2Uj/FGqYAQgT8f/Zd71CEB2VqHPGzINaN7fVoMlYHddxlBK4tqJGyXXUcjVuAj1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517764; c=relaxed/simple; bh=g1Sicjo3WPDOTEntvGipxb6vfLBqpP+1OtFMoBb3F/s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Kf/IR9XyqMMA6PJf9fBrHsU/PLPmHDGxwBal1bkjd9kLOyjbtS9ekecB8eZVLGqPSh0yKAHpNHjuPju8TAp481GvdfBGDJFgK0ytYV/0tZiUrVOIqXFI0uUKBs/BGdQCWDs87OQ0J6WHaEFjbeq7SvGuHsynIROjOIl4WW+3I58= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EE7ZUPdE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EE7ZUPdE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07671C4CED7; Mon, 30 Dec 2024 00:16:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517764; bh=g1Sicjo3WPDOTEntvGipxb6vfLBqpP+1OtFMoBb3F/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EE7ZUPdEs6gq4ZJHyxBrpeS0Zbu0Fd/jruAryECxZgzOYDaBtapoVF25BOdhXwAMB TfZ4VjuSnswCJUYhKp1W700mb+gBZAnjn+8OeoTxueTyi8iRuOs/s46tXfKxu5vJq2 c9Hlh0Dj9x4lEfpGYie0kV0Y/yNn0zQuCBOstT2Dt9Yw0Lt/SvR0KLc4D5PcGgXWf5 9O5QEOS2DtiE5mU2z9UU5m1etRpbrDHhujFxwsymU4KjtSxSZBCTWFcvx9WGy2RTkJ dj7SFRZuZNRxINaIdpOWGNJBuzZesaa/1ru1tnAXj6J9AX7HNzvOi2pQxmEgQCJXyE 6FSK0oxlRsaxQ== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Boris Pismenny , Jakub Kicinski , John Fastabend Subject: [PATCH v2 15/29] crypto: scatterwalk - add scatterwalk_get_sglist() Date: Sun, 29 Dec 2024 16:14:04 -0800 Message-ID: <20241230001418.74739-16-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Add a function that creates a scatterlist that represents the remaining data in a walk. This will be used to replace chain_to_walk() in net/tls/tls_device_fallback.c so that it will no longer need to reach into the internals of struct scatter_walk. Cc: Boris Pismenny Cc: Jakub Kicinski Cc: John Fastabend Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. include/crypto/scatterwalk.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index 1689ecd7ddaf..f6262d05a3c7 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -67,10 +67,27 @@ static inline unsigned int scatterwalk_clamp(struct scatter_walk *walk, static inline struct page *scatterwalk_page(struct scatter_walk *walk) { return sg_page(walk->sg) + (walk->offset >> PAGE_SHIFT); } +/* + * Create a scatterlist that represents the remaining data in a walk. Uses + * chaining to reference the original scatterlist, so this uses at most two + * entries in @sg_out regardless of the number of entries in the original list. + * Assumes that sg_init_table() was already done. + */ +static inline void scatterwalk_get_sglist(struct scatter_walk *walk, + struct scatterlist sg_out[2]) +{ + if (walk->offset >= walk->sg->offset + walk->sg->length) + scatterwalk_start(walk, sg_next(walk->sg)); + sg_set_page(sg_out, sg_page(walk->sg), + walk->sg->offset + walk->sg->length - walk->offset, + walk->offset); + scatterwalk_crypto_chain(sg_out, sg_next(walk->sg), 2); +} + static inline void scatterwalk_unmap(void *vaddr) { kunmap_local(vaddr); } From patchwork Mon Dec 30 00:14:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922919 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B39319D06B; Mon, 30 Dec 2024 00:16:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517764; cv=none; b=cUzUcElA+q169eMNJnl9uu2VeqVRsxPYlyAW/qK+/K53HjaATTMJ00Qg6unpPA0wV3PzvF1GsSuyPyh/5yaOTXO3nuxYj/R5JtpLc/semJDnmyAaFzM5YxnV00vtMM+an4jnh9h9RPAJDRBsTbIbQUQ7pLodGp2948xGWq3SRx4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517764; c=relaxed/simple; bh=gTqiJ83GC+y9ntGuuGLBowL4XoaSaAg3tom8EPVefuE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CXTOY+KWfvxDXlbm8uLNemo+cWnH1bZnnkXfs6Yorgtj7cN/XBnE6nvMJ9vhFrQd5704VmO2t/otbUWPd/toZKD75Ub/WKiaYhgQcCqXg7yZDls+KtvPqFvcWG0W7AuDoCJP669uExya0Jiy7NFNvI7v7wKGLEr+87zPCg9nnc4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IUdR1nFG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IUdR1nFG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 66AD3C4CED1; Mon, 30 Dec 2024 00:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517764; bh=gTqiJ83GC+y9ntGuuGLBowL4XoaSaAg3tom8EPVefuE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IUdR1nFG3ngakh3NdimrWkCgN4PdE+m133S7ezsBJQzwfeLSp1idebo1Sj5TQBfuf IS0S1ZOxyqFbq7kjdQvuBkFlk/tByoIvo+qP/b6pAb34m8PQucBmP6hKLRaaknPbbr llhwJtck7fOI/JgdBLjQZzE2SlSF5Y6v2Ma9g95AfDpEfkg/1BxzO6E2HnPDYM4MX9 fXvl0PnvrvJ6m6urLykmjXd8Mb1dza5s4M6gbcjTI4W/T/B3/klzpRKeXm+W1CFTLW FRK6mucYaxwHxT19nG5D6kQzLWdJWpBJqxEvr/jUvDVui9wGzIIh2jMCIhb9t0iREW IvAiOjk1GwqYA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 16/29] crypto: skcipher - use scatterwalk_start_at_pos() Date: Sun, 29 Dec 2024 16:14:05 -0800 Message-ID: <20241230001418.74739-17-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In skcipher_walk_aead_common(), use scatterwalk_start_at_pos() instead of a sequence of scatterwalk_start(), scatterwalk_copychunks(..., 2), and scatterwalk_done(). This is simpler and faster. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 441e1d254d36..7abafe385fd5 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -355,18 +355,12 @@ static int skcipher_walk_aead_common(struct skcipher_walk *walk, walk->flags = 0; if (unlikely(!walk->total)) return 0; - scatterwalk_start(&walk->in, req->src); - scatterwalk_start(&walk->out, req->dst); - - scatterwalk_copychunks(NULL, &walk->in, req->assoclen, 2); - scatterwalk_copychunks(NULL, &walk->out, req->assoclen, 2); - - scatterwalk_done(&walk->in, 0, walk->total); - scatterwalk_done(&walk->out, 0, walk->total); + scatterwalk_start_at_pos(&walk->in, req->src, req->assoclen); + scatterwalk_start_at_pos(&walk->out, req->dst, req->assoclen); walk->blocksize = alg->base.cra_blocksize; walk->stride = alg->chunksize; walk->ivsize = alg->ivsize; walk->alignmask = alg->base.cra_alignmask; From patchwork Mon Dec 30 00:14:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922920 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF9F919D89B; Mon, 30 Dec 2024 00:16:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; cv=none; b=tBYYfSafFiCT6vhQxxjOWoUTEsr3O4MX20nmzC0JBMAzI0qUsk339wKnLMT2q9jDZ6ICLfMwEjIrirdhEMRZZ4ncFsN22UJilHtYJIuOAQxaboN95/AJaqi7Qbui3Lx2FxgL00evqbV3ZeDXHpczaViG+bKko7eZe87ahUZ5Pnw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; c=relaxed/simple; bh=ouuUySUoLhMrrlBcXOe8B+9HrJzjwC7w9DOOQhZHjCM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l3gqEX8Z1ybg0wJYi3uQsWQBaolDAXNh3QeAY9c5NMHOByaVa+1vn9akb3jJyrDxMgbGkfFRf2NaqAQKKezWoio1PrHdeK6Q6E7I7GShpyP0CNHVKyQM/2+dXPckajoz/U5n0Enk95fCkTQxxwpvjMplf/dkltDVjbh37WdJvbU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=b0MaNvjW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="b0MaNvjW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A798AC4CEDC; Mon, 30 Dec 2024 00:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517764; bh=ouuUySUoLhMrrlBcXOe8B+9HrJzjwC7w9DOOQhZHjCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b0MaNvjWDn5EygCKZDuBSrXH9cuut+CXK5MMrnjLuPZPxISLNyN+xGBZj7m5WGgTj gouA3Dy+g+1qJAbmU6Ilf84nvIluzwGuXuQBnEephADIj0iaHc79hZdceRz04abiw0 k9uLxF0HCLe/2Vyh1XR1PWcc9Y6u4u0yDLG2IcAq7nj/BmEFYMN5r6cOokThLkFXpK KWgvlQsEPSVumt2pUKeDwRPdj4GtU3QrjTxdzYqb5vmgpwsKUncU8QGPQOoXP5uIUT LfrLCUF88jsRMubvRfag9dEex2Gd4mNFlaPDLUkB3gNoUTU6eL954Bpx6Cf3cs8AEj e2D9+s2b9P/tA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 17/29] crypto: aegis - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:06 -0800 Message-ID: <20241230001418.74739-18-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Use scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(), and use scatterwalk_done_src() which consolidates scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Signed-off-by: Eric Biggers --- crypto/aegis128-core.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/crypto/aegis128-core.c b/crypto/aegis128-core.c index 6cbff298722b..15d64d836356 100644 --- a/crypto/aegis128-core.c +++ b/crypto/aegis128-core.c @@ -282,14 +282,14 @@ static void crypto_aegis128_process_ad(struct aegis_state *state, union aegis_block buf; unsigned int pos = 0; scatterwalk_start(&walk, sg_src); while (assoclen != 0) { - unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int size; + const u8 *mapped = scatterwalk_next(&walk, assoclen, &size); unsigned int left = size; - void *mapped = scatterwalk_map(&walk); - const u8 *src = (const u8 *)mapped; + const u8 *src = mapped; if (pos + size >= AEGIS_BLOCK_SIZE) { if (pos > 0) { unsigned int fill = AEGIS_BLOCK_SIZE - pos; memcpy(buf.bytes + pos, src, fill); @@ -306,13 +306,11 @@ static void crypto_aegis128_process_ad(struct aegis_state *state, memcpy(buf.bytes + pos, src, left); pos += left; assoclen -= size; - scatterwalk_unmap(mapped); - scatterwalk_advance(&walk, size); - scatterwalk_done(&walk, 0, assoclen); + scatterwalk_done_src(&walk, mapped, size); } if (pos > 0) { memset(buf.bytes + pos, 0, AEGIS_BLOCK_SIZE - pos); crypto_aegis128_update_a(state, &buf, do_simd); From patchwork Mon Dec 30 00:14:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922922 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41C7A19DFAB; Mon, 30 Dec 2024 00:16:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; cv=none; b=W3hAq6ejlKajX2+7pPZxRbbiB9YeFKIlzo+lR+4+hpPOwzQUHZtai1myXd1VuUZbgSAomV/PrYtDPLeWl7F0mbqYND9gDedwMLZdnQMbBxFT9IGRh9hHy4juP2hfEP4Ang8oHrWzddvfrnPOx24Cp9XcL2w8WJ+0yextXGPiEPY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; c=relaxed/simple; bh=flgH9kGUEUUdU5GGIoLcecbOKpZ/9P71he8YGiVnKlw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MvHrZ9bDUyAYh74MWEuC2k9pd/vJm2Rl+7CWsDEWmXVP9yUVpiplzV2DFh/cpEl8Gx7xf7410LeBtP2bs3fdFknnQh0Mbbl7g3HlbDdyNHLJXMvOXZiYKIXug7G7LVLQSoTZ2o6p92Jh6qSXYG+1mqSnYEG4qH7wa/XGxKBaAiQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JtFliWsz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JtFliWsz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EA647C4AF0F; Mon, 30 Dec 2024 00:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517765; bh=flgH9kGUEUUdU5GGIoLcecbOKpZ/9P71he8YGiVnKlw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JtFliWsz1MQWt1rERIsI2ACTKt7OkpSZV/Ws3CTI/hJqzB5D+kTwjNVynffI78p3L mX2IDLsguSW5936FvY2DfgWFk8ehPLyx/UrUFmVm84FPMdFaRxLcqUgQzL0RvsW4YE SLfK1NxZgtKxTVFLNOmrcnDk20B3fhaa8wEPh+svrQSRFfSZMFA16cuRzDvgORuNji H1egmLKAUNtkE8V08WKXPqgzr8GNP0dNiqBN5ClYdolActcKsV0Oy7wUdwba+bvndg QNahR/v56j+mt67qLDUHeFxjTsHhfKpCaZQijTVdy9m6nyc4Coi7PrDG1gBo5N1nAJ oTtmS9pWaxRsA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 18/29] crypto: arm/ghash - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:07 -0800 Message-ID: <20241230001418.74739-19-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Use scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(), and use scatterwalk_done_src() which consolidates scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Remove unnecessary code that seemed to be intended to advance to the next sg entry, which is already handled by the scatterwalk functions. Signed-off-by: Eric Biggers --- arch/arm/crypto/ghash-ce-glue.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/arch/arm/crypto/ghash-ce-glue.c b/arch/arm/crypto/ghash-ce-glue.c index 3af997082534..9613ffed84f9 100644 --- a/arch/arm/crypto/ghash-ce-glue.c +++ b/arch/arm/crypto/ghash-ce-glue.c @@ -457,30 +457,23 @@ static void gcm_calculate_auth_mac(struct aead_request *req, u64 dg[], u32 len) int buf_count = 0; scatterwalk_start(&walk, req->src); do { - u32 n = scatterwalk_clamp(&walk, len); - u8 *p; + unsigned int n; + const u8 *p; - if (!n) { - scatterwalk_start(&walk, sg_next(walk.sg)); - n = scatterwalk_clamp(&walk, len); - } - - p = scatterwalk_map(&walk); + p = scatterwalk_next(&walk, len, &n); gcm_update_mac(dg, p, n, buf, &buf_count, ctx); - scatterwalk_unmap(p); + scatterwalk_done_src(&walk, p, n); if (unlikely(len / SZ_4K > (len - n) / SZ_4K)) { kernel_neon_end(); kernel_neon_begin(); } len -= n; - scatterwalk_advance(&walk, n); - scatterwalk_done(&walk, 0, len); } while (len); if (buf_count) { memset(&buf[buf_count], 0, GHASH_BLOCK_SIZE - buf_count); pmull_ghash_update_p64(1, dg, buf, ctx->h, NULL); From patchwork Mon Dec 30 00:14:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922923 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCF4419F422; Mon, 30 Dec 2024 00:16:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; cv=none; b=JIPtTacoLyf8lipcwW85eeG6aetixjyVwKnF4yOMDpu+0cizA59UkSYNM0W9DtWHpcugvJ+lZwdoX3880RhhrIV4GoniQ8owS7ELa3g7eNV5CDEjOaJp30mS/XtQZuPoJo7aMQf5k7xVHxn1WQ1oGDh8l8ZwmNzBJ+Zkqp36yqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517765; c=relaxed/simple; bh=mSW8mWLKgO+z9pqafkyRDJs4ecTLipklsIRH/V4AmPg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I2sWBFXsCgGgK10yyAzxxFP5hvf0YQrOgHB/orut4Ikd5esV8Z6iWCfP4wm/oUU4ZdMLt5tyLa4+6ppancPfquJhSbXysnnYKJzPcoLYt+8iJsJpirbl4F4icApSmUpW7P+SjlIDUx5qs1uCvHJWr+RvuG4m1hhrugourSB1Rzc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mGwfZiGv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mGwfZiGv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 47A76C4CED1; Mon, 30 Dec 2024 00:16:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517765; bh=mSW8mWLKgO+z9pqafkyRDJs4ecTLipklsIRH/V4AmPg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mGwfZiGv8SNZTPwDcSbed7NAZKzMTdKgsQkuDSFs4BRuMCw26DdqqhnNJq2phNgSb HAzQuxY9y/wG2AwkYD64KKxZX6ed4GceDye68R/wAQnLjpqVCX0wKMsUCH3Yc5Vdfb evCztyi1a9vNWQG/BOXLcIRsceXbDE+EA05+WpRFdzKXP1n3v8WEbXx0SXdP+OuQ8G QlLMUA1jI7T8PAidXNpf4Fs1BpgK8nyJKrZgaLFJFIFZgLN62oHUrIVZBAXTmrXQ7b 1SmgvwQpWi/QWdxLmAExfxpSzZNFc/nwm+UDnHLaNNIbt7W2FIMLFZwf2uz+jHKrLD J6TJX1jpsVDcg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 19/29] crypto: arm64 - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:08 -0800 Message-ID: <20241230001418.74739-20-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Use scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(), and use scatterwalk_done_src() which consolidates scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Remove unnecessary code that seemed to be intended to advance to the next sg entry, which is already handled by the scatterwalk functions. Adjust variable naming slightly to keep things consistent. Signed-off-by: Eric Biggers --- arch/arm64/crypto/aes-ce-ccm-glue.c | 17 ++++------------ arch/arm64/crypto/ghash-ce-glue.c | 16 ++++----------- arch/arm64/crypto/sm4-ce-ccm-glue.c | 27 ++++++++++--------------- arch/arm64/crypto/sm4-ce-gcm-glue.c | 31 ++++++++++++----------------- 4 files changed, 32 insertions(+), 59 deletions(-) diff --git a/arch/arm64/crypto/aes-ce-ccm-glue.c b/arch/arm64/crypto/aes-ce-ccm-glue.c index a2b5d6f20f4d..1c29546983bf 100644 --- a/arch/arm64/crypto/aes-ce-ccm-glue.c +++ b/arch/arm64/crypto/aes-ce-ccm-glue.c @@ -154,27 +154,18 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) macp = ce_aes_ccm_auth_data(mac, (u8 *)<ag, ltag.len, macp, ctx->key_enc, num_rounds(ctx)); scatterwalk_start(&walk, req->src); do { - u32 n = scatterwalk_clamp(&walk, len); - u8 *p; - - if (!n) { - scatterwalk_start(&walk, sg_next(walk.sg)); - n = scatterwalk_clamp(&walk, len); - } - p = scatterwalk_map(&walk); + unsigned int n; + const u8 *p; + p = scatterwalk_next(&walk, len, &n); macp = ce_aes_ccm_auth_data(mac, p, n, macp, ctx->key_enc, num_rounds(ctx)); - + scatterwalk_done_src(&walk, p, n); len -= n; - - scatterwalk_unmap(p); - scatterwalk_advance(&walk, n); - scatterwalk_done(&walk, 0, len); } while (len); } static int ccm_encrypt(struct aead_request *req) { diff --git a/arch/arm64/crypto/ghash-ce-glue.c b/arch/arm64/crypto/ghash-ce-glue.c index da7b7ec1a664..69d4fb78c30d 100644 --- a/arch/arm64/crypto/ghash-ce-glue.c +++ b/arch/arm64/crypto/ghash-ce-glue.c @@ -306,25 +306,17 @@ static void gcm_calculate_auth_mac(struct aead_request *req, u64 dg[], u32 len) int buf_count = 0; scatterwalk_start(&walk, req->src); do { - u32 n = scatterwalk_clamp(&walk, len); - u8 *p; - - if (!n) { - scatterwalk_start(&walk, sg_next(walk.sg)); - n = scatterwalk_clamp(&walk, len); - } - p = scatterwalk_map(&walk); + unsigned int n; + const u8 *p; + p = scatterwalk_next(&walk, len, &n); gcm_update_mac(dg, p, n, buf, &buf_count, ctx); + scatterwalk_done_src(&walk, p, n); len -= n; - - scatterwalk_unmap(p); - scatterwalk_advance(&walk, n); - scatterwalk_done(&walk, 0, len); } while (len); if (buf_count) { memset(&buf[buf_count], 0, GHASH_BLOCK_SIZE - buf_count); ghash_do_simd_update(1, dg, buf, &ctx->ghash_key, NULL, diff --git a/arch/arm64/crypto/sm4-ce-ccm-glue.c b/arch/arm64/crypto/sm4-ce-ccm-glue.c index 5e7e17bbec81..119f86eb7cc9 100644 --- a/arch/arm64/crypto/sm4-ce-ccm-glue.c +++ b/arch/arm64/crypto/sm4-ce-ccm-glue.c @@ -110,21 +110,16 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) crypto_xor(mac, (const u8 *)&aadlen, len); scatterwalk_start(&walk, req->src); do { - u32 n = scatterwalk_clamp(&walk, assoclen); - u8 *p, *ptr; + unsigned int n, orig_n; + const u8 *p, *orig_p; - if (!n) { - scatterwalk_start(&walk, sg_next(walk.sg)); - n = scatterwalk_clamp(&walk, assoclen); - } - - p = ptr = scatterwalk_map(&walk); - assoclen -= n; - scatterwalk_advance(&walk, n); + orig_p = scatterwalk_next(&walk, assoclen, &orig_n); + p = orig_p; + n = orig_n; while (n > 0) { unsigned int l, nblocks; if (len == SM4_BLOCK_SIZE) { @@ -134,30 +129,30 @@ static void ccm_calculate_auth_mac(struct aead_request *req, u8 mac[]) len = 0; } else { nblocks = n / SM4_BLOCK_SIZE; sm4_ce_cbcmac_update(ctx->rkey_enc, - mac, ptr, nblocks); + mac, p, nblocks); - ptr += nblocks * SM4_BLOCK_SIZE; + p += nblocks * SM4_BLOCK_SIZE; n %= SM4_BLOCK_SIZE; continue; } } l = min(n, SM4_BLOCK_SIZE - len); if (l) { - crypto_xor(mac + len, ptr, l); + crypto_xor(mac + len, p, l); len += l; - ptr += l; + p += l; n -= l; } } - scatterwalk_unmap(p); - scatterwalk_done(&walk, 0, assoclen); + scatterwalk_done_src(&walk, orig_p, orig_n); + assoclen -= orig_n; } while (assoclen); } static int ccm_crypt(struct aead_request *req, struct skcipher_walk *walk, u32 *rkey_enc, u8 mac[], diff --git a/arch/arm64/crypto/sm4-ce-gcm-glue.c b/arch/arm64/crypto/sm4-ce-gcm-glue.c index 73bfb6972d3a..2e27d7752d4f 100644 --- a/arch/arm64/crypto/sm4-ce-gcm-glue.c +++ b/arch/arm64/crypto/sm4-ce-gcm-glue.c @@ -80,53 +80,48 @@ static void gcm_calculate_auth_mac(struct aead_request *req, u8 ghash[]) unsigned int buflen = 0; scatterwalk_start(&walk, req->src); do { - u32 n = scatterwalk_clamp(&walk, assoclen); - u8 *p, *ptr; + unsigned int n, orig_n; + const u8 *p, *orig_p; - if (!n) { - scatterwalk_start(&walk, sg_next(walk.sg)); - n = scatterwalk_clamp(&walk, assoclen); - } - - p = ptr = scatterwalk_map(&walk); - assoclen -= n; - scatterwalk_advance(&walk, n); + orig_p = scatterwalk_next(&walk, assoclen, &orig_n); + p = orig_p; + n = orig_n; if (n + buflen < GHASH_BLOCK_SIZE) { - memcpy(&buffer[buflen], ptr, n); + memcpy(&buffer[buflen], p, n); buflen += n; } else { unsigned int nblocks; if (buflen) { unsigned int l = GHASH_BLOCK_SIZE - buflen; - memcpy(&buffer[buflen], ptr, l); - ptr += l; + memcpy(&buffer[buflen], p, l); + p += l; n -= l; pmull_ghash_update(ctx->ghash_table, ghash, buffer, 1); } nblocks = n / GHASH_BLOCK_SIZE; if (nblocks) { pmull_ghash_update(ctx->ghash_table, ghash, - ptr, nblocks); - ptr += nblocks * GHASH_BLOCK_SIZE; + p, nblocks); + p += nblocks * GHASH_BLOCK_SIZE; } buflen = n % GHASH_BLOCK_SIZE; if (buflen) - memcpy(&buffer[0], ptr, buflen); + memcpy(&buffer[0], p, buflen); } - scatterwalk_unmap(p); - scatterwalk_done(&walk, 0, assoclen); + scatterwalk_done_src(&walk, orig_p, orig_n); + assoclen -= orig_n; } while (assoclen); /* padding with '0' */ if (buflen) { memset(&buffer[buflen], 0, GHASH_BLOCK_SIZE - buflen); From patchwork Mon Dec 30 00:14:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922926 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 394A11A23A7; Mon, 30 Dec 2024 00:16:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; cv=none; b=hwe7I+lomIVlwBMmbYG7aHymZSge5ZUb6jIIOl4g0FKqLdWii3mMlTxuXNkOLlN28J9b0/MVYvBCHRcOFeySpbEVFUf+0XQwJI3zu5LDmdyddDpInbcvyzhtp3Jev9Zmv5G6rStlHu375EX7S840Lb0V0orL9U9iuAr5kN3o0BM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; c=relaxed/simple; bh=FDzNrJCoVdAmqgX/rTeMmUPgE6IOzOA1GEP6JN6T8HU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d5w0+FxE6JRjAaCyrpn27laACkykDuGdUwdGYNy09oUwnjtWOOb7kjxRZmyJEFhvD/yVYEZXy3RVSx5fr3Mx4t71K6pXpzBajRQsQ0ppXpOnBG06OaOc3pOTmsbYIEuswt4iDzPPjOX/HBKTEKo9M4RxV8vx+jBI2E9yjXNsC7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=T/UvuFwR; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="T/UvuFwR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 89B7FC4CEDD; Mon, 30 Dec 2024 00:16:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517765; bh=FDzNrJCoVdAmqgX/rTeMmUPgE6IOzOA1GEP6JN6T8HU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T/UvuFwRWdKLULyZldooFldhH+tu37W6X22cmGB8QkaaVYdGUqE2vErdBb3LOEZO/ gZ7fJYbPzIF/smtDRzJE7n1NbbUqOI921yjPfm8iNWjQRA2hPN9l65SIy7+fvftSDT JempkVhxX8ZqgZ+D3JV9BMUUqPPu3rnxwgkJQ3idcCGc7mAa6af3GogSRr6269qD0F 7gFNFvUU5uEllXPOgUP8xIIqaJeJLfa+6v+SnZV81TKv/RQrSQ/RYCAV5wwt2i3OvS wpvZamcvU+Wez3NStN+LB8Q5H7TF9CqVwwuPQ6d3G+FC6cijFylUrNdOxJvuYyzWOn gx68h5vs9jGAg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Christophe Leroy , Madhavan Srinivasan , Michael Ellerman , Naveen N Rao , Nicholas Piggin , linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 20/29] crypto: nx - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:09 -0800 Message-ID: <20241230001418.74739-21-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers - In nx_walk_and_build(), use scatterwalk_start_at_pos() instead of a more complex way to achieve the same result. - Also in nx_walk_and_build(), use the new functions scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(), and use scatterwalk_done_src() which consolidates scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Remove unnecessary code that seemed to be intended to advance to the next sg entry, which is already handled by the scatterwalk functions. Note that nx_walk_and_build() does not actually read or write the mapped virtual address, and thus it is misusing the scatter_walk API. It really should just access the scatterlist directly. This patch does not try to address this existing issue. - In nx_gca(), use memcpy_from_sglist() instead of a more complex way to achieve the same result. - In various functions, replace calls to scatterwalk_map_and_copy() with memcpy_from_sglist() or memcpy_to_sglist() as appropriate. Note that this eliminates the confusing 'out' argument (which this driver had tried to work around by defining the missing constants for it...) Cc: Christophe Leroy Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Naveen N Rao Cc: Nicholas Piggin Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. drivers/crypto/nx/nx-aes-ccm.c | 16 ++++++---------- drivers/crypto/nx/nx-aes-gcm.c | 17 ++++++----------- drivers/crypto/nx/nx.c | 31 +++++-------------------------- drivers/crypto/nx/nx.h | 3 --- 4 files changed, 17 insertions(+), 50 deletions(-) diff --git a/drivers/crypto/nx/nx-aes-ccm.c b/drivers/crypto/nx/nx-aes-ccm.c index c843f4c6f684..56a0b3a67c33 100644 --- a/drivers/crypto/nx/nx-aes-ccm.c +++ b/drivers/crypto/nx/nx-aes-ccm.c @@ -215,17 +215,15 @@ static int generate_pat(u8 *iv, */ if (b1) { memset(b1, 0, 16); if (assoclen <= 65280) { *(u16 *)b1 = assoclen; - scatterwalk_map_and_copy(b1 + 2, req->src, 0, - iauth_len, SCATTERWALK_FROM_SG); + memcpy_from_sglist(b1 + 2, req->src, 0, iauth_len); } else { *(u16 *)b1 = (u16)(0xfffe); *(u32 *)&b1[2] = assoclen; - scatterwalk_map_and_copy(b1 + 6, req->src, 0, - iauth_len, SCATTERWALK_FROM_SG); + memcpy_from_sglist(b1 + 6, req->src, 0, iauth_len); } } /* now copy any remaining AAD to scatterlist and call nx... */ if (!assoclen) { @@ -339,13 +337,12 @@ static int ccm_nx_decrypt(struct aead_request *req, spin_lock_irqsave(&nx_ctx->lock, irq_flags); nbytes -= authsize; /* copy out the auth tag to compare with later */ - scatterwalk_map_and_copy(priv->oauth_tag, - req->src, nbytes + req->assoclen, authsize, - SCATTERWALK_FROM_SG); + memcpy_from_sglist(priv->oauth_tag, req->src, nbytes + req->assoclen, + authsize); rc = generate_pat(iv, req, nx_ctx, authsize, nbytes, assoclen, csbcpb->cpb.aes_ccm.in_pat_or_b0); if (rc) goto out; @@ -463,13 +460,12 @@ static int ccm_nx_encrypt(struct aead_request *req, processed += to_process; } while (processed < nbytes); /* copy out the auth tag */ - scatterwalk_map_and_copy(csbcpb->cpb.aes_ccm.out_pat_or_mac, - req->dst, nbytes + req->assoclen, authsize, - SCATTERWALK_TO_SG); + memcpy_to_sglist(req->dst, nbytes + req->assoclen, + csbcpb->cpb.aes_ccm.out_pat_or_mac, authsize); out: spin_unlock_irqrestore(&nx_ctx->lock, irq_flags); return rc; } diff --git a/drivers/crypto/nx/nx-aes-gcm.c b/drivers/crypto/nx/nx-aes-gcm.c index 4a796318b430..b7fe2de96d96 100644 --- a/drivers/crypto/nx/nx-aes-gcm.c +++ b/drivers/crypto/nx/nx-aes-gcm.c @@ -101,20 +101,17 @@ static int nx_gca(struct nx_crypto_ctx *nx_ctx, u8 *out, unsigned int assoclen) { int rc; struct nx_csbcpb *csbcpb_aead = nx_ctx->csbcpb_aead; - struct scatter_walk walk; struct nx_sg *nx_sg = nx_ctx->in_sg; unsigned int nbytes = assoclen; unsigned int processed = 0, to_process; unsigned int max_sg_len; if (nbytes <= AES_BLOCK_SIZE) { - scatterwalk_start(&walk, req->src); - scatterwalk_copychunks(out, &walk, nbytes, SCATTERWALK_FROM_SG); - scatterwalk_done(&walk, SCATTERWALK_FROM_SG, 0); + memcpy_from_sglist(out, req->src, 0, nbytes); return 0; } NX_CPB_FDM(csbcpb_aead) &= ~NX_FDM_CONTINUATION; @@ -389,23 +386,21 @@ static int gcm_aes_nx_crypt(struct aead_request *req, int enc, } while (processed < nbytes); mac: if (enc) { /* copy out the auth tag */ - scatterwalk_map_and_copy( - csbcpb->cpb.aes_gcm.out_pat_or_mac, + memcpy_to_sglist( req->dst, req->assoclen + nbytes, - crypto_aead_authsize(crypto_aead_reqtfm(req)), - SCATTERWALK_TO_SG); + csbcpb->cpb.aes_gcm.out_pat_or_mac, + crypto_aead_authsize(crypto_aead_reqtfm(req))); } else { u8 *itag = nx_ctx->priv.gcm.iauth_tag; u8 *otag = csbcpb->cpb.aes_gcm.out_pat_or_mac; - scatterwalk_map_and_copy( + memcpy_from_sglist( itag, req->src, req->assoclen + nbytes, - crypto_aead_authsize(crypto_aead_reqtfm(req)), - SCATTERWALK_FROM_SG); + crypto_aead_authsize(crypto_aead_reqtfm(req))); rc = crypto_memneq(itag, otag, crypto_aead_authsize(crypto_aead_reqtfm(req))) ? -EBADMSG : 0; } out: diff --git a/drivers/crypto/nx/nx.c b/drivers/crypto/nx/nx.c index 010e87d9da36..dd95e5361d88 100644 --- a/drivers/crypto/nx/nx.c +++ b/drivers/crypto/nx/nx.c @@ -151,44 +151,23 @@ struct nx_sg *nx_walk_and_build(struct nx_sg *nx_dst, unsigned int start, unsigned int *src_len) { struct scatter_walk walk; struct nx_sg *nx_sg = nx_dst; - unsigned int n, offset = 0, len = *src_len; + unsigned int n, len = *src_len; char *dst; /* we need to fast forward through @start bytes first */ - for (;;) { - scatterwalk_start(&walk, sg_src); - - if (start < offset + sg_src->length) - break; - - offset += sg_src->length; - sg_src = sg_next(sg_src); - } - - /* start - offset is the number of bytes to advance in the scatterlist - * element we're currently looking at */ - scatterwalk_advance(&walk, start - offset); + scatterwalk_start_at_pos(&walk, sg_src, start); while (len && (nx_sg - nx_dst) < sglen) { - n = scatterwalk_clamp(&walk, len); - if (!n) { - /* In cases where we have scatterlist chain sg_next - * handles with it properly */ - scatterwalk_start(&walk, sg_next(walk.sg)); - n = scatterwalk_clamp(&walk, len); - } - dst = scatterwalk_map(&walk); + dst = scatterwalk_next(&walk, len, &n); nx_sg = nx_build_sg_list(nx_sg, dst, &n, sglen - (nx_sg - nx_dst)); - len -= n; - scatterwalk_unmap(dst); - scatterwalk_advance(&walk, n); - scatterwalk_done(&walk, SCATTERWALK_FROM_SG, len); + scatterwalk_done_src(&walk, dst, n); + len -= n; } /* update to_process */ *src_len -= len; /* return the moved destination pointer */ diff --git a/drivers/crypto/nx/nx.h b/drivers/crypto/nx/nx.h index 2697baebb6a3..e1b4b6927bec 100644 --- a/drivers/crypto/nx/nx.h +++ b/drivers/crypto/nx/nx.h @@ -187,9 +187,6 @@ extern struct shash_alg nx_shash_aes_xcbc_alg; extern struct shash_alg nx_shash_sha512_alg; extern struct shash_alg nx_shash_sha256_alg; extern struct nx_crypto_driver nx_driver; -#define SCATTERWALK_TO_SG 1 -#define SCATTERWALK_FROM_SG 0 - #endif From patchwork Mon Dec 30 00:14:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922924 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 929291A0B15; Mon, 30 Dec 2024 00:16:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517766; cv=none; b=dccLli6OUecuJ2L8AgjrFr2O1df9+0p5mRvtamYIFgu/5AJCAEm4wsaEqT+LGinEZXYZ1P+6xnrw/do5MIj5g9qIaM6iSafLWyAfbmMDWx9pVknsMQlT5pAXXeXUlmsFJl4UWmvndA89YncCzPS8QbhW4FEVC3OCl7cv+bNLaZk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517766; c=relaxed/simple; bh=H979TZHDgKLaUbglwnxL0UQbpoLtqMQtPXY6pBrdkHY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K1XM3f5rYBTkfR5oKW98i9T6hWb3wWKTIPJFsn1sYJDdpTOsrrX+q6XOVFur8lWLlhuHxSC/CamzgkY3iipL+/mKdqFjAg5Yobbr/Opc5bPmqViUMsLBv8unpZEJ3pJqqMy0uJu+ymgqpShsdF+n5lhLswVqLd1rhbkfnIdbOFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OVv7ieP8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OVv7ieP8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2CCCC4CEDC; Mon, 30 Dec 2024 00:16:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517766; bh=H979TZHDgKLaUbglwnxL0UQbpoLtqMQtPXY6pBrdkHY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OVv7ieP8SjYk/HjaXd0tHHAvWhYyEeizW7IRUKiiA8xqB5Yq63CHK6thVfRetcYdY XrzADxDGMfIio8KG4x38IIh558UFNmRQ95drvi+0u275GXhE/ZJT3YiTenqo2BB9JC LLGpYM7FxTJ/b8vgYPOfCIIKMZZ9IGbVCuWwV9AXSaIEJvlrfQdUEXToZD2+TQ5zA0 uU8xcsI/+F8GRsBGYCAjCT5PpF456KOLQn7/0vH8LjIm08dA0oEbZpEzrYSmIcshQ0 d3ObtqAq5VVH6+bHquDAr3RJevWu8J+5hWPWQF6dqDblo5zHstQJQrO0TMQZUtiS7v wlKCqJxFSDOQA== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Harald Freudenberger , Holger Dengler , linux-s390@vger.kernel.org Subject: [PATCH v2 21/29] crypto: s390/aes-gcm - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:10 -0800 Message-ID: <20241230001418.74739-22-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Use scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(). Use scatterwalk_done_src() and scatterwalk_done_dst() which consolidate scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Besides the new functions being a bit easier to use, this is necessary because scatterwalk_done() is planned to be removed. Cc: Harald Freudenberger Cc: Holger Dengler Cc: linux-s390@vger.kernel.org Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. arch/s390/crypto/aes_s390.c | 33 +++++++++++++-------------------- 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/arch/s390/crypto/aes_s390.c b/arch/s390/crypto/aes_s390.c index 9c46b1b630b1..7fd303df05ab 100644 --- a/arch/s390/crypto/aes_s390.c +++ b/arch/s390/crypto/aes_s390.c @@ -785,32 +785,25 @@ static void gcm_walk_start(struct gcm_sg_walk *gw, struct scatterlist *sg, scatterwalk_start(&gw->walk, sg); } static inline unsigned int _gcm_sg_clamp_and_map(struct gcm_sg_walk *gw) { - struct scatterlist *nextsg; - - gw->walk_bytes = scatterwalk_clamp(&gw->walk, gw->walk_bytes_remain); - while (!gw->walk_bytes) { - nextsg = sg_next(gw->walk.sg); - if (!nextsg) - return 0; - scatterwalk_start(&gw->walk, nextsg); - gw->walk_bytes = scatterwalk_clamp(&gw->walk, - gw->walk_bytes_remain); - } - gw->walk_ptr = scatterwalk_map(&gw->walk); + if (gw->walk_bytes_remain == 0) + return 0; + gw->walk_ptr = scatterwalk_next(&gw->walk, gw->walk_bytes_remain, + &gw->walk_bytes); return gw->walk_bytes; } static inline void _gcm_sg_unmap_and_advance(struct gcm_sg_walk *gw, - unsigned int nbytes) + unsigned int nbytes, bool out) { gw->walk_bytes_remain -= nbytes; - scatterwalk_unmap(gw->walk_ptr); - scatterwalk_advance(&gw->walk, nbytes); - scatterwalk_done(&gw->walk, 0, gw->walk_bytes_remain); + if (out) + scatterwalk_done_dst(&gw->walk, gw->walk_ptr, nbytes); + else + scatterwalk_done_src(&gw->walk, gw->walk_ptr, nbytes); gw->walk_ptr = NULL; } static int gcm_in_walk_go(struct gcm_sg_walk *gw, unsigned int minbytesneeded) { @@ -842,11 +835,11 @@ static int gcm_in_walk_go(struct gcm_sg_walk *gw, unsigned int minbytesneeded) while (1) { n = min(gw->walk_bytes, AES_BLOCK_SIZE - gw->buf_bytes); memcpy(gw->buf + gw->buf_bytes, gw->walk_ptr, n); gw->buf_bytes += n; - _gcm_sg_unmap_and_advance(gw, n); + _gcm_sg_unmap_and_advance(gw, n, false); if (gw->buf_bytes >= minbytesneeded) { gw->ptr = gw->buf; gw->nbytes = gw->buf_bytes; goto out; } @@ -902,11 +895,11 @@ static int gcm_in_walk_done(struct gcm_sg_walk *gw, unsigned int bytesdone) memmove(gw->buf, gw->buf + bytesdone, n); gw->buf_bytes = n; } else gw->buf_bytes = 0; } else - _gcm_sg_unmap_and_advance(gw, bytesdone); + _gcm_sg_unmap_and_advance(gw, bytesdone, false); return bytesdone; } static int gcm_out_walk_done(struct gcm_sg_walk *gw, unsigned int bytesdone) @@ -920,14 +913,14 @@ static int gcm_out_walk_done(struct gcm_sg_walk *gw, unsigned int bytesdone) for (i = 0; i < bytesdone; i += n) { if (!_gcm_sg_clamp_and_map(gw)) return i; n = min(gw->walk_bytes, bytesdone - i); memcpy(gw->walk_ptr, gw->buf + i, n); - _gcm_sg_unmap_and_advance(gw, n); + _gcm_sg_unmap_and_advance(gw, n, true); } } else - _gcm_sg_unmap_and_advance(gw, bytesdone); + _gcm_sg_unmap_and_advance(gw, bytesdone, true); return bytesdone; } static int gcm_aes_crypt(struct aead_request *req, unsigned int flags) From patchwork Mon Dec 30 00:14:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922925 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 198D51A239A; Mon, 30 Dec 2024 00:16:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; cv=none; b=pNzmV7qeFNWeGwr32QM6tIqJwd5F+ZsFDwf2oOYPx8m+1nHHOvqr28h8PdgCwBITtL8imIKoc4Mc32hZHoimU1qcXtl3xHOpPT1zWJIkTLqdh7xr7sqShssga2jvrqFBIKZj/VzwC37W183OcfrtIOIyNzAyN4OqsCh4uIIyz1E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; c=relaxed/simple; bh=dJNf7Ef27a6NAHmh6+QRwRI/qSPxeV2L1TYNFHLLkzw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mVWt5328LQLau88Aakmx0q+vfGdXdrdkryMnzziVvIncqFfdB9gx2B/MKBVj4DvX9QgY15Qc9chOQIBrfQ6LZTzMjztNhFY46panqLGjoEK9sncTUOLH2GqJKKvN9OXalapP3bTg+dDBvzr7dUbKiW4v5zgb+KQKbC4NE+WIJGE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nZjYwxLN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nZjYwxLN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCD69C4CED7; Mon, 30 Dec 2024 00:16:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517767; bh=dJNf7Ef27a6NAHmh6+QRwRI/qSPxeV2L1TYNFHLLkzw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nZjYwxLNIsnI3UEDjC6onJ649O/YHCR0ndz/rCRokJNf/9Ow1DTkYDe+gQe0jtBlF 3Ac6R+iqoTEu5gb0n31NlsaPBnJm2xOTgr7cCi1jxN97iPeYyumX8mowsSYYXnq6bN tjNVTZPAaj6kIZTFOjSMdQ9VP1RSESU9CDrvc/zUTA5tJ4hTIfsE9Up5+a349A8hmp mHj9pCazllwDRqNxyvrNbiWGMxSQvIdz6iFvMA2MIb8tyKDfY/SSJ9/sJXTvgb4usq gr/o2b6vkiRaWnkWHBHvFcbF83yyNdX0ZU/eSG6KdVNXl0g3lndVIzwDG0kjiCWHbY fnnqkKM1ob26A== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Krzysztof Kozlowski , Vladimir Zapolskiy , linux-samsung-soc@vger.kernel.org Subject: [PATCH v2 22/29] crypto: s5p-sss - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:11 -0800 Message-ID: <20241230001418.74739-23-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers s5p_sg_copy_buf() open-coded a copy from/to a scatterlist using scatterwalk_* functions that are planned for removal. Replace it with the new functions memcpy_from_sglist() and memcpy_to_sglist() instead. Also take the opportunity to replace calls to scatterwalk_map_and_copy() in the same file; this eliminates the confusing 'out' argument. Cc: Krzysztof Kozlowski Cc: Vladimir Zapolskiy Cc: linux-samsung-soc@vger.kernel.org Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. drivers/crypto/s5p-sss.c | 38 +++++++++++--------------------------- 1 file changed, 11 insertions(+), 27 deletions(-) diff --git a/drivers/crypto/s5p-sss.c b/drivers/crypto/s5p-sss.c index 57ab237e899e..b4c3c14dafd5 100644 --- a/drivers/crypto/s5p-sss.c +++ b/drivers/crypto/s5p-sss.c @@ -456,34 +456,21 @@ static void s5p_free_sg_cpy(struct s5p_aes_dev *dev, struct scatterlist **sg) kfree(*sg); *sg = NULL; } -static void s5p_sg_copy_buf(void *buf, struct scatterlist *sg, - unsigned int nbytes, int out) -{ - struct scatter_walk walk; - - if (!nbytes) - return; - - scatterwalk_start(&walk, sg); - scatterwalk_copychunks(buf, &walk, nbytes, out); - scatterwalk_done(&walk, out, 0); -} - static void s5p_sg_done(struct s5p_aes_dev *dev) { struct skcipher_request *req = dev->req; struct s5p_aes_reqctx *reqctx = skcipher_request_ctx(req); if (dev->sg_dst_cpy) { dev_dbg(dev->dev, "Copying %d bytes of output data back to original place\n", dev->req->cryptlen); - s5p_sg_copy_buf(sg_virt(dev->sg_dst_cpy), dev->req->dst, - dev->req->cryptlen, 1); + memcpy_to_sglist(dev->req->dst, 0, sg_virt(dev->sg_dst_cpy), + dev->req->cryptlen); } s5p_free_sg_cpy(dev, &dev->sg_src_cpy); s5p_free_sg_cpy(dev, &dev->sg_dst_cpy); if (reqctx->mode & FLAGS_AES_CBC) memcpy_fromio(req->iv, dev->aes_ioaddr + SSS_REG_AES_IV_DATA(0), AES_BLOCK_SIZE); @@ -524,11 +511,11 @@ static int s5p_make_sg_cpy(struct s5p_aes_dev *dev, struct scatterlist *src, kfree(*dst); *dst = NULL; return -ENOMEM; } - s5p_sg_copy_buf(pages, src, dev->req->cryptlen, 0); + memcpy_from_sglist(pages, src, 0, dev->req->cryptlen); sg_init_table(*dst, 1); sg_set_buf(*dst, pages, len); return 0; @@ -1033,12 +1020,11 @@ static int s5p_hash_copy_sgs(struct s5p_hash_reqctx *ctx, } if (ctx->bufcnt) memcpy(buf, ctx->dd->xmit_buf, ctx->bufcnt); - scatterwalk_map_and_copy(buf + ctx->bufcnt, sg, ctx->skip, - new_len, 0); + memcpy_from_sglist(buf + ctx->bufcnt, sg, ctx->skip, new_len); sg_init_table(ctx->sgl, 1); sg_set_buf(ctx->sgl, buf, len); ctx->sg = ctx->sgl; ctx->sg_len = 1; ctx->bufcnt = 0; @@ -1227,12 +1213,11 @@ static int s5p_hash_prepare_request(struct ahash_request *req, bool update) int len = BUFLEN - ctx->bufcnt % BUFLEN; if (len > nbytes) len = nbytes; - scatterwalk_map_and_copy(ctx->buffer + ctx->bufcnt, req->src, - 0, len, 0); + memcpy_from_sglist(ctx->buffer + ctx->bufcnt, req->src, 0, len); ctx->bufcnt += len; nbytes -= len; ctx->skip = len; } else { ctx->skip = 0; @@ -1251,13 +1236,12 @@ static int s5p_hash_prepare_request(struct ahash_request *req, bool update) xmit_len -= xmit_len & (BUFLEN - 1); hash_later = ctx->total - xmit_len; /* copy hash_later bytes from end of req->src */ /* previous bytes are in xmit_buf, so no overwrite */ - scatterwalk_map_and_copy(ctx->buffer, req->src, - req->nbytes - hash_later, - hash_later, 0); + memcpy_from_sglist(ctx->buffer, req->src, + req->nbytes - hash_later, hash_later); } if (xmit_len > BUFLEN) { ret = s5p_hash_prepare_sgs(ctx, req->src, nbytes - hash_later, final); @@ -1265,12 +1249,12 @@ static int s5p_hash_prepare_request(struct ahash_request *req, bool update) return ret; } else { /* have buffered data only */ if (unlikely(!ctx->bufcnt)) { /* first update didn't fill up buffer */ - scatterwalk_map_and_copy(ctx->dd->xmit_buf, req->src, - 0, xmit_len, 0); + memcpy_from_sglist(ctx->dd->xmit_buf, req->src, + 0, xmit_len); } sg_init_table(ctx->sgl, 1); sg_set_buf(ctx->sgl, ctx->dd->xmit_buf, xmit_len); @@ -1504,12 +1488,12 @@ static int s5p_hash_update(struct ahash_request *req) if (!req->nbytes) return 0; if (ctx->bufcnt + req->nbytes <= BUFLEN) { - scatterwalk_map_and_copy(ctx->buffer + ctx->bufcnt, req->src, - 0, req->nbytes, 0); + memcpy_from_sglist(ctx->buffer + ctx->bufcnt, req->src, + 0, req->nbytes); ctx->bufcnt += req->nbytes; return 0; } return s5p_hash_enqueue(req, true); /* HASH_OP_UPDATE */ From patchwork Mon Dec 30 00:14:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922927 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 845791A2567; Mon, 30 Dec 2024 00:16:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; cv=none; b=g3aNWccQxwg3H6UdnWMaZucq3oe3JAndBqErRAPaGly6CWE2+5458oL8gHhmcSbyil9oYehgf2l2hOPJEoPkUl7hsLOCBrmu459xzFxwyx8Wvb31IgAyGKyXS8PieIlkMjD2ooKKBhx2kqG+9q/v+3OZoOQ4kd72vjEIuFTUDVc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; c=relaxed/simple; bh=Zhg/KDiZCPgy+MF6pUjpqhVE/E/er2PToHEWYDXwwVE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=POXL8y2A72f6fafolC0tDW9XI7wrcJMGgJcNNomqtZwT0inu6lOFkuLSToEamn2Vr0bdCvXTGRXMs2NyBABeI1M/X0ZKQda/d6pSg5naXT2+2M09ehKOqd73l8VkPzR11YJ7echmfPKb00XKdEfu4zGbAstrLtwVZm8xYOEA6dQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UQeQWSWG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UQeQWSWG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26F18C4CEDC; Mon, 30 Dec 2024 00:16:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517767; bh=Zhg/KDiZCPgy+MF6pUjpqhVE/E/er2PToHEWYDXwwVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UQeQWSWGrhHBPwio5Jg0oNJ3NF0/xyx1ANYtKFo++QjWgDy9z/zithnQznylaiBPP 2oQGvntdxCqhBUGU2Y/pZQTFpcHDvY8e8qYJEKuOgpKN7KyMNTmPg+ZQeVDQ3zpWvo AZvhLxC2+/m28vE+PWJhDrES5EKpMCwYrzAiFI/bEoz/qIvKAbl5KlQJHZthdTvFjV /3pIYi5hn3cUFwx7IrfTU8myRkoCRJAo1KH6MkpD2Jaxrd0gtPiQYoSHwXukFAVpr0 42LRTkDtK5SWA/S7odPwpeumj0a4zPYFIIxJ3qKZeMuNT7K+Wsb7znPzRmRJGSbLWE 3ZiK9qncSkiKw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Alexandre Torgue , Maxime Coquelin , =?utf-8?b?TWF4aW1lIE3DqXLDqQ==?= , Thomas Bourgoin , linux-stm32@st-md-mailman.stormreply.com Subject: [PATCH v2 23/29] crypto: stm32 - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:12 -0800 Message-ID: <20241230001418.74739-24-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Replace calls to the deprecated function scatterwalk_copychunks() with memcpy_from_scatterwalk(), memcpy_to_scatterwalk(), scatterwalk_skip(), or scatterwalk_start_at_pos() as appropriate. Cc: Alexandre Torgue Cc: Maxime Coquelin Cc: Maxime Méré Cc: Thomas Bourgoin Cc: linux-stm32@st-md-mailman.stormreply.com Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. drivers/crypto/stm32/stm32-cryp.c | 34 +++++++++++++++---------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/drivers/crypto/stm32/stm32-cryp.c b/drivers/crypto/stm32/stm32-cryp.c index 14c6339c2e43..5ce88e7a8f65 100644 --- a/drivers/crypto/stm32/stm32-cryp.c +++ b/drivers/crypto/stm32/stm32-cryp.c @@ -664,11 +664,11 @@ static void stm32_cryp_write_ccm_first_header(struct stm32_cryp *cryp) len = 6; } written = min_t(size_t, AES_BLOCK_SIZE - len, alen); - scatterwalk_copychunks((char *)block + len, &cryp->in_walk, written, 0); + memcpy_from_scatterwalk((char *)block + len, &cryp->in_walk, written); writesl(cryp->regs + cryp->caps->din, block, AES_BLOCK_32); cryp->header_in -= written; @@ -991,11 +991,11 @@ static int stm32_cryp_header_dma_start(struct stm32_cryp *cryp) tx_in->callback_param = cryp; tx_in->callback = stm32_cryp_header_dma_callback; /* Advance scatterwalk to not DMA'ed data */ align_size = ALIGN_DOWN(cryp->header_in, cryp->hw_blocksize); - scatterwalk_copychunks(NULL, &cryp->in_walk, align_size, 2); + scatterwalk_skip(&cryp->in_walk, align_size); cryp->header_in -= align_size; ret = dma_submit_error(dmaengine_submit(tx_in)); if (ret < 0) { dev_err(cryp->dev, "DMA in submit failed\n"); @@ -1054,22 +1054,22 @@ static int stm32_cryp_dma_start(struct stm32_cryp *cryp) tx_out->callback = stm32_cryp_dma_callback; tx_out->callback_param = cryp; /* Advance scatterwalk to not DMA'ed data */ align_size = ALIGN_DOWN(cryp->payload_in, cryp->hw_blocksize); - scatterwalk_copychunks(NULL, &cryp->in_walk, align_size, 2); + scatterwalk_skip(&cryp->in_walk, align_size); cryp->payload_in -= align_size; ret = dma_submit_error(dmaengine_submit(tx_in)); if (ret < 0) { dev_err(cryp->dev, "DMA in submit failed\n"); return ret; } dma_async_issue_pending(cryp->dma_lch_in); /* Advance scatterwalk to not DMA'ed data */ - scatterwalk_copychunks(NULL, &cryp->out_walk, align_size, 2); + scatterwalk_skip(&cryp->out_walk, align_size); cryp->payload_out -= align_size; ret = dma_submit_error(dmaengine_submit(tx_out)); if (ret < 0) { dev_err(cryp->dev, "DMA out submit failed\n"); return ret; @@ -1735,13 +1735,13 @@ static int stm32_cryp_prepare_req(struct skcipher_request *req, in_sg = areq->src; out_sg = areq->dst; scatterwalk_start(&cryp->in_walk, in_sg); - scatterwalk_start(&cryp->out_walk, out_sg); /* In output, jump after assoc data */ - scatterwalk_copychunks(NULL, &cryp->out_walk, cryp->areq->assoclen, 2); + scatterwalk_start_at_pos(&cryp->out_walk, out_sg, + areq->assoclen); ret = stm32_cryp_hw_init(cryp); if (ret) return ret; @@ -1871,16 +1871,16 @@ static int stm32_cryp_read_auth_tag(struct stm32_cryp *cryp) if (is_encrypt(cryp)) { u32 out_tag[AES_BLOCK_32]; /* Get and write tag */ readsl(cryp->regs + cryp->caps->dout, out_tag, AES_BLOCK_32); - scatterwalk_copychunks(out_tag, &cryp->out_walk, cryp->authsize, 1); + memcpy_to_scatterwalk(&cryp->out_walk, out_tag, cryp->authsize); } else { /* Get and check tag */ u32 in_tag[AES_BLOCK_32], out_tag[AES_BLOCK_32]; - scatterwalk_copychunks(in_tag, &cryp->in_walk, cryp->authsize, 0); + memcpy_from_scatterwalk(in_tag, &cryp->in_walk, cryp->authsize); readsl(cryp->regs + cryp->caps->dout, out_tag, AES_BLOCK_32); if (crypto_memneq(in_tag, out_tag, cryp->authsize)) ret = -EBADMSG; } @@ -1921,22 +1921,22 @@ static void stm32_cryp_check_ctr_counter(struct stm32_cryp *cryp) static void stm32_cryp_irq_read_data(struct stm32_cryp *cryp) { u32 block[AES_BLOCK_32]; readsl(cryp->regs + cryp->caps->dout, block, cryp->hw_blocksize / sizeof(u32)); - scatterwalk_copychunks(block, &cryp->out_walk, min_t(size_t, cryp->hw_blocksize, - cryp->payload_out), 1); + memcpy_to_scatterwalk(&cryp->out_walk, block, min_t(size_t, cryp->hw_blocksize, + cryp->payload_out)); cryp->payload_out -= min_t(size_t, cryp->hw_blocksize, cryp->payload_out); } static void stm32_cryp_irq_write_block(struct stm32_cryp *cryp) { u32 block[AES_BLOCK_32] = {0}; - scatterwalk_copychunks(block, &cryp->in_walk, min_t(size_t, cryp->hw_blocksize, - cryp->payload_in), 0); + memcpy_from_scatterwalk(block, &cryp->in_walk, min_t(size_t, cryp->hw_blocksize, + cryp->payload_in)); writesl(cryp->regs + cryp->caps->din, block, cryp->hw_blocksize / sizeof(u32)); cryp->payload_in -= min_t(size_t, cryp->hw_blocksize, cryp->payload_in); } static void stm32_cryp_irq_write_gcm_padded_data(struct stm32_cryp *cryp) @@ -1979,12 +1979,12 @@ static void stm32_cryp_irq_write_gcm_padded_data(struct stm32_cryp *cryp) * Same code as stm32_cryp_irq_read_data(), but we want to store * block value */ readsl(cryp->regs + cryp->caps->dout, block, cryp->hw_blocksize / sizeof(u32)); - scatterwalk_copychunks(block, &cryp->out_walk, min_t(size_t, cryp->hw_blocksize, - cryp->payload_out), 1); + memcpy_to_scatterwalk(&cryp->out_walk, block, min_t(size_t, cryp->hw_blocksize, + cryp->payload_out)); cryp->payload_out -= min_t(size_t, cryp->hw_blocksize, cryp->payload_out); /* d) change mode back to AES GCM */ cfg &= ~CR_ALGO_MASK; @@ -2077,12 +2077,12 @@ static void stm32_cryp_irq_write_ccm_padded_data(struct stm32_cryp *cryp) * Same code as stm32_cryp_irq_read_data(), but we want to store * block value */ readsl(cryp->regs + cryp->caps->dout, block, cryp->hw_blocksize / sizeof(u32)); - scatterwalk_copychunks(block, &cryp->out_walk, min_t(size_t, cryp->hw_blocksize, - cryp->payload_out), 1); + memcpy_to_scatterwalk(&cryp->out_walk, block, min_t(size_t, cryp->hw_blocksize, + cryp->payload_out)); cryp->payload_out -= min_t(size_t, cryp->hw_blocksize, cryp->payload_out); /* d) Load again CRYP_CSGCMCCMxR */ for (i = 0; i < ARRAY_SIZE(cstmp2); i++) cstmp2[i] = stm32_cryp_read(cryp, CRYP_CSGCMCCM0R + i * 4); @@ -2159,11 +2159,11 @@ static void stm32_cryp_irq_write_gcmccm_header(struct stm32_cryp *cryp) u32 block[AES_BLOCK_32] = {0}; size_t written; written = min_t(size_t, AES_BLOCK_SIZE, cryp->header_in); - scatterwalk_copychunks(block, &cryp->in_walk, written, 0); + memcpy_from_scatterwalk(block, &cryp->in_walk, written); writesl(cryp->regs + cryp->caps->din, block, AES_BLOCK_32); cryp->header_in -= written; From patchwork Mon Dec 30 00:14:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922928 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CAE971A3031; Mon, 30 Dec 2024 00:16:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; cv=none; b=JkB68abSAHv9+nkIf+of9gR6TTRqV9bG07FX492ZsBq58EWAybTpFJpJv05fKm9zchjxLsQV56nkrZBWkJc6mRz6tYuGw510HFi2+J1qVUFdnMz9Eu2RN3FMAzDE7MWfce7Qhhrc1llgfREs26c0VNK/ReTBjuuUxXlmreYE9hM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517767; c=relaxed/simple; bh=Id9OlnOz/d+SiVoBEHgQnOUjd+cNJqgxBPN0qKJ4qTw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kHlr8g1sHJJwrt6DUvXSgY+RAh3CVbE+urvCGJK2+39FdIslAy4hYU/6+QTdJGUe1fFRD58v7ehRHZfxTRgFoK3PEb1bHAqc0nkiBlZX/udYKmgwCguq0mlnlQImhmWpHe+pmhsHq6Yy9uzOi2HfBTF1mOIqpel29kJIt/BUSEQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kTWlGtUP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kTWlGtUP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B1E7C4CED1; Mon, 30 Dec 2024 00:16:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517767; bh=Id9OlnOz/d+SiVoBEHgQnOUjd+cNJqgxBPN0qKJ4qTw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kTWlGtUP6QMja6h5RgOVuD1T6HdMyJyy6Vr46sHKS9dvrg+mUj7QHHKHHzH/nGmog 043XOYVSuxYfCP/6JvA2/tWqixBMZs18+3ZI40n7Tom0OjsHdxwogEcn+dZHXig67v 5ch83O2l9x0CSPuJeyNVa7n3jP3yGY9LIXhn3p9F5hRRhu9UtLk9zXZ9Qe0NS8aI2P WjyB13eMb/Q+CVsEX+Msim6kgdCPhqgMZNDyFYNI3wms0FWdGNeUNG6Sa0kCnwWAdE oJ26qevfjPPByvynZeOS7nYJYMtErlv+JoljMA5DCgQ0fq6FMGJgYSHQdush4vz7hO G0xpE53SXAaNg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 24/29] crypto: x86/aes-gcm - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:13 -0800 Message-ID: <20241230001418.74739-25-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In gcm_process_assoc(), use scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(). Use scatterwalk_done_src() which consolidates scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Also rename some variables to avoid implying that anything is actually mapped (it's not), or that the loop is going page by page (it is for now, but nothing actually requires that to be the case). Signed-off-by: Eric Biggers --- arch/x86/crypto/aesni-intel_glue.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c index 11e95fc62636..22e61efbf5fe 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -1289,45 +1289,45 @@ static void gcm_process_assoc(const struct aes_gcm_key *key, u8 ghash_acc[16], memset(ghash_acc, 0, 16); scatterwalk_start(&walk, sg_src); while (assoclen) { - unsigned int len_this_page = scatterwalk_clamp(&walk, assoclen); - void *mapped = scatterwalk_map(&walk); - const void *src = mapped; + unsigned int orig_len_this_step; + const u8 *orig_src = scatterwalk_next(&walk, assoclen, + &orig_len_this_step); + unsigned int len_this_step = orig_len_this_step; unsigned int len; + const u8 *src = orig_src; - assoclen -= len_this_page; - scatterwalk_advance(&walk, len_this_page); if (unlikely(pos)) { - len = min(len_this_page, 16 - pos); + len = min(len_this_step, 16 - pos); memcpy(&buf[pos], src, len); pos += len; src += len; - len_this_page -= len; + len_this_step -= len; if (pos < 16) goto next; aes_gcm_aad_update(key, ghash_acc, buf, 16, flags); pos = 0; } - len = len_this_page; + len = len_this_step; if (unlikely(assoclen)) /* Not the last segment yet? */ len = round_down(len, 16); aes_gcm_aad_update(key, ghash_acc, src, len, flags); src += len; - len_this_page -= len; - if (unlikely(len_this_page)) { - memcpy(buf, src, len_this_page); - pos = len_this_page; + len_this_step -= len; + if (unlikely(len_this_step)) { + memcpy(buf, src, len_this_step); + pos = len_this_step; } next: - scatterwalk_unmap(mapped); - scatterwalk_pagedone(&walk, 0, assoclen); + scatterwalk_done_src(&walk, orig_src, orig_len_this_step); if (need_resched()) { kernel_fpu_end(); kernel_fpu_begin(); } + assoclen -= orig_len_this_step; } if (unlikely(pos)) aes_gcm_aad_update(key, ghash_acc, buf, pos, flags); } From patchwork Mon Dec 30 00:14:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922929 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 229D01A4E98; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517768; cv=none; b=iNXq4oonBu0Oj3rLWPzDuCkoB09vHamRBDb1TVTQ8q4rByMs0PWVhyF6e/1c6bKrsKS+LfBV87ofBHpTZ0YvtgLZTylWReRyKTQFxzdzOBgr+iO+3zskR9vqQZTdsLzOzStH9SzrA/KaQW+UcS5uwgZItFFSQGR90+hsgqGyIrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517768; c=relaxed/simple; bh=5pj/D1hOOx/ISCCP0hxDOq++HsD61KuEGNf//Yqj0yQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uj5D57REkcS750r62xUG53kl0qNw7Z0uPBbR6EizjGYfh/t0TGSF7SxYI+1KU7ryE3qnikVHleb5n+W8wRLa4z2u1NG++jju2mrgWyxnUX3BbwPKrLoIpTGfYhQu1ED1xMhqNxosl9ox/36Aeb3E/UP3riIJ8Ca8STJNC8oce3k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fHMdydOK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fHMdydOK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CC581C4CED4; Mon, 30 Dec 2024 00:16:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517768; bh=5pj/D1hOOx/ISCCP0hxDOq++HsD61KuEGNf//Yqj0yQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fHMdydOK+6R0PBeMmR9Fiv3z0HFaKdoTqKa4U+3aCesV5aowRKnGWolDkOrs2Q8hR UJ7g48Lfa/Cb4gHzUc6Nk1spA/TafZjhN0yMC1SVt61SI9BbZ5MfjrEx+w673I4TKS oX8pFqW355F8IuOgTRBmrfqTHJXHmkOv8qzm/Yrt+Gf2ZxDsaIKs2FoAePrkLFDw1S yDpyCgduattu7wJVlU3m/Zoh+QpciQnsD9HaKaHjmaLKFr+waxe2xwrDo2ofwtwuDH QmeAloq2OHYbGfTHbWHALCBKWnDuC2vBDw8VNIBOULAbSu0b/nbDGig/Ojrq+P1cZN LGpmg8O46fhDw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 25/29] crypto: x86/aegis - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:14 -0800 Message-ID: <20241230001418.74739-26-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers In crypto_aegis128_aesni_process_ad(), use scatterwalk_next() which consolidates scatterwalk_clamp() and scatterwalk_map(). Use scatterwalk_done_src() which consolidates scatterwalk_unmap(), scatterwalk_advance(), and scatterwalk_done(). Signed-off-by: Eric Biggers --- arch/x86/crypto/aegis128-aesni-glue.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/x86/crypto/aegis128-aesni-glue.c b/arch/x86/crypto/aegis128-aesni-glue.c index 01fa568dc5fc..1bd093d073ed 100644 --- a/arch/x86/crypto/aegis128-aesni-glue.c +++ b/arch/x86/crypto/aegis128-aesni-glue.c @@ -69,14 +69,14 @@ static void crypto_aegis128_aesni_process_ad( struct aegis_block buf; unsigned int pos = 0; scatterwalk_start(&walk, sg_src); while (assoclen != 0) { - unsigned int size = scatterwalk_clamp(&walk, assoclen); + unsigned int size; + const u8 *mapped = scatterwalk_next(&walk, assoclen, &size); unsigned int left = size; - void *mapped = scatterwalk_map(&walk); - const u8 *src = (const u8 *)mapped; + const u8 *src = mapped; if (pos + size >= AEGIS128_BLOCK_SIZE) { if (pos > 0) { unsigned int fill = AEGIS128_BLOCK_SIZE - pos; memcpy(buf.bytes + pos, src, fill); @@ -95,13 +95,11 @@ static void crypto_aegis128_aesni_process_ad( memcpy(buf.bytes + pos, src, left); pos += left; assoclen -= size; - scatterwalk_unmap(mapped); - scatterwalk_advance(&walk, size); - scatterwalk_done(&walk, 0, assoclen); + scatterwalk_done_src(&walk, mapped, size); } if (pos > 0) { memset(buf.bytes + pos, 0, AEGIS128_BLOCK_SIZE - pos); aegis128_aesni_ad(state, buf.bytes, AEGIS128_BLOCK_SIZE); From patchwork Mon Dec 30 00:14:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922930 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C33D51A83F8; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517768; cv=none; b=cY/xMXDTlnVJU2AGpcd+awAhp/rcgDFtm+6e7VWUPmIL1gaC/flj9W01RmQOVRLbsduj7YdG2puN6mQOn8syhGim6nx1gemKOIb8z/KPi6ZLWDjqVHDafJyIG7pGxVdUfw+bd/X463jqP0xNqzRovJuKw2xjjEXbtXhHDGRdOSc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517768; c=relaxed/simple; bh=5aJ14qFdsp3EjNcjdgSBEGYXvCZe209dFMhq2DFCmN0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JtFo1lBM/sHfXfmsc9Gv+ZC1vSlhs22ZtFpDFwWHix41A49jjzMu13M413FVVbowxTe9oUZ30nOJ166QprapPN3syM72jLn2ezKeTeCxgTdMIOs1at+z9qp0wO00EtHEMQTiOiWYUyQnd94qhGbkF9UmLhI+DEJqMZSjCcHiBbI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Zrt3lNbn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Zrt3lNbn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E78AC4CEE0; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517768; bh=5aJ14qFdsp3EjNcjdgSBEGYXvCZe209dFMhq2DFCmN0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zrt3lNbnAS43581HvbVTvyRO4az38cnZlbFuRpBQrSK3GsYN4oF3p3E0VWeTa3Duo Ll9IO0k0TUUpHi4tdE7bWdKGOZPBXxgiJ8snA8VnFtmbzWB46jZ1RGojGVpW4srM7W LTdaS88R/xy77WlL7Gl9N+6jW3o3cx5xnDvm5bCV5GC3QgkRfO3qtk9hAOKgKuN/U9 eUKxW+W/ZLpFHtTlLCxNXTo1ZyPvuZD1WPM/rj5qiyDWX4GO7W07zTOYmitk5T6wDI okvbOlw6X4VLjdj69i6K5OyVowqZb8R1gQFPxwUfDI+0eFCMux+vZLXw5k04KSYWiB Yx5ecKor+UVnw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Boris Pismenny , Jakub Kicinski , John Fastabend Subject: [PATCH v2 26/29] net/tls: use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:15 -0800 Message-ID: <20241230001418.74739-27-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Eric Biggers Replace calls to the deprecated function scatterwalk_copychunks() with memcpy_from_scatterwalk(), memcpy_to_scatterwalk(), or scatterwalk_skip() as appropriate. The new functions generally behave more as expected and eliminate the need to call scatterwalk_done() or scatterwalk_pagedone(). However, the new functions intentionally do not advance to the next sg entry right away, which would have broken chain_to_walk() which is accessing the fields of struct scatter_walk directly. To avoid this, replace chain_to_walk() with scatterwalk_get_sglist() which supports the needed functionality. Cc: Boris Pismenny Cc: Jakub Kicinski Cc: John Fastabend Signed-off-by: Eric Biggers --- This patch is part of a long series touching many files, so I have limited the Cc list on the full series. If you want the full series and did not receive it, please retrieve it from lore.kernel.org. net/tls/tls_device_fallback.c | 31 ++++++------------------------- 1 file changed, 6 insertions(+), 25 deletions(-) diff --git a/net/tls/tls_device_fallback.c b/net/tls/tls_device_fallback.c index f9e3d3d90dcf..03d508a45aae 100644 --- a/net/tls/tls_device_fallback.c +++ b/net/tls/tls_device_fallback.c @@ -35,21 +35,10 @@ #include #include #include "tls.h" -static void chain_to_walk(struct scatterlist *sg, struct scatter_walk *walk) -{ - struct scatterlist *src = walk->sg; - int diff = walk->offset - src->offset; - - sg_set_page(sg, sg_page(src), - src->length - diff, walk->offset); - - scatterwalk_crypto_chain(sg, sg_next(src), 2); -} - static int tls_enc_record(struct aead_request *aead_req, struct crypto_aead *aead, char *aad, char *iv, __be64 rcd_sn, struct scatter_walk *in, struct scatter_walk *out, int *in_len, @@ -67,20 +56,17 @@ static int tls_enc_record(struct aead_request *aead_req, DEBUG_NET_WARN_ON_ONCE(!cipher_desc || !cipher_desc->offloadable); buf_size = TLS_HEADER_SIZE + cipher_desc->iv; len = min_t(int, *in_len, buf_size); - scatterwalk_copychunks(buf, in, len, 0); - scatterwalk_copychunks(buf, out, len, 1); + memcpy_from_scatterwalk(buf, in, len); + memcpy_to_scatterwalk(out, buf, len); *in_len -= len; if (!*in_len) return 0; - scatterwalk_pagedone(in, 0, 1); - scatterwalk_pagedone(out, 1, 1); - len = buf[4] | (buf[3] << 8); len -= cipher_desc->iv; tls_make_aad(aad, len - cipher_desc->tag, (char *)&rcd_sn, buf[0], prot); @@ -88,12 +74,12 @@ static int tls_enc_record(struct aead_request *aead_req, sg_init_table(sg_in, ARRAY_SIZE(sg_in)); sg_init_table(sg_out, ARRAY_SIZE(sg_out)); sg_set_buf(sg_in, aad, TLS_AAD_SPACE_SIZE); sg_set_buf(sg_out, aad, TLS_AAD_SPACE_SIZE); - chain_to_walk(sg_in + 1, in); - chain_to_walk(sg_out + 1, out); + scatterwalk_get_sglist(in, sg_in + 1); + scatterwalk_get_sglist(out, sg_out + 1); *in_len -= len; if (*in_len < 0) { *in_len += cipher_desc->tag; /* the input buffer doesn't contain the entire record. @@ -108,14 +94,12 @@ static int tls_enc_record(struct aead_request *aead_req, *in_len = 0; } if (*in_len) { - scatterwalk_copychunks(NULL, in, len, 2); - scatterwalk_pagedone(in, 0, 1); - scatterwalk_copychunks(NULL, out, len, 2); - scatterwalk_pagedone(out, 1, 1); + scatterwalk_skip(in, len); + scatterwalk_skip(out, len); } len -= cipher_desc->tag; aead_request_set_crypt(aead_req, sg_in, sg_out, len, iv); @@ -160,13 +144,10 @@ static int tls_enc_records(struct aead_request *aead_req, cpu_to_be64(rcd_sn), &in, &out, &len, prot); rcd_sn++; } while (rc == 0 && len); - scatterwalk_done(&in, 0, 0); - scatterwalk_done(&out, 1, 0); - return rc; } /* Can't use icsk->icsk_af_ops->send_check here because the ip addresses * might have been changed by NAT. From patchwork Mon Dec 30 00:14:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922931 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E50701A840C; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517769; cv=none; b=F4u3dZnqfM/paerTQCcyKGCAe3qHCCM4nVq72/cLnL3MXMmsn01uq3F/qr4Vq5we4JHWfLzp2vhVyp6NutQUMCifXWdYY7WXuD/ljOub/JnJnIrH5X9yvbXPHCtqUG2jfiCja1mJ2E5QMRBVIuCrpv1yyOi4+OmD+fI+GEYpNns= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517769; c=relaxed/simple; bh=9+aJ/7IU1xoI3FM4cnTH4BXwVQl9uMeigli+J8RUP4E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i62b0seDsoi8cy9cqc9QNchpJ8yqo4fVb+JI1CfFYFo+UbyoO4KDOFIhJ4LtHw0R/XafXC2Y8crThitzgHyEqsBMiYdo+u2Yaj8+X6z2LX0H0WBuKKTjk+2f5q/so61Q9LRS1ulzawWB3t59UEysboWkImxHPIU3b+3jwBYsaVc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=p/oa3Z9E; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="p/oa3Z9E" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 769EFC4CED1; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517768; bh=9+aJ/7IU1xoI3FM4cnTH4BXwVQl9uMeigli+J8RUP4E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p/oa3Z9Et7ds2XrJW44o0AsYO6FdUgAqMWzZTTNEUqMlALdh71a5+Ol4b6JbuRlu6 AvNEbDQzhAWASmt9vnzAWq7bmGrOv2BM6cOv4b3PsgIFLGTs4wQ3249oLfSay4qw/M vUpqj9Wr5gaCSISTv1pkIBQgAvxNTP+TLoAb6kyovWk8B4erX2r29V2S19b6FE33Gx ExUawssLiX5oHZ+DugMEos1gRBttlOrvdcbGf9UX1lwbYVnyT5goUxY0q+FHTPbHyO xEEBYOAnIN56Nenkb9SNjPkjFf/WjagN1VzXW09c276Y4wQr4pkZIFFxsF2/OFKVlY HIL6hObTgePxw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 27/29] crypto: skcipher - use the new scatterwalk functions Date: Sun, 29 Dec 2024 16:14:16 -0800 Message-ID: <20241230001418.74739-28-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Convert skcipher_walk to use the new scatterwalk functions. This includes a few changes to exactly where the different parts of the iteration happen. For example the dcache flush that previously happened in scatterwalk_done() now happens in scatterwalk_dst_done() or in memcpy_to_scatterwalk(). Advancing to the next sg entry now happens just-in-time in scatterwalk_clamp() instead of in scatterwalk_done(). Signed-off-by: Eric Biggers --- crypto/skcipher.c | 51 ++++++++++++++++++----------------------------- 1 file changed, 19 insertions(+), 32 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 7abafe385fd5..8f6b09377368 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -46,20 +46,10 @@ static inline void skcipher_map_src(struct skcipher_walk *walk) static inline void skcipher_map_dst(struct skcipher_walk *walk) { walk->dst.virt.addr = scatterwalk_map(&walk->out); } -static inline void skcipher_unmap_src(struct skcipher_walk *walk) -{ - scatterwalk_unmap(walk->src.virt.addr); -} - -static inline void skcipher_unmap_dst(struct skcipher_walk *walk) -{ - scatterwalk_unmap(walk->dst.virt.addr); -} - static inline gfp_t skcipher_walk_gfp(struct skcipher_walk *walk) { return walk->flags & SKCIPHER_WALK_SLEEP ? GFP_KERNEL : GFP_ATOMIC; } @@ -67,18 +57,10 @@ static inline struct skcipher_alg *__crypto_skcipher_alg( struct crypto_alg *alg) { return container_of(alg, struct skcipher_alg, base); } -static int skcipher_done_slow(struct skcipher_walk *walk, unsigned int bsize) -{ - u8 *addr = PTR_ALIGN(walk->buffer, walk->alignmask + 1); - - scatterwalk_copychunks(addr, &walk->out, bsize, 1); - return 0; -} - /** * skcipher_walk_done() - finish one step of a skcipher_walk * @walk: the skcipher_walk * @res: number of bytes *not* processed (>= 0) from walk->nbytes, * or a -errno value to terminate the walk due to an error @@ -109,44 +91,45 @@ int skcipher_walk_done(struct skcipher_walk *walk, int res) } if (likely(!(walk->flags & (SKCIPHER_WALK_SLOW | SKCIPHER_WALK_COPY | SKCIPHER_WALK_DIFF)))) { -unmap_src: - skcipher_unmap_src(walk); + scatterwalk_advance(&walk->in, n); } else if (walk->flags & SKCIPHER_WALK_DIFF) { - skcipher_unmap_dst(walk); - goto unmap_src; + scatterwalk_unmap(walk->src.virt.addr); + scatterwalk_advance(&walk->in, n); } else if (walk->flags & SKCIPHER_WALK_COPY) { + scatterwalk_advance(&walk->in, n); skcipher_map_dst(walk); memcpy(walk->dst.virt.addr, walk->page, n); - skcipher_unmap_dst(walk); } else { /* SKCIPHER_WALK_SLOW */ if (res > 0) { /* * Didn't process all bytes. Either the algorithm is * broken, or this was the last step and it turned out * the message wasn't evenly divisible into blocks but * the algorithm requires it. */ res = -EINVAL; total = 0; - } else - n = skcipher_done_slow(walk, n); + } else { + u8 *buf = PTR_ALIGN(walk->buffer, walk->alignmask + 1); + + memcpy_to_scatterwalk(&walk->out, buf, n); + } + goto dst_done; } + scatterwalk_done_dst(&walk->out, walk->dst.virt.addr, n); +dst_done: + if (res > 0) res = 0; walk->total = total; walk->nbytes = 0; - scatterwalk_advance(&walk->in, n); - scatterwalk_advance(&walk->out, n); - scatterwalk_done(&walk->in, 0, total); - scatterwalk_done(&walk->out, 1, total); - if (total) { if (walk->flags & SKCIPHER_WALK_SLEEP) cond_resched(); walk->flags &= ~(SKCIPHER_WALK_SLOW | SKCIPHER_WALK_COPY | SKCIPHER_WALK_DIFF); @@ -189,11 +172,11 @@ static int skcipher_next_slow(struct skcipher_walk *walk, unsigned int bsize) walk->buffer = buffer; } walk->dst.virt.addr = PTR_ALIGN(buffer, alignmask + 1); walk->src.virt.addr = walk->dst.virt.addr; - scatterwalk_copychunks(walk->src.virt.addr, &walk->in, bsize, 0); + memcpy_from_scatterwalk(walk->src.virt.addr, &walk->in, bsize); walk->nbytes = bsize; walk->flags |= SKCIPHER_WALK_SLOW; return 0; @@ -203,11 +186,15 @@ static int skcipher_next_copy(struct skcipher_walk *walk) { u8 *tmp = walk->page; skcipher_map_src(walk); memcpy(tmp, walk->src.virt.addr, walk->nbytes); - skcipher_unmap_src(walk); + scatterwalk_unmap(walk->src.virt.addr); + /* + * walk->in is advanced later when the number of bytes actually + * processed (which might be less than walk->nbytes) is known. + */ walk->src.virt.addr = tmp; walk->dst.virt.addr = tmp; return 0; } From patchwork Mon Dec 30 00:14:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922932 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEF441A8411; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517769; cv=none; b=rILCd4lJHG1eNmxu96LP/CuhA6UxZaDF2x5EDEBLoIRZh9JSfFnsbkaBqfe4d1FGwB/DFq9Rmn7Sm6DxPNlOTnTw/cNz0v9pFJJUFnqs1ICaboCNsR7GNW6+hguFWkG8c7T8qX0d+dEIFH85nGytjds3BXadx59R03Lsoe0YHn0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517769; c=relaxed/simple; bh=pMrTGKnRhvoJeyHAUlKFSPsItxe+R2jmbyf6RObZplM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KPcLkGwZw47F/dPbJqPykL5DMqkSbplYPp1/ZJo1UDyCJotb7SDwX1txsQIHcSqeKrIpzvJmqDMUviXDHY74Ij95Pj+ORxMASxlNgXZ2PeE6yhgZNeeugVV6LVQ6FwXBwkcw5DNe+7A3YU0lJwx2OpwA+KcvyEVdWN0vy9ezt+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j43P1jYL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j43P1jYL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B62FDC4CEDC; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517768; bh=pMrTGKnRhvoJeyHAUlKFSPsItxe+R2jmbyf6RObZplM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=j43P1jYLNwozTAxRKkOgNrcVDZOpRlOgVajvZusCTpMbHeUq1irw/uz8v6ENkXYC4 8c1s4cvnItiCoA1dRuG7UaQl05bzs4AELaX312Bpeyg0esmTtGaoKbCnxvfAPL4pPz J/oWgmQDfFBN02jdyHtgCWFI9p+rwMEHuwzFCub+ih4KLuMAIYhfVseIPK704VYJ1K SruQxkn9rE/0vFy7i55ZU3Ndddv9kcNUKHIEsh/0J6oQQIfULPyUGCWVzcPw9RlyoN YS1b5NN60T+rfHozm8xPjQbNiMrGeA0jMwDx+N5euiX3bhSdxdKgJgLxzxG+1EIPyc NcTzzu1K+h/Zg== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 28/29] crypto: scatterwalk - remove obsolete functions Date: Sun, 29 Dec 2024 16:14:17 -0800 Message-ID: <20241230001418.74739-29-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers Remove various functions that are no longer used. Signed-off-by: Eric Biggers --- crypto/scatterwalk.c | 37 ------------------------------------ include/crypto/scatterwalk.h | 25 ------------------------ 2 files changed, 62 deletions(-) diff --git a/crypto/scatterwalk.c b/crypto/scatterwalk.c index 2e7a532152d6..87c080f565d4 100644 --- a/crypto/scatterwalk.c +++ b/crypto/scatterwalk.c @@ -28,47 +28,10 @@ void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes) walk->sg = sg; walk->offset = sg->offset + nbytes; } EXPORT_SYMBOL_GPL(scatterwalk_skip); -static inline void memcpy_dir(void *buf, void *sgdata, size_t nbytes, int out) -{ - void *src = out ? buf : sgdata; - void *dst = out ? sgdata : buf; - - memcpy(dst, src, nbytes); -} - -void scatterwalk_copychunks(void *buf, struct scatter_walk *walk, - size_t nbytes, int out) -{ - for (;;) { - unsigned int len_this_page = scatterwalk_pagelen(walk); - u8 *vaddr; - - if (len_this_page > nbytes) - len_this_page = nbytes; - - if (out != 2) { - vaddr = scatterwalk_map(walk); - memcpy_dir(buf, vaddr, len_this_page, out); - scatterwalk_unmap(vaddr); - } - - scatterwalk_advance(walk, len_this_page); - - if (nbytes == len_this_page) - break; - - buf += len_this_page; - nbytes -= len_this_page; - - scatterwalk_pagedone(walk, out & 1, 1); - } -} -EXPORT_SYMBOL_GPL(scatterwalk_copychunks); - inline void memcpy_from_scatterwalk(void *buf, struct scatter_walk *walk, unsigned int nbytes) { do { const void *src_addr; diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index f6262d05a3c7..ac03fdf88b2a 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -113,32 +113,10 @@ static inline void *scatterwalk_next(struct scatter_walk *walk, { *nbytes_ret = scatterwalk_clamp(walk, total); return scatterwalk_map(walk); } -static inline void scatterwalk_pagedone(struct scatter_walk *walk, int out, - unsigned int more) -{ - if (out) { - struct page *page; - - page = sg_page(walk->sg) + ((walk->offset - 1) >> PAGE_SHIFT); - flush_dcache_page(page); - } - - if (more && walk->offset >= walk->sg->offset + walk->sg->length) - scatterwalk_start(walk, sg_next(walk->sg)); -} - -static inline void scatterwalk_done(struct scatter_walk *walk, int out, - int more) -{ - if (!more || walk->offset >= walk->sg->offset + walk->sg->length || - !(walk->offset & (PAGE_SIZE - 1))) - scatterwalk_pagedone(walk, out, more); -} - static inline void scatterwalk_advance(struct scatter_walk *walk, unsigned int nbytes) { walk->offset += nbytes; } @@ -182,13 +160,10 @@ static inline void scatterwalk_done_dst(struct scatter_walk *walk, scatterwalk_advance(walk, nbytes); } void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes); -void scatterwalk_copychunks(void *buf, struct scatter_walk *walk, - size_t nbytes, int out); - void memcpy_from_scatterwalk(void *buf, struct scatter_walk *walk, unsigned int nbytes); void memcpy_to_scatterwalk(struct scatter_walk *walk, const void *buf, unsigned int nbytes); From patchwork Mon Dec 30 00:14:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Biggers X-Patchwork-Id: 13922933 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4369C1A9B4A; Mon, 30 Dec 2024 00:16:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517769; cv=none; b=cLnrWlNMonarbDLXPbdrS+C540FBEGP7DMiao4FCiy2NwrYSDFomCLXXDn6G05y94To00O0iO5KJqHAPE5K28vJESTz6TPVZbWaOM3ViOIDZOGnT3Xqlbh7vXL0Z/8bkjBm8Ts8U0jizNCXrK5h9WP1T+5GfhInzCyKdEWiJUcY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1735517769; c=relaxed/simple; bh=h7JvPT5ldnzaJjp9xBtbjrtty6vLenGC6gV46dAS22E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TMtkK3drU4olaQXaWrFOPjQXFrGv528i2VLl0lsgSzg1dtHrguQNPBmcRy0sdtlOgbu6LglZ/M4jD/uX0qrx26sT/YYWrkg0sjcJbrKsXjCqmyWH9Vl35kqEEpcclHeaYsVcu8LI3paMF3a+Z6DW9/DKEO5+bBRJtX2jCYpdn+8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KWEx5RBD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KWEx5RBD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 04626C4CEDD; Mon, 30 Dec 2024 00:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1735517769; bh=h7JvPT5ldnzaJjp9xBtbjrtty6vLenGC6gV46dAS22E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KWEx5RBDard8qO0nqmNDkTszFv9IUAeJCqmRE9vkVhAtuWdEDnF8/VUq24foLco0w VqeIZr1ITUNh0RTKl1/FFN+ufwW72YnLPDtUMV6c6Y5c+cCG54k9xftTJ2tTfPTQnr RSqNhxxLzXB6+w5RUhcYZdOo/Rkb8unTry/uHnVBNsqDJN10+iPJpUmfuZt3x16MXH VNLKtNytgjIxOOwzxF+KAZ9Er2Qw9pza2Q4gnbNln0IA6DOeGxj5dZBFMIqgUS6m/D L3gpukT656pnPBYJ8Kmr3Iqzz4DUuLyXAbwtdWO64Yw335pJcIMGYWB1rhcj3Cg1FI EJwRn5aMqieQw== From: Eric Biggers To: linux-crypto@vger.kernel.org Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 29/29] crypto: scatterwalk - don't split at page boundaries when !HIGHMEM Date: Sun, 29 Dec 2024 16:14:18 -0800 Message-ID: <20241230001418.74739-30-ebiggers@kernel.org> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230001418.74739-1-ebiggers@kernel.org> References: <20241230001418.74739-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Eric Biggers When !HIGHMEM, the kmap_local_page() in the scatterlist walker does not actually map anything, and the address it returns is just the address from the kernel's direct map, where each sg entry's data is virtually contiguous. To improve performance, stop unnecessarily clamping data segments to page boundaries in this case. For now, still limit segments to PAGE_SIZE. This is needed to prevent preemption from being disabled for too long when SIMD is used, and to support the alignmask case which still uses a page-sized bounce buffer. Even so, this change still helps a lot in cases where messages cross a page boundary. For example, testing IPsec with AES-GCM on x86_64, the messages are 1424 bytes which is less than PAGE_SIZE, but on the Rx side over a third cross a page boundary. These ended up being processed in three parts, with the middle part going through skcipher_next_slow which uses a 16-byte bounce buffer. That was causing a significant amount of overhead which unnecessarily reduced the performance benefit of the new x86_64 AES-GCM assembly code. This change solves the problem; all these messages now get passed to the assembly code in one part. Signed-off-by: Eric Biggers --- crypto/skcipher.c | 4 +- include/crypto/scatterwalk.h | 79 ++++++++++++++++++++++++++---------- 2 files changed, 59 insertions(+), 24 deletions(-) diff --git a/crypto/skcipher.c b/crypto/skcipher.c index 8f6b09377368..16db19663c3d 100644 --- a/crypto/skcipher.c +++ b/crypto/skcipher.c @@ -203,12 +203,12 @@ static int skcipher_next_fast(struct skcipher_walk *walk) { unsigned long diff; diff = offset_in_page(walk->in.offset) - offset_in_page(walk->out.offset); - diff |= (u8 *)scatterwalk_page(&walk->in) - - (u8 *)scatterwalk_page(&walk->out); + diff |= (u8 *)(sg_page(walk->in.sg) + (walk->in.offset >> PAGE_SHIFT)) - + (u8 *)(sg_page(walk->out.sg) + (walk->out.offset >> PAGE_SHIFT)); skcipher_map_src(walk); walk->dst.virt.addr = walk->src.virt.addr; if (diff) { diff --git a/include/crypto/scatterwalk.h b/include/crypto/scatterwalk.h index ac03fdf88b2a..3024adbdd443 100644 --- a/include/crypto/scatterwalk.h +++ b/include/crypto/scatterwalk.h @@ -47,28 +47,39 @@ static inline void scatterwalk_start_at_pos(struct scatter_walk *walk, } walk->sg = sg; walk->offset = sg->offset + pos; } -static inline unsigned int scatterwalk_pagelen(struct scatter_walk *walk) -{ - unsigned int len = walk->sg->offset + walk->sg->length - walk->offset; - unsigned int len_this_page = offset_in_page(~walk->offset) + 1; - return len_this_page > len ? len : len_this_page; -} - static inline unsigned int scatterwalk_clamp(struct scatter_walk *walk, unsigned int nbytes) { + unsigned int len_this_sg; + unsigned int limit; + if (walk->offset >= walk->sg->offset + walk->sg->length) scatterwalk_start(walk, sg_next(walk->sg)); - return min(nbytes, scatterwalk_pagelen(walk)); -} + len_this_sg = walk->sg->offset + walk->sg->length - walk->offset; -static inline struct page *scatterwalk_page(struct scatter_walk *walk) -{ - return sg_page(walk->sg) + (walk->offset >> PAGE_SHIFT); + /* + * HIGHMEM case: the page may have to be mapped into memory. To avoid + * the complexity of having to map multiple pages at once per sg entry, + * clamp the returned length to not cross a page boundary. + * + * !HIGHMEM case: no mapping is needed; all pages of the sg entry are + * already mapped contiguously in the kernel's direct map. For improved + * performance, allow the walker to return data segments that cross a + * page boundary. Do still cap the length to PAGE_SIZE, since some + * users rely on that to avoid disabling preemption for too long when + * using SIMD. It's also needed for when skcipher_walk uses a bounce + * page due to the data not being aligned to the algorithm's alignmask. + */ + if (IS_ENABLED(CONFIG_HIGHMEM)) + limit = PAGE_SIZE - offset_in_page(walk->offset); + else + limit = PAGE_SIZE; + + return min3(nbytes, len_this_sg, limit); } /* * Create a scatterlist that represents the remaining data in a walk. Uses * chaining to reference the original scatterlist, so this uses at most two @@ -84,19 +95,27 @@ static inline void scatterwalk_get_sglist(struct scatter_walk *walk, walk->sg->offset + walk->sg->length - walk->offset, walk->offset); scatterwalk_crypto_chain(sg_out, sg_next(walk->sg), 2); } -static inline void scatterwalk_unmap(void *vaddr) -{ - kunmap_local(vaddr); -} - static inline void *scatterwalk_map(struct scatter_walk *walk) { - return kmap_local_page(scatterwalk_page(walk)) + - offset_in_page(walk->offset); + struct page *base_page = sg_page(walk->sg); + + if (IS_ENABLED(CONFIG_HIGHMEM)) + return kmap_local_page(base_page + (walk->offset >> PAGE_SHIFT)) + + offset_in_page(walk->offset); + /* + * When !HIGHMEM we allow the walker to return segments that span a page + * boundary; see scatterwalk_clamp(). To make it clear that in this + * case we're working in the linear buffer of the whole sg entry in the + * kernel's direct map rather than within the mapped buffer of a single + * page, compute the address as an offset from the page_address() of the + * first page of the sg entry. Either way the result is the address in + * the direct map, but this makes it clearer what is really going on. + */ + return page_address(base_page) + walk->offset; } /** * scatterwalk_next() - Get the next data buffer in a scatterlist walk * @walk: the scatter_walk @@ -113,10 +132,16 @@ static inline void *scatterwalk_next(struct scatter_walk *walk, { *nbytes_ret = scatterwalk_clamp(walk, total); return scatterwalk_map(walk); } +static inline void scatterwalk_unmap(const void *vaddr) +{ + if (IS_ENABLED(CONFIG_HIGHMEM)) + kunmap_local(vaddr); +} + static inline void scatterwalk_advance(struct scatter_walk *walk, unsigned int nbytes) { walk->offset += nbytes; } @@ -131,11 +156,11 @@ static inline void scatterwalk_advance(struct scatter_walk *walk, * Use this if the @vaddr was not written to, i.e. it is source data. */ static inline void scatterwalk_done_src(struct scatter_walk *walk, const void *vaddr, unsigned int nbytes) { - scatterwalk_unmap((void *)vaddr); + scatterwalk_unmap(vaddr); scatterwalk_advance(walk, nbytes); } /** * scatterwalk_done_dst() - Finish one step of a walk of destination scatterlist @@ -152,13 +177,23 @@ static inline void scatterwalk_done_dst(struct scatter_walk *walk, scatterwalk_unmap(vaddr); /* * Explicitly check ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE instead of just * relying on flush_dcache_page() being a no-op when not implemented, * since otherwise the BUG_ON in sg_page() does not get optimized out. + * This also avoids having to consider whether the loop would get + * reliably optimized out or not. */ - if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE) - flush_dcache_page(scatterwalk_page(walk)); + if (ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE) { + struct page *base_page, *start_page, *end_page, *page; + + base_page = sg_page(walk->sg); + start_page = base_page + (walk->offset >> PAGE_SHIFT); + end_page = base_page + ((walk->offset + nbytes + + PAGE_SIZE - 1) >> PAGE_SHIFT); + for (page = start_page; page < end_page; page++) + flush_dcache_page(page); + } scatterwalk_advance(walk, nbytes); } void scatterwalk_skip(struct scatter_walk *walk, unsigned int nbytes);