From patchwork Wed Jan 3 09:50:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13509812 X-Patchwork-Delegate: herbert@gondor.apana.org.au Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C64E918C18; Wed, 3 Jan 2024 09:50:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ldjPPMSl" Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-dbe87cbc052so1124982276.2; Wed, 03 Jan 2024 01:50:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704275434; x=1704880234; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HwysZAJ6ghv3K37MmkiHDv50fMbQAcaQTVa/Nper+78=; b=ldjPPMSl1PG1zd92nHUJeehHW9xucdJ7r0i/VTZmdUP2P9qW44SFPGYK9Mo9IJACiE NMrHMnI1iP9oWztGlL4VHkczAbw9G1mB0oOoC23FFNaGrjK1dDZVPlXbo6TvgDE0AM4J BwvHd557gTCA7MYFUqBv51Nv4WLXwgqDAOpGucOZYFBsW9IjIXP+yWyJmAbm3coPEXZd jsnzNQdFMYgiwLmx80ubFROChudp1IydjZWF1ynd+pzxKtgLSop7vuRWk5+4xGlqn3PE oq7VuEn/4cIL7pMZ7dx/78uIU3tKnCLwl1AHBc0Sf1pDuvItEYwGU41tngkQ6/Hi2IKz wPgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704275434; x=1704880234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HwysZAJ6ghv3K37MmkiHDv50fMbQAcaQTVa/Nper+78=; b=e4bc1SIPesnJN9fkDw9jLAKVwWQIgo0+eCwdtg9pyadz9wHHg9fAvQNn33Ov/ZydkP BdkW2NOY+1J/WBhOR8EYG0gplkiDvgmMfMIPCNhrcHtc4p00LfN3xm8zRAdYwX7qcN1d H1h9Mxgm6wrNtl9/qZr58eH06AHqrUKSDYqr9Gshc5V6XU53kutE+I7TCFk61qE/xt61 Rw3IJ0wc7E87rztVigPolNvCesd/PXnn96G4Tcdc50n4gLRFMv075SjQbOp4xFGZJrdA //Bi017HpcOUgJ9qUhG4Mpz43dsYBXi3RYWWWDbm6gS3ooaJazPnr8Nesw/EBWim+5L2 Ix/A== X-Gm-Message-State: AOJu0YzBldatd7WDNqKN4/bUhVYZVw1PuXx4SJ4X+WsImvQrlEYOYGjO 4iQXuD2YDWlADNXWwyynvNc= X-Google-Smtp-Source: AGHT+IH42YZIV6OEBL93QwLYO40q0ASQ2KNUbZP1KhhoeQJlaCT6y4cGhqSERL4Av/vp/qrwoEL/nA== X-Received: by 2002:a25:b8b:0:b0:db5:4ece:ad01 with SMTP id 133-20020a250b8b000000b00db54ecead01mr8579026ybl.25.1704275433672; Wed, 03 Jan 2024 01:50:33 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:a7d6:f37a:9130:cd96]) by smtp.gmail.com with ESMTPSA id a34-20020a631a22000000b005c65e82a0cbsm21823483pga.17.2024.01.03.01.50.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 01:50:33 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: herbert@gondor.apana.org.au, davem@davemloft.net, akpm@linux-foundation.org, ddstreet@ieee.org, sjenning@redhat.com, vitaly.wool@konsulko.com, linux-crypto@vger.kernel.org Cc: chriscli@google.com, chrisl@kernel.org, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, yosryahmed@google.com, zhouchengming@bytedance.com, Barry Song Subject: [PATCH 1/3] crypto: introduce acomp_is_async to expose if a acomp has a scomp backend Date: Wed, 3 Jan 2024 22:50:04 +1300 Message-Id: <20240103095006.608744-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240103095006.608744-1-21cnbao@gmail.com> References: <20240103095006.608744-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Barry Song Almost all CPU-based compressors/decompressors are actually synchronous though they support acomp APIs. While some hardware has hardware-based accelerators to offload CPU's work such as hisilicon and intel/qat/, their drivers are working in async mode. Letting acomp's users know exactly if the acomp is really async will help users know if the compression and decompression procedure can sleep. Signed-off-by: Barry Song Tested-by: Chengming Zhou --- crypto/acompress.c | 8 ++++++++ include/crypto/acompress.h | 9 +++++++++ 2 files changed, 17 insertions(+) diff --git a/crypto/acompress.c b/crypto/acompress.c index 1c682810a484..99118e879a4a 100644 --- a/crypto/acompress.c +++ b/crypto/acompress.c @@ -152,6 +152,14 @@ struct crypto_acomp *crypto_alloc_acomp_node(const char *alg_name, u32 type, } EXPORT_SYMBOL_GPL(crypto_alloc_acomp_node); +bool acomp_is_async(struct crypto_acomp *acomp) +{ + struct crypto_tfm *tfm = crypto_acomp_tfm(acomp); + + return tfm->__crt_alg->cra_type == &crypto_acomp_type; +} +EXPORT_SYMBOL_GPL(acomp_is_async); + struct acomp_req *acomp_request_alloc(struct crypto_acomp *acomp) { struct crypto_tfm *tfm = crypto_acomp_tfm(acomp); diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h index 574cffc90730..d91830c2d442 100644 --- a/include/crypto/acompress.h +++ b/include/crypto/acompress.h @@ -204,6 +204,15 @@ struct acomp_req *acomp_request_alloc(struct crypto_acomp *tfm); */ void acomp_request_free(struct acomp_req *req); +/** + * acomp_is_async() -- check if an acomp is asynchronous(can sleep) + * + * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp() + * + * Return: true if the acomp is asynchronous, otherwise, false + */ +bool acomp_is_async(struct crypto_acomp *tfm); + /** * acomp_request_set_callback() -- Sets an asynchronous callback * From patchwork Wed Jan 3 09:50:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13509813 X-Patchwork-Delegate: herbert@gondor.apana.org.au Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D61118E0C; Wed, 3 Jan 2024 09:50:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VTRu68CT" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6d9bec20980so2479131b3a.2; Wed, 03 Jan 2024 01:50:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704275440; x=1704880240; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HF2P2SEmq7vRAtkf5d5h6lFP1PY7Yzig5CUd/10AwIg=; b=VTRu68CTFBoX1r0Mj1eTNq+EXifaN3iJsc1gO40YkQrHijbSbvHDsMwEQiuK2uPR0v eEy2MTuIU7ZJ7VJRIMeOy2tkaQAIJ2PkVEPPzipSzaww4UXcS/6y9Nf1cISOEwDhMiXW XK72tIQ0TkIO6Z8jDkF+xMHUMsYJ7a2+Mvd5jCBhe91OHy9LXQRpL08y7J3rjFxKe9o/ lJhzlVF1suxWh2uhSpFVLISKc/xxWV2BvwZj47DQfpGXmyigc36402nkY4b3SJdzBZKr S3Xm7pgmnWmSVbusBgNQ9USXOgMBbdmIgVGmSan+xCSHhzrjoJ0jrX7oJxGzTi3D5osD MjpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704275440; x=1704880240; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HF2P2SEmq7vRAtkf5d5h6lFP1PY7Yzig5CUd/10AwIg=; b=p2jnxtZdG/aM6y0jSNjjx4HgTzpP3U63bUFfWqemL/n0RCIl2hf5c70BwFlBvYuZGx T5IuLBTNGT0IHUXiKPse6yPxS7iDmMyu3mRvs/JOUK82SicQmUlU1Fv4x5gtATmIeSXQ 1Twgu5bbXh7SdOwYa+kkgE3HDhRSE9tquc00ASaj9c+da1XCQl4hCGTYLbECzBmmzVoH SzNdZlm18AMaGEFNP569KygghG7XgJrp+dEwhdEubeoR77NEZkgMIX1CQDnq/rfO2sWQ FMM0HApetX/TnCH2fIbNJOd0SrXPsuXIkDbCVKRLFBzvJbV96lEBrT8bURKCDVzQDZus /s4A== X-Gm-Message-State: AOJu0Yzbgs2kc9oot2B901oHR0IR+mfvM3gRL6rWm3oYncbJ4AwMvjo1 h4xrpDN92mhIa4/QLwWvOM8= X-Google-Smtp-Source: AGHT+IFhQWnb8yS8UAuFwFy8vb78sg6hucpOXjJ6Kq+ulrOpwt7NgLV10YadMn0i2Bik7OAWQ2a24g== X-Received: by 2002:a05:6a21:a585:b0:197:7997:b69 with SMTP id gd5-20020a056a21a58500b0019779970b69mr296409pzc.69.1704275440398; Wed, 03 Jan 2024 01:50:40 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:a7d6:f37a:9130:cd96]) by smtp.gmail.com with ESMTPSA id a34-20020a631a22000000b005c65e82a0cbsm21823483pga.17.2024.01.03.01.50.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 01:50:40 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: herbert@gondor.apana.org.au, davem@davemloft.net, akpm@linux-foundation.org, ddstreet@ieee.org, sjenning@redhat.com, vitaly.wool@konsulko.com, linux-crypto@vger.kernel.org Cc: chriscli@google.com, chrisl@kernel.org, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, yosryahmed@google.com, zhouchengming@bytedance.com, Barry Song Subject: [PATCH 2/3] mm/zswap: remove the memcpy if acomp is not asynchronous Date: Wed, 3 Jan 2024 22:50:05 +1300 Message-Id: <20240103095006.608744-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240103095006.608744-1-21cnbao@gmail.com> References: <20240103095006.608744-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Barry Song Most compressors are actually CPU-based and won't sleep during compression and decompression. We should remove the redundant memcpy for them. Signed-off-by: Barry Song Tested-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index ca25b676048e..36898614ebcc 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -168,6 +168,7 @@ struct crypto_acomp_ctx { struct crypto_wait wait; u8 *buffer; struct mutex mutex; + bool is_async; /* if acomp can sleep */ }; /* @@ -716,6 +717,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) goto acomp_fail; } acomp_ctx->acomp = acomp; + acomp_ctx->is_async = acomp_is_async(acomp); req = acomp_request_alloc(acomp_ctx->acomp); if (!req) { @@ -1370,7 +1372,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); - if (!zpool_can_sleep_mapped(zpool)) { + if (acomp_ctx->is_async && !zpool_can_sleep_mapped(zpool)) { memcpy(acomp_ctx->buffer, src, entry->length); src = acomp_ctx->buffer; zpool_unmap_handle(zpool, entry->handle); @@ -1384,7 +1386,7 @@ static void __zswap_load(struct zswap_entry *entry, struct page *page) BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); mutex_unlock(&acomp_ctx->mutex); - if (zpool_can_sleep_mapped(zpool)) + if (!acomp_ctx->is_async || zpool_can_sleep_mapped(zpool)) zpool_unmap_handle(zpool, entry->handle); } From patchwork Wed Jan 3 09:50:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13509814 X-Patchwork-Delegate: herbert@gondor.apana.org.au Received: from mail-oo1-f51.google.com (mail-oo1-f51.google.com [209.85.161.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0AB1818625; Wed, 3 Jan 2024 09:50:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S0RGhNyF" Received: by mail-oo1-f51.google.com with SMTP id 006d021491bc7-595ac2b6c59so920878eaf.2; Wed, 03 Jan 2024 01:50:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1704275447; x=1704880247; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Spb8sXhq6BQgZHawgszBp/yoNsRKTOzDWAXTsmTatzQ=; b=S0RGhNyFR7clISnCqdo9DeCZh1DBZmfSOtPWA2QtAOK7SuC8DUK77nawxsQN1daiAM Ys3k8LzZrlfza6QsQLsPUQfn5YhZEGD22tn/p+8dKj4uTW+mQOneoLAhGL7qb6QxtEf1 y7tuaJlen53L0FYJB/OR40/hrDTiiB7K+yO8ovZasAPSejIOTlDKrI3ccM8E7mc9jLtx y3j9RbJISQSkIUgMR1heqe/hgPSh/tKtKOyyRkleZNB/Q+ElOja3Tr4sGRQABe+43+c/ wrMz2BAVYLmWNcqtjAcN9TLjcBtKgP4DG1Rlb1M8KzmdWnYfI/1Jq0PEvZGeN8K+JsCm D9jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704275447; x=1704880247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Spb8sXhq6BQgZHawgszBp/yoNsRKTOzDWAXTsmTatzQ=; b=E87ZtfuLoV2nDr2fpciIc1OXpXO3HkDdMVFkUSwgUQqeeHCPUje2Awl8ra5SnvDXst NDd5klZXyp/2IG96BHCM+m24+TnW8lJ2OzHaeUfvrsVRTMi9d+5z0RzregIGnQSnLz96 FEV0DFFTe+PEfAXdqh7k00DinRMsCZ9MGK4YTjNEdMVUARIrcUvvIkHRW73n3JVq36mp las5TAZSF3kOQD7kfeTfL+g7pNAgv0ZxrAEJB88CyYWeFULAoB6O5mYI6FhuX/rWL7SV GxjgWBt5Ldxga1ml/tZEfuzKce+jZ9Yjwn5EI2wD46lQNyKwqHwiw/iphiQO7K8DqQRX vaNA== X-Gm-Message-State: AOJu0Yw/4QlamqbFL/ukPLTvno/oJfs2LZlxa935a5GROoRQDqtjM6Vf 7BYc8llFAAfX3mw4GIT79EY= X-Google-Smtp-Source: AGHT+IGWL4R/FI9TYmsj+r7+/K6fqa31rQlyfEGBvFn+SUDBIfMJqSQf7Q/5XUbHS6iSKuuDD2GKeA== X-Received: by 2002:a05:6358:7f1e:b0:172:e3f6:12c0 with SMTP id p30-20020a0563587f1e00b00172e3f612c0mr17022730rwn.46.1704275446906; Wed, 03 Jan 2024 01:50:46 -0800 (PST) Received: from barry-desktop.hub ([2407:7000:8942:5500:a7d6:f37a:9130:cd96]) by smtp.gmail.com with ESMTPSA id a34-20020a631a22000000b005c65e82a0cbsm21823483pga.17.2024.01.03.01.50.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 01:50:46 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: herbert@gondor.apana.org.au, davem@davemloft.net, akpm@linux-foundation.org, ddstreet@ieee.org, sjenning@redhat.com, vitaly.wool@konsulko.com, linux-crypto@vger.kernel.org Cc: chriscli@google.com, chrisl@kernel.org, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, yosryahmed@google.com, zhouchengming@bytedance.com, Barry Song Subject: [PATCH 3/3] crypto: scompress: remove memcpy if sg_nents is 1 Date: Wed, 3 Jan 2024 22:50:06 +1300 Message-Id: <20240103095006.608744-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240103095006.608744-1-21cnbao@gmail.com> References: <20240103095006.608744-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-crypto@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Barry Song while sg_nents is 1 which is always true for the current kernel as the only user - zswap is the case, we should remove two big memcpy. Signed-off-by: Barry Song Tested-by: Chengming Zhou --- crypto/scompress.c | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/crypto/scompress.c b/crypto/scompress.c index 442a82c9de7d..d1bb40ef83a2 100644 --- a/crypto/scompress.c +++ b/crypto/scompress.c @@ -117,6 +117,7 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) struct crypto_scomp *scomp = *tfm_ctx; void **ctx = acomp_request_ctx(req); struct scomp_scratch *scratch; + void *src, *dst; int ret; if (!req->src || !req->slen || req->slen > SCOMP_SCRATCH_SIZE) @@ -131,13 +132,26 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) scratch = raw_cpu_ptr(&scomp_scratch); spin_lock(&scratch->lock); - scatterwalk_map_and_copy(scratch->src, req->src, 0, req->slen, 0); + if (sg_nents(req->src) == 1) { + src = kmap_local_page(sg_page(req->src)) + req->src->offset; + } else { + scatterwalk_map_and_copy(scratch->src, req->src, 0, + req->slen, 0); + src = scratch->src; + } + + if (req->dst && sg_nents(req->dst) == 1) { + dst = kmap_local_page(sg_page(req->dst)) + req->dst->offset; + } else { + dst = scratch->dst; + } + if (dir) - ret = crypto_scomp_compress(scomp, scratch->src, req->slen, - scratch->dst, &req->dlen, *ctx); + ret = crypto_scomp_compress(scomp, src, req->slen, + dst, &req->dlen, *ctx); else - ret = crypto_scomp_decompress(scomp, scratch->src, req->slen, - scratch->dst, &req->dlen, *ctx); + ret = crypto_scomp_decompress(scomp, src, req->slen, + dst, &req->dlen, *ctx); if (!ret) { if (!req->dst) { req->dst = sgl_alloc(req->dlen, GFP_ATOMIC, NULL); @@ -146,10 +160,17 @@ static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir) goto out; } } - scatterwalk_map_and_copy(scratch->dst, req->dst, 0, req->dlen, - 1); + if (dst == scratch->dst) { + scatterwalk_map_and_copy(scratch->dst, req->dst, 0, + req->dlen, 1); + } } out: + if (src != scratch->src) + kunmap_local(src); + if (dst != scratch->dst) + kunmap_local(dst); + spin_unlock(&scratch->lock); return ret; }