From patchwork Fri Jun 26 07:09:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Song Bao Hua (Barry Song)" X-Patchwork-Id: 11626629 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DACCA14E3 for ; Fri, 26 Jun 2020 07:11:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A41AA20768 for ; Fri, 26 Jun 2020 07:11:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A41AA20768 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=hisilicon.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B01636B0006; Fri, 26 Jun 2020 03:11:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AB24F6B0007; Fri, 26 Jun 2020 03:11:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A1266B0008; Fri, 26 Jun 2020 03:11:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 7EDB06B0006 for ; Fri, 26 Jun 2020 03:11:02 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 19DF333CD for ; Fri, 26 Jun 2020 07:11:02 +0000 (UTC) X-FDA: 76970491164.30.chain02_0f0359f26e53 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id E5D0E180B3AA7 for ; Fri, 26 Jun 2020 07:11:01 +0000 (UTC) X-Spam-Summary: 1,0,0,08e96b7fbe98a5d6,d41d8cd98f00b204,song.bao.hua@hisilicon.com,,RULES_HIT:1:41:69:355:379:541:582:800:960:966:968:973:988:989:1152:1260:1261:1277:1311:1313:1314:1345:1431:1437:1515:1516:1518:1605:1730:1747:1777:1792:2196:2199:2393:2559:2562:2637:2693:3138:3139:3140:3141:3142:3622:3865:3866:3867:3868:3870:3871:3874:4250:4321:4385:4605:5007:6117:6119:6261:6742:7514:8603:9010:9040:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12679:12683:12895:12986:13894:14096:14097:14394:21080:21324:21451:21627:21740:21987:21990:30017:30054,0,RBL:45.249.212.190:@hisilicon.com:.lbl8.mailshell.net-64.201.201.201 62.14.2.100;04ygtaj8qb8r7mjw8pzhuhxzb9czmyckaks44uy4ytmqss8dzx7je4xdf5zdi5j.7itw8ngsf7np5y4kgsqmd4stxskfb9g6d93geen3wdns5wnoempf8dgh9r8ob54.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: chain02_0f0359f26e53 X-Filterd-Recvd-Size: 13121 Received: from huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 26 Jun 2020 07:11:00 +0000 (UTC) Received: from DGGEMS402-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id C3743B90A9A25EF6118C; Fri, 26 Jun 2020 15:10:54 +0800 (CST) Received: from SWX921481.china.huawei.com (10.126.200.84) by DGGEMS402-HUB.china.huawei.com (10.3.19.202) with Microsoft SMTP Server id 14.3.487.0; Fri, 26 Jun 2020 15:10:43 +0800 From: Barry Song To: CC: , , , Barry Song , "Luis Claudio R . Goncalves" , Sebastian Andrzej Siewior , Herbert Xu , "David S . Miller" , Mahipal Challa , Seth Jennings , Dan Streetman , Vitaly Wool , Zhou Wang , Colin Ian King Subject: [PATCH v3] mm/zswap: move to use crypto_acomp API for hardware acceleration Date: Fri, 26 Jun 2020 19:09:03 +1200 Message-ID: <20200626070903.27988-1-song.bao.hua@hisilicon.com> X-Mailer: git-send-email 2.21.0.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.126.200.84] X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: E5D0E180B3AA7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: right now, all new ZIP drivers are using crypto_acomp APIs rather than legacy crypto_comp APIs. But zswap.c is still using the old APIs. That means zswap won't be able to use any new zip drivers in kernel. This patch moves to use cryto_acomp APIs to fix the problem. On the other hand, tradiontal compressors like lz4,lzo etc have been wrapped into acomp via scomp backend. So platforms without async compressors can fallback to use acomp via scomp backend. Cc: Luis Claudio R. Goncalves Cc: Sebastian Andrzej Siewior Cc: Andrew Morton Cc: Herbert Xu Cc: David S. Miller Cc: Mahipal Challa Cc: Seth Jennings Cc: Dan Streetman Cc: Vitaly Wool Cc: Zhou Wang Cc: Colin Ian King Signed-off-by: Barry Song --- -v3: fix resource leak in error handle path; remove redundant assignment according to Colin King, thanks! mm/zswap.c | 160 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 117 insertions(+), 43 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index fbb782924ccc..d170ef06c693 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -24,8 +24,10 @@ #include #include #include +#include #include #include +#include #include #include @@ -127,9 +129,17 @@ module_param_named(same_filled_pages_enabled, zswap_same_filled_pages_enabled, * data structures **********************************/ +struct crypto_acomp_ctx { + struct crypto_acomp *acomp; + struct acomp_req *req; + struct crypto_wait wait; + u8 *dstmem; + struct mutex mutex; +}; + struct zswap_pool { struct zpool *zpool; - struct crypto_comp * __percpu *tfm; + struct crypto_acomp_ctx * __percpu *acomp_ctx; struct kref kref; struct list_head list; struct work_struct release_work; @@ -415,30 +425,68 @@ static int zswap_dstmem_dead(unsigned int cpu) static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_comp *tfm; + struct crypto_acomp *acomp; + struct acomp_req *req; + struct crypto_acomp_ctx *acomp_ctx; + int ret; - if (WARN_ON(*per_cpu_ptr(pool->tfm, cpu))) + if (WARN_ON(*per_cpu_ptr(pool->acomp_ctx, cpu))) return 0; - tfm = crypto_alloc_comp(pool->tfm_name, 0, 0); - if (IS_ERR_OR_NULL(tfm)) { - pr_err("could not alloc crypto comp %s : %ld\n", - pool->tfm_name, PTR_ERR(tfm)); + acomp_ctx = kzalloc(sizeof(*acomp_ctx), GFP_KERNEL); + if (!acomp_ctx) return -ENOMEM; + + acomp = crypto_alloc_acomp(pool->tfm_name, 0, 0); + if (IS_ERR(acomp)) { + pr_err("could not alloc crypto acomp %s : %ld\n", + pool->tfm_name, PTR_ERR(acomp)); + ret = PTR_ERR(acomp); + goto free_ctx; + } + acomp_ctx->acomp = acomp; + + req = acomp_request_alloc(acomp_ctx->acomp); + if (!req) { + pr_err("could not alloc crypto acomp_request %s\n", + pool->tfm_name); + ret = -ENOMEM; + goto free_acomp; } - *per_cpu_ptr(pool->tfm, cpu) = tfm; + acomp_ctx->req = req; + + mutex_init(&acomp_ctx->mutex); + crypto_init_wait(&acomp_ctx->wait); + acomp_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, &acomp_ctx->wait); + + acomp_ctx->dstmem = per_cpu(zswap_dstmem, cpu); + *per_cpu_ptr(pool->acomp_ctx, cpu) = acomp_ctx; + return 0; + +free_acomp: + crypto_free_acomp(acomp_ctx->acomp); +free_ctx: + kfree(acomp_ctx); + return ret; } static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); - struct crypto_comp *tfm; + struct crypto_acomp_ctx *acomp_ctx; + + acomp_ctx = *per_cpu_ptr(pool->acomp_ctx, cpu); + if (!IS_ERR_OR_NULL(acomp_ctx)) { + if (!IS_ERR_OR_NULL(acomp_ctx->req)) + acomp_request_free(acomp_ctx->req); + if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) + crypto_free_acomp(acomp_ctx->acomp); + kfree(acomp_ctx); + } + *per_cpu_ptr(pool->acomp_ctx, cpu) = NULL; - tfm = *per_cpu_ptr(pool->tfm, cpu); - if (!IS_ERR_OR_NULL(tfm)) - crypto_free_comp(tfm); - *per_cpu_ptr(pool->tfm, cpu) = NULL; return 0; } @@ -561,8 +609,9 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) pr_debug("using %s zpool\n", zpool_get_type(pool->zpool)); strlcpy(pool->tfm_name, compressor, sizeof(pool->tfm_name)); - pool->tfm = alloc_percpu(struct crypto_comp *); - if (!pool->tfm) { + + pool->acomp_ctx = alloc_percpu(struct crypto_acomp_ctx *); + if (!pool->acomp_ctx) { pr_err("percpu alloc failed\n"); goto error; } @@ -585,7 +634,7 @@ static struct zswap_pool *zswap_pool_create(char *type, char *compressor) return pool; error: - free_percpu(pool->tfm); + free_percpu(pool->acomp_ctx); if (pool->zpool) zpool_destroy_pool(pool->zpool); kfree(pool); @@ -596,14 +645,14 @@ static __init struct zswap_pool *__zswap_pool_create_fallback(void) { bool has_comp, has_zpool; - has_comp = crypto_has_comp(zswap_compressor, 0, 0); + has_comp = crypto_has_acomp(zswap_compressor, 0, 0); if (!has_comp && strcmp(zswap_compressor, CONFIG_ZSWAP_COMPRESSOR_DEFAULT)) { pr_err("compressor %s not available, using default %s\n", zswap_compressor, CONFIG_ZSWAP_COMPRESSOR_DEFAULT); param_free_charp(&zswap_compressor); zswap_compressor = CONFIG_ZSWAP_COMPRESSOR_DEFAULT; - has_comp = crypto_has_comp(zswap_compressor, 0, 0); + has_comp = crypto_has_acomp(zswap_compressor, 0, 0); } if (!has_comp) { pr_err("default compressor %s not available\n", @@ -639,7 +688,7 @@ static void zswap_pool_destroy(struct zswap_pool *pool) zswap_pool_debug("destroying", pool); cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node); - free_percpu(pool->tfm); + free_percpu(pool->acomp_ctx); zpool_destroy_pool(pool->zpool); kfree(pool); } @@ -723,7 +772,7 @@ static int __zswap_param_set(const char *val, const struct kernel_param *kp, } type = s; } else if (!compressor) { - if (!crypto_has_comp(s, 0, 0)) { + if (!crypto_has_acomp(s, 0, 0)) { pr_err("compressor %s not available\n", s); return -ENOENT; } @@ -774,7 +823,7 @@ static int __zswap_param_set(const char *val, const struct kernel_param *kp, * failed, maybe both compressor and zpool params were bad. * Allow changing this param, so pool creation will succeed * when the other param is changed. We already verified this - * param is ok in the zpool_has_pool() or crypto_has_comp() + * param is ok in the zpool_has_pool() or crypto_has_acomp() * checks above. */ ret = param_set_charp(s, kp); @@ -876,7 +925,9 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) pgoff_t offset; struct zswap_entry *entry; struct page *page; - struct crypto_comp *tfm; + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; + u8 *src, *dst; unsigned int dlen; int ret; @@ -916,14 +967,21 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) case ZSWAP_SWAPCACHE_NEW: /* page is locked */ /* decompress */ + acomp_ctx = *this_cpu_ptr(entry->pool->acomp_ctx); + dlen = PAGE_SIZE; src = (u8 *)zhdr + sizeof(struct zswap_header); - dst = kmap_atomic(page); - tfm = *get_cpu_ptr(entry->pool->tfm); - ret = crypto_comp_decompress(tfm, src, entry->length, - dst, &dlen); - put_cpu_ptr(entry->pool->tfm); - kunmap_atomic(dst); + dst = kmap(page); + + mutex_lock(&acomp_ctx->mutex); + sg_init_one(&input, src, entry->length); + sg_init_one(&output, dst, dlen); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); + ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + mutex_unlock(&acomp_ctx->mutex); + + kunmap(page); BUG_ON(ret); BUG_ON(dlen != PAGE_SIZE); @@ -1004,7 +1062,8 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, { struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry, *dupentry; - struct crypto_comp *tfm; + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; int ret; unsigned int hlen, dlen = PAGE_SIZE; unsigned long handle, value; @@ -1074,12 +1133,20 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, } /* compress */ - dst = get_cpu_var(zswap_dstmem); - tfm = *get_cpu_ptr(entry->pool->tfm); - src = kmap_atomic(page); - ret = crypto_comp_compress(tfm, src, PAGE_SIZE, dst, &dlen); - kunmap_atomic(src); - put_cpu_ptr(entry->pool->tfm); + acomp_ctx = *this_cpu_ptr(entry->pool->acomp_ctx); + + mutex_lock(&acomp_ctx->mutex); + + src = kmap(page); + dst = acomp_ctx->dstmem; + sg_init_one(&input, src, PAGE_SIZE); + /* zswap_dstmem is of size (PAGE_SIZE * 2). Reflect same in sg_list */ + sg_init_one(&output, dst, PAGE_SIZE * 2); + acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); + ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); + dlen = acomp_ctx->req->dlen; + kunmap(page); + if (ret) { ret = -EINVAL; goto put_dstmem; @@ -1103,7 +1170,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, memcpy(buf, &zhdr, hlen); memcpy(buf + hlen, dst, dlen); zpool_unmap_handle(entry->pool->zpool, handle); - put_cpu_var(zswap_dstmem); + mutex_unlock(&acomp_ctx->mutex); /* populate entry */ entry->offset = offset; @@ -1131,7 +1198,7 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset, return 0; put_dstmem: - put_cpu_var(zswap_dstmem); + mutex_unlock(&acomp_ctx->mutex); zswap_pool_put(entry->pool); freepage: zswap_entry_cache_free(entry); @@ -1148,7 +1215,8 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, { struct zswap_tree *tree = zswap_trees[type]; struct zswap_entry *entry; - struct crypto_comp *tfm; + struct scatterlist input, output; + struct crypto_acomp_ctx *acomp_ctx; u8 *src, *dst; unsigned int dlen; int ret; @@ -1175,11 +1243,17 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, src = zpool_map_handle(entry->pool->zpool, entry->handle, ZPOOL_MM_RO); if (zpool_evictable(entry->pool->zpool)) src += sizeof(struct zswap_header); - dst = kmap_atomic(page); - tfm = *get_cpu_ptr(entry->pool->tfm); - ret = crypto_comp_decompress(tfm, src, entry->length, dst, &dlen); - put_cpu_ptr(entry->pool->tfm); - kunmap_atomic(dst); + dst = kmap(page); + + acomp_ctx = *this_cpu_ptr(entry->pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); + sg_init_one(&input, src, entry->length); + sg_init_one(&output, dst, dlen); + acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); + ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); + mutex_unlock(&acomp_ctx->mutex); + + kunmap(page); zpool_unmap_handle(entry->pool->zpool, entry->handle); BUG_ON(ret);