From patchwork Wed Oct 25 08:58:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yael Chemla X-Patchwork-Id: 10026235 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A17906032C for ; Wed, 25 Oct 2017 09:43:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E5C628AF9 for ; Wed, 25 Oct 2017 09:43:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 935AA28B3E; Wed, 25 Oct 2017 09:43:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BBA4728AF9 for ; Wed, 25 Oct 2017 09:43:31 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D248962E82; Wed, 25 Oct 2017 09:43:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com D248962E82 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=foss.arm.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com D248962E82 Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8F86661F3C; Wed, 25 Oct 2017 09:43:30 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 559EE1804485; Wed, 25 Oct 2017 09:43:30 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v9P8wWBh025794 for ; Wed, 25 Oct 2017 04:58:32 -0400 Received: by smtp.corp.redhat.com (Postfix) id B217617578; Wed, 25 Oct 2017 08:58:32 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from mx1.redhat.com (ext-mx06.extmail.prod.ext.phx2.redhat.com [10.5.110.30]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AA17D173F4 for ; Wed, 25 Oct 2017 08:58:30 +0000 (UTC) Received: from foss.arm.com (usa-sjc-mx-foss1.foss.arm.com [217.140.101.70]) by mx1.redhat.com (Postfix) with ESMTP id 2A7221B5E45 for ; Wed, 25 Oct 2017 08:58:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 2A7221B5E45 Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=foss.arm.com Authentication-Results: ext-mx06.extmail.prod.ext.phx2.redhat.com; spf=none smtp.mailfrom=yael.chemla@foss.arm.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 2A7221B5E45 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0BCDC15AD for ; Wed, 25 Oct 2017 01:58:29 -0700 (PDT) Received: from ssg-dev-vb.kfn.arm.com (unknown [10.45.48.52]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3E40E3F246; Wed, 25 Oct 2017 01:58:28 -0700 (PDT) From: Yael Chemla To: dm-devel@redhat.com Date: Wed, 25 Oct 2017 09:58:19 +0100 Message-Id: <1508921899-24801-1-git-send-email-yael.chemla@foss.arm.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 205 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 25 Oct 2017 08:58:29 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Wed, 25 Oct 2017 08:58:29 +0000 (UTC) for IP:'217.140.101.70' DOMAIN:'usa-sjc-mx-foss1.foss.arm.com' HELO:'foss.arm.com' FROM:'yael.chemla@foss.arm.com' RCPT:'' X-RedHat-Spam-Score: -5.001 (RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD) 217.140.101.70 usa-sjc-mx-foss1.foss.arm.com 217.140.101.70 usa-sjc-mx-foss1.foss.arm.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.30 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Wed, 25 Oct 2017 05:41:58 -0400 Cc: Yael Chemla Subject: [dm-devel] [PATCH] md: dm-verity: aggregate crypto API calls X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 25 Oct 2017 09:43:31 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Current implementation makes multiple crypto API calls for a single block, forcing underlying crypto tfm implementation to "stop & go", leading to under utilization of CPU (e.g. SIMD state saves) or HW engines. To fix it unify calls to crypto init/update/final into a digest call with a single sg which contains multiple buffers. This also opens the way later on to parallelizing the work on different blocks. Tested on physical Arm 32 and x86_64 and virtual ARM 64. Performance numbers were not changed. Signed-off-by: Yael Chemla --- drivers/md/dm-verity-target.c | 226 +++++++++++++++++++++++++----------------- 1 file changed, 137 insertions(+), 89 deletions(-) diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index bda3caca23ca..c8758443ef64 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -35,10 +35,18 @@ #define DM_VERITY_OPTS_MAX (2 + DM_VERITY_OPTS_FEC) +/* only two elements in static scatter list: salt and data */ +#define SG_FIXED_ITEMS 2 + static unsigned dm_verity_prefetch_cluster = DM_VERITY_DEFAULT_PREFETCH_SIZE; module_param_named(prefetch_cluster, dm_verity_prefetch_cluster, uint, S_IRUGO | S_IWUSR); +enum salt_location { + START_SG, + END_SG +}; + struct dm_verity_prefetch_work { struct work_struct work; struct dm_verity *v; @@ -133,80 +141,67 @@ static inline int verity_complete_op(struct verity_result *res, int ret) return ret; } -static int verity_hash_update(struct dm_verity *v, struct ahash_request *req, - const u8 *data, size_t len, - struct verity_result *res) -{ - struct scatterlist sg; - - sg_init_one(&sg, data, len); - ahash_request_set_crypt(req, &sg, NULL, len); - - return verity_complete_op(res, crypto_ahash_update(req)); -} - /* - * Wrapper for crypto_ahash_init, which handles verity salting. + * verity_is_salt_required - check if according to verity version and + * verity salt's size there's a need to insert a salt. + * note: verity's version indicates where the salt should be added. + * (before or after buffer) + * @where - START_SG - before buffer / END_SG - after buffer */ -static int verity_hash_init(struct dm_verity *v, struct ahash_request *req, - struct verity_result *res) +static inline bool verity_is_salt_required(struct dm_verity *v, + enum salt_location where) { - int r; - - ahash_request_set_tfm(req, v->tfm); - ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP | - CRYPTO_TFM_REQ_MAY_BACKLOG, - verity_op_done, (void *)res); - init_completion(&res->completion); - - r = verity_complete_op(res, crypto_ahash_init(req)); - - if (unlikely(r < 0)) { - DMERR("crypto_ahash_init failed: %d", r); - return r; + if (likely(v->salt_size) && + ((where == START_SG && likely(v->version >= 1)) || + (where == END_SG && unlikely(!v->version)))) { + return true; } - - if (likely(v->salt_size && (v->version >= 1))) - r = verity_hash_update(v, req, v->salt, v->salt_size, res); - - return r; + return false; } -static int verity_hash_final(struct dm_verity *v, struct ahash_request *req, - u8 *digest, struct verity_result *res) +/* + * verity_add_salt - add verity's salt into a scatterlist + * @nents - number of elements already inserted into sg + * @total_len - total number of items in scatterlist array + */ +static void verity_add_salt(struct dm_verity *v, struct scatterlist *sg, + unsigned int *nents, unsigned int *total_len) { - int r; - - if (unlikely(v->salt_size && (!v->version))) { - r = verity_hash_update(v, req, v->salt, v->salt_size, res); - - if (r < 0) { - DMERR("verity_hash_final failed updating salt: %d", r); - goto out; - } - } - - ahash_request_set_crypt(req, NULL, digest, 0); - r = verity_complete_op(res, crypto_ahash_final(req)); -out: - return r; + sg_set_buf(&sg[*nents], v->salt, v->salt_size); + (*nents)++; + (*total_len) += v->salt_size; } int verity_hash(struct dm_verity *v, struct ahash_request *req, const u8 *data, size_t len, u8 *digest) { - int r; + int r, total_len = 0, indx = 0; struct verity_result res; + struct scatterlist sg[SG_FIXED_ITEMS]; - r = verity_hash_init(v, req, &res); - if (unlikely(r < 0)) - goto out; + sg_init_table(sg, SG_FIXED_ITEMS); + ahash_request_set_tfm(req, v->tfm); + ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP | + CRYPTO_TFM_REQ_MAY_BACKLOG, + verity_op_done, (void *)&res); + if (verity_is_salt_required(v, START_SG) == true) + verity_add_salt(v, sg, &indx, &total_len); - r = verity_hash_update(v, req, data, len, &res); - if (unlikely(r < 0)) - goto out; + sg_set_buf(&sg[indx], data, len); + indx++; + total_len += len; + if (verity_is_salt_required(v, END_SG) == true) + verity_add_salt(v, sg, &indx, &total_len); + + ahash_request_set_crypt(req, sg, digest, len+v->salt_size); + init_completion(&res.completion); - r = verity_hash_final(v, req, digest, &res); + r = verity_complete_op(&res, crypto_ahash_digest(req)); + + if (unlikely(r < 0)) { + DMERR("ahash_request_set_crypt failed: %d", r); + goto out; + } out: return r; @@ -389,20 +384,17 @@ int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io, * Calculates the digest for the given bio */ int verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io, - struct bvec_iter *iter, struct verity_result *res) + struct bvec_iter *iter, struct verity_result *res, + struct scatterlist *sg, unsigned int *nents, + unsigned int *total_len) { unsigned int todo = 1 << v->data_dev_block_bits; struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); - struct scatterlist sg; - struct ahash_request *req = verity_io_hash_req(v, io); do { - int r; unsigned int len; struct bio_vec bv = bio_iter_iovec(bio, *iter); - sg_init_table(&sg, 1); - len = bv.bv_len; if (likely(len >= todo)) @@ -412,22 +404,43 @@ int verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io, * until you consider the typical block size is 4,096B. * Going through this loops twice should be very rare. */ - sg_set_page(&sg, bv.bv_page, len, bv.bv_offset); - ahash_request_set_crypt(req, &sg, NULL, len); - r = verity_complete_op(res, crypto_ahash_update(req)); - - if (unlikely(r < 0)) { - DMERR("verity_for_io_block crypto op failed: %d", r); - return r; - } + sg_set_page(&sg[*nents], bv.bv_page, len, bv.bv_offset); bio_advance_iter(bio, iter, len); todo -= len; + (*nents)++; + (*total_len) += len; } while (todo); return 0; } +/* calculate how many buffers required to accomudate bio_vec starting + * from iter + */ +unsigned int verity_calc_buffs_for_bv(struct dm_verity *v, + struct dm_verity_io *io, struct bvec_iter iter) +{ + unsigned int todo = 1 << v->data_dev_block_bits; + struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); + unsigned int buff_count = 0; + + do { + unsigned int len; + struct bio_vec bv = bio_iter_iovec(bio, iter); + + len = bv.bv_len; + if (likely(len >= todo)) + len = todo; + + bio_advance_iter(bio, &iter, len); + todo -= len; + buff_count++; + } while (todo); + + return buff_count; +} + /* * Calls function process for 1 << v->data_dev_block_bits bytes in the bio_vec * starting from iter. @@ -483,16 +496,30 @@ static int verity_verify_io(struct dm_verity_io *io) struct bvec_iter start; unsigned b; struct verity_result res; + struct scatterlist *sg; + int r; for (b = 0; b < io->n_blocks; b++) { - int r; + unsigned int nents; + unsigned int total_len = 0; + unsigned int num_of_buffs = 0; struct ahash_request *req = verity_io_hash_req(v, io); + /* an extra one for the salt buffer */ + num_of_buffs = verity_calc_buffs_for_bv(v, io, io->iter) + 1; + WARN_ON(num_of_buffs < 1); + + sg = kmalloc_array(num_of_buffs, sizeof(struct scatterlist), + GFP_KERNEL); + if (!sg) + return -ENOMEM; + sg_init_table(sg, num_of_buffs); + r = verity_hash_for_block(v, io, io->block + b, verity_io_want_digest(v, io), &is_zero); if (unlikely(r < 0)) - return r; + goto err_memfree; if (is_zero) { /* @@ -502,37 +529,58 @@ static int verity_verify_io(struct dm_verity_io *io) r = verity_for_bv_block(v, io, &io->iter, verity_bv_zero); if (unlikely(r < 0)) - return r; + goto err_memfree; continue; } - r = verity_hash_init(v, req, &res); - if (unlikely(r < 0)) - return r; + ahash_request_set_tfm(req, v->tfm); + ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP | + CRYPTO_TFM_REQ_MAY_BACKLOG, + verity_op_done, (void *)&res); - start = io->iter; - r = verity_for_io_block(v, io, &io->iter, &res); - if (unlikely(r < 0)) - return r; + nents = 0; + total_len = 0; + if (verity_is_salt_required(v, START_SG) == true) + verity_add_salt(v, sg, &nents, &total_len); - r = verity_hash_final(v, req, verity_io_real_digest(v, io), - &res); - if (unlikely(r < 0)) - return r; + start = io->iter; + verity_for_io_block(v, io, &io->iter, &res, sg, &nents, + &total_len); + if (verity_is_salt_required(v, END_SG) == true) + verity_add_salt(v, sg, &nents, &total_len); + /* + * need to mark end of chain, since we might have allocated + * more than we actually use + */ + sg_mark_end(&sg[nents-1]); + ahash_request_set_crypt(req, sg, verity_io_real_digest(v, io), + total_len); + init_completion(&res.completion); + r = verity_complete_op(&res, crypto_ahash_digest(req)); + kfree(sg); + if (unlikely(r < 0)) { + DMERR("crypto_ahash_digest failed: %d", r); + goto err_memfree; + } if (likely(memcmp(verity_io_real_digest(v, io), - verity_io_want_digest(v, io), v->digest_size) == 0)) + verity_io_want_digest(v, io), v->digest_size) == 0)) continue; else if (verity_fec_decode(v, io, DM_VERITY_BLOCK_TYPE_DATA, io->block + b, NULL, &start) == 0) continue; else if (verity_handle_err(v, DM_VERITY_BLOCK_TYPE_DATA, - io->block + b)) - return -EIO; + io->block + b)) { + goto err_memfree; + } } return 0; + +err_memfree: + kfree(sg); + return r; } /*