From patchwork Sun Mar 25 18:41:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yael Chemla X-Patchwork-Id: 10307279 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EF47360212 for ; Mon, 26 Mar 2018 07:37:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE6CE205FD for ; Mon, 26 Mar 2018 07:37:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C1F5B2823D; Mon, 26 Mar 2018 07:37:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D26C2205FD for ; Mon, 26 Mar 2018 07:37:35 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E0AE7C04B923; Mon, 26 Mar 2018 07:37:34 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B6BF95C890; Mon, 26 Mar 2018 07:37:34 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 594364CA9C; Mon, 26 Mar 2018 07:37:34 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id w2PIfiG9016021 for ; Sun, 25 Mar 2018 14:41:44 -0400 Received: by smtp.corp.redhat.com (Postfix) id 2432C17DC0; Sun, 25 Mar 2018 18:41:44 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from mx1.redhat.com (ext-mx09.extmail.prod.ext.phx2.redhat.com [10.5.110.38]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5CEEB17D79; Sun, 25 Mar 2018 18:41:39 +0000 (UTC) Received: from foss.arm.com (usa-sjc-mx-foss1.foss.arm.com [217.140.101.70]) by mx1.redhat.com (Postfix) with ESMTP id F21EF4E4CA; Sun, 25 Mar 2018 18:41:37 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 706A01529; Sun, 25 Mar 2018 11:41:37 -0700 (PDT) Received: from ssg-dev-vb.kfn.arm.com (unknown [10.45.47.140]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 887C43F557; Sun, 25 Mar 2018 11:41:35 -0700 (PDT) From: Yael Chemla To: Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, linux-kernel@vger.kernel.org, ofir.drang@gmail.com, Yael Chemla Date: Sun, 25 Mar 2018 19:41:29 +0100 Message-Id: <1522003290-27243-1-git-send-email-yael.chemla@foss.arm.com> X-Greylist: Sender passed SPF test, Sender IP whitelisted by DNSRBL, ACL 207 matched, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Sun, 25 Mar 2018 18:41:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Sun, 25 Mar 2018 18:41:38 +0000 (UTC) for IP:'217.140.101.70' DOMAIN:'usa-sjc-mx-foss1.foss.arm.com' HELO:'foss.arm.com' FROM:'yael.chemla@foss.arm.com' RCPT:'' X-RedHat-Spam-Score: -5.01 (RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD) 217.140.101.70 usa-sjc-mx-foss1.foss.arm.com 217.140.101.70 usa-sjc-mx-foss1.foss.arm.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.38 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 26 Mar 2018 03:37:09 -0400 Cc: Yael Chemla Subject: [dm-devel] [PATCH 1/2] md: dm-verity: aggregate crypto API calls X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 26 Mar 2018 07:37:35 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Current implementation makes multiple crypto API calls per block implementation makes multiple crypto API calls per single block, forcing underlying crypto tfm implementation to "stop & go", leading to under utilization of HW engines. To fix it unify calls to crypto init/update/final into a digest call with a single sg which contains multiple buffers. This is also needed as an enabler for the next patch in the series. Signed-off-by: Yael Chemla --- drivers/md/dm-verity-target.c | 220 ++++++++++++++++++++++++------------------ 1 file changed, 127 insertions(+), 93 deletions(-) diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index aedb822..a281b83 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -35,10 +35,18 @@ #define DM_VERITY_OPTS_MAX (2 + DM_VERITY_OPTS_FEC) +/* only two elements in static scatter list: salt and data */ +#define SG_FIXED_ITEMS 2 + static unsigned dm_verity_prefetch_cluster = DM_VERITY_DEFAULT_PREFETCH_SIZE; module_param_named(prefetch_cluster, dm_verity_prefetch_cluster, uint, S_IRUGO | S_IWUSR); +enum salt_location { + START_SG, + END_SG +}; + struct dm_verity_prefetch_work { struct work_struct work; struct dm_verity *v; @@ -92,82 +100,68 @@ static sector_t verity_position_at_level(struct dm_verity *v, sector_t block, return block >> (level * v->hash_per_block_bits); } -static int verity_hash_update(struct dm_verity *v, struct ahash_request *req, - const u8 *data, size_t len, - struct crypto_wait *wait) -{ - struct scatterlist sg; - - sg_init_one(&sg, data, len); - ahash_request_set_crypt(req, &sg, NULL, len); - - return crypto_wait_req(crypto_ahash_update(req), wait); -} - /* - * Wrapper for crypto_ahash_init, which handles verity salting. + * verity_is_salt_required - check if according to verity version and + * verity salt's size if there's a need to insert a salt. + * note: salt goes last for 0th version and first for all others + * @where - START_SG - before buffer / END_SG - after buffer */ -static int verity_hash_init(struct dm_verity *v, struct ahash_request *req, - struct crypto_wait *wait) +static inline bool verity_is_salt_required(struct dm_verity *v, + enum salt_location where) { - int r; + /* No salt, no problem */ + if (likely(!v->salt_size)) + return false; - ahash_request_set_tfm(req, v->tfm); - ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP | - CRYPTO_TFM_REQ_MAY_BACKLOG, - crypto_req_done, (void *)wait); - crypto_init_wait(wait); - - r = crypto_wait_req(crypto_ahash_init(req), wait); - - if (unlikely(r < 0)) { - DMERR("crypto_ahash_init failed: %d", r); - return r; - } - - if (likely(v->salt_size && (v->version >= 1))) - r = verity_hash_update(v, req, v->salt, v->salt_size, wait); - - return r; + if (likely(v->version)) + return (where == START_SG); + else + return (where == END_SG); } -static int verity_hash_final(struct dm_verity *v, struct ahash_request *req, - u8 *digest, struct crypto_wait *wait) +/* + * verity_add_salt - add verity's salt into a scatterlist + * @nents - number of elements already inserted into sg + * @total_len - total number of items in scatterlist array + */ +static void verity_add_salt(struct dm_verity *v, struct scatterlist *sg, + unsigned int *nents, unsigned int *total_len) { - int r; - - if (unlikely(v->salt_size && (!v->version))) { - r = verity_hash_update(v, req, v->salt, v->salt_size, wait); - - if (r < 0) { - DMERR("verity_hash_final failed updating salt: %d", r); - goto out; - } - } - - ahash_request_set_crypt(req, NULL, digest, 0); - r = crypto_wait_req(crypto_ahash_final(req), wait); -out: - return r; + sg_set_buf(&sg[*nents], v->salt, v->salt_size); + (*nents)++; + (*total_len) += v->salt_size; } int verity_hash(struct dm_verity *v, struct ahash_request *req, const u8 *data, size_t len, u8 *digest) { - int r; + int r, total_len = 0, indx = 0; + struct scatterlist sg[SG_FIXED_ITEMS]; struct crypto_wait wait; - r = verity_hash_init(v, req, &wait); - if (unlikely(r < 0)) - goto out; + sg_init_table(sg, SG_FIXED_ITEMS); + ahash_request_set_tfm(req, v->tfm); + ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP | + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, (void *)&wait); + if (verity_is_salt_required(v, START_SG)) + verity_add_salt(v, sg, &indx, &total_len); - r = verity_hash_update(v, req, data, len, &wait); - if (unlikely(r < 0)) - goto out; + sg_set_buf(&sg[indx], data, len); + indx++; + total_len += len; + if (verity_is_salt_required(v, END_SG)) + verity_add_salt(v, sg, &indx, &total_len); - r = verity_hash_final(v, req, digest, &wait); + ahash_request_set_crypt(req, sg, digest, len + v->salt_size); + crypto_init_wait(&wait); -out: + r = crypto_wait_req(crypto_ahash_digest(req), &wait); + + if (unlikely(r < 0)) { + DMERR("crypto_ahash_digest failed: %d", r); + return r; + } return r; } @@ -347,44 +341,54 @@ int verity_hash_for_block(struct dm_verity *v, struct dm_verity_io *io, /* * Calculates the digest for the given bio */ -int verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io, - struct bvec_iter *iter, struct crypto_wait *wait) +void verity_for_io_block(struct dm_verity *v, struct dm_verity_io *io, + struct bvec_iter *iter, struct scatterlist *sg, + unsigned int *nents, unsigned int *total_len) { unsigned int todo = 1 << v->data_dev_block_bits; struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); - struct scatterlist sg; - struct ahash_request *req = verity_io_hash_req(v, io); do { - int r; unsigned int len; struct bio_vec bv = bio_iter_iovec(bio, *iter); - sg_init_table(&sg, 1); - len = bv.bv_len; if (likely(len >= todo)) len = todo; - /* - * Operating on a single page at a time looks suboptimal - * until you consider the typical block size is 4,096B. - * Going through this loops twice should be very rare. - */ - sg_set_page(&sg, bv.bv_page, len, bv.bv_offset); - ahash_request_set_crypt(req, &sg, NULL, len); - r = crypto_wait_req(crypto_ahash_update(req), wait); - - if (unlikely(r < 0)) { - DMERR("verity_for_io_block crypto op failed: %d", r); - return r; - } + sg_set_page(&sg[*nents], bv.bv_page, len, bv.bv_offset); bio_advance_iter(bio, iter, len); todo -= len; + (*nents)++; + (*total_len) += len; } while (todo); +} - return 0; +/* calculate how many buffers required to accomudate bio_vec starting + * from iter + */ +unsigned int verity_calc_buffs_for_bv(struct dm_verity *v, + struct dm_verity_io *io, struct bvec_iter iter) +{ + unsigned int todo = 1 << v->data_dev_block_bits; + struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); + unsigned int buff_count = 0; + + do { + unsigned int len; + struct bio_vec bv = bio_iter_iovec(bio, iter); + + len = bv.bv_len; + if (likely(len >= todo)) + len = todo; + + bio_advance_iter(bio, &iter, len); + todo -= len; + buff_count++; + } while (todo); + + return buff_count; } /* @@ -442,16 +446,30 @@ static int verity_verify_io(struct dm_verity_io *io) struct bvec_iter start; unsigned b; struct crypto_wait wait; + struct scatterlist *sg; + int r; for (b = 0; b < io->n_blocks; b++) { - int r; + unsigned int nents; + unsigned int total_len = 0; + unsigned int num_of_buffs = 0; struct ahash_request *req = verity_io_hash_req(v, io); + /* an extra one for the salt buffer */ + num_of_buffs = verity_calc_buffs_for_bv(v, io, io->iter) + 1; + WARN_ON(num_of_buffs < 1); + + sg = kmalloc_array(num_of_buffs, sizeof(struct scatterlist), + GFP_KERNEL); + if (!sg) + return -ENOMEM; + sg_init_table(sg, num_of_buffs); + r = verity_hash_for_block(v, io, io->block + b, verity_io_want_digest(v, io), &is_zero); if (unlikely(r < 0)) - return r; + goto err_memfree; if (is_zero) { /* @@ -461,25 +479,37 @@ static int verity_verify_io(struct dm_verity_io *io) r = verity_for_bv_block(v, io, &io->iter, verity_bv_zero); if (unlikely(r < 0)) - return r; + goto err_memfree; continue; } - r = verity_hash_init(v, req, &wait); - if (unlikely(r < 0)) - return r; + ahash_request_set_tfm(req, v->tfm); + ahash_request_set_callback(req, CRYPTO_TFM_REQ_MAY_SLEEP | + CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, (void *)&wait); + nents = 0; + total_len = 0; + if (verity_is_salt_required(v, START_SG)) + verity_add_salt(v, sg, &nents, &total_len); start = io->iter; - r = verity_for_io_block(v, io, &io->iter, &wait); - if (unlikely(r < 0)) - return r; + verity_for_io_block(v, io, &io->iter, sg, &nents, &total_len); + if (verity_is_salt_required(v, END_SG)) + verity_add_salt(v, sg, &nents, &total_len); + /* + * need to mark end of chain, since we might have allocated + * more than we actually use + */ + sg_mark_end(&sg[nents-1]); + ahash_request_set_crypt(req, sg, verity_io_real_digest(v, io), + total_len); + crypto_init_wait(&wait); + r = crypto_wait_req(crypto_ahash_digest(req), &wait); - r = verity_hash_final(v, req, verity_io_real_digest(v, io), - &wait); if (unlikely(r < 0)) - return r; - + goto err_memfree; + kfree(sg); if (likely(memcmp(verity_io_real_digest(v, io), verity_io_want_digest(v, io), v->digest_size) == 0)) continue; @@ -492,6 +522,10 @@ static int verity_verify_io(struct dm_verity_io *io) } return 0; + +err_memfree: + kfree(sg); + return r; } /*