From patchwork Mon Feb 19 20:27:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 13563086 X-Patchwork-Delegate: snitzer@redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36F4954BFA for ; Mon, 19 Feb 2024 20:27:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374464; cv=none; b=urZKzgc4RLSa1oKIk8+mXh0jGMWBttddhCgikFi9UmC217x399hJRIfeuR/arz9yrijNEe5YabOYy+AClz83T5kYWQ6ur8BC96KFTwwfARum1Yqeby3kpNzigvj4/k92zTziyapjzEnRjRIX5q+PGF0A0hhfsTXbJqTqSyM48W4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374464; c=relaxed/simple; bh=1SY5s5odwrmeKC3lqMnA42I0IQF+B+XBjmN94TubDzg=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=q/eV/aMKsozsiJP12Vz2vRtWen3HTfthgfFzATwx9IdL1hBjBEg3AzukLcLbxwr9CLxdInxWFSENvJ/3sTJKqUBOHw8H/yb5ZRgKC1CRuFje4adyTM0JdLsGjtEaTe2qfsOUg3woBbu+Gux23cRZI3p3Lb2QB7sX83gNYEINjtM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=hwV4RPEc; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hwV4RPEc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1708374460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=CZ9nzIp6AU6eCjQhGqjijvyhMpPA9H4GRA2l8LFyClA=; b=hwV4RPEcevHWnFwU6XeyLJIZ5Hz4Kz/IdJUsiwc8SfgLD7Hgkc63sPt7eZ7V4hhkGTSVHv 2yzQHteK4wM9We494ri1wqnukb87Q5fvSnO5jpMc3czJHRCConIa7LFA3hBDiUSUmnT/nD 8szmNpBkqj9c1M09qhBBejmLm38kZco= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-645-w7mGHixTMzeoW-kVYsc4vQ-1; Mon, 19 Feb 2024 15:27:39 -0500 X-MC-Unique: w7mGHixTMzeoW-kVYsc4vQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4D3623C0008C for ; Mon, 19 Feb 2024 20:27:39 +0000 (UTC) Received: from file1-rdu.file-001.prod.rdu2.dc.redhat.com (unknown [10.11.5.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 480A31121312; Mon, 19 Feb 2024 20:27:39 +0000 (UTC) Received: by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix, from userid 12668) id 4086630C1B8F; Mon, 19 Feb 2024 20:27:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix) with ESMTP id 3FB983FB4E; Mon, 19 Feb 2024 21:27:39 +0100 (CET) Date: Mon, 19 Feb 2024 21:27:39 +0100 (CET) From: Mikulas Patocka To: Mike Snitzer cc: dm-devel@lists.linux.dev Subject: [PATCH 1/5] dm-integrity: recheck the integrity tag after a failure In-Reply-To: Message-ID: <19a8fd4a-8ac6-51d-87ed-ed8f10825aac@redhat.com> References: Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com If a userspace process reads (with O_DIRECT) multiple blocks into the same buffer, dm-integrity reports an error [1]. The error is reported in a log and it may cause RAID leg being kicked out of the array. This commit fixes dm-integrity, so that if integrity verification fails, the data is read again into a kernel buffer (where userspace can't modify it) and the integrity tag is rechecked. If the recheck succeeds, the content of the kernel buffer is copied into the user buffer; if the recheck fails, an integrity error is reported. [1] https://people.redhat.com/~mpatocka/testcases/blk-auth-modify/read2.c Signed-off-by: Mikulas Patocka Cc: stable@vger.kernel.org --- drivers/md/dm-integrity.c | 92 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 82 insertions(+), 10 deletions(-) Index: linux-2.6/drivers/md/dm-integrity.c =================================================================== --- linux-2.6.orig/drivers/md/dm-integrity.c 2024-02-18 18:04:00.000000000 +0100 +++ linux-2.6/drivers/md/dm-integrity.c 2024-02-19 17:22:29.000000000 +0100 @@ -278,6 +278,8 @@ struct dm_integrity_c { atomic64_t number_of_mismatches; + mempool_t recheck_pool; + struct notifier_block reboot_notifier; }; @@ -1689,6 +1691,76 @@ failed: get_random_bytes(result, ic->tag_size); } +static void integrity_recheck(struct dm_integrity_io *dio) +{ + struct bio *bio = dm_bio_from_per_bio_data(dio, sizeof(struct dm_integrity_io)); + struct dm_integrity_c *ic = dio->ic; + struct bvec_iter iter; + struct bio_vec bv; + sector_t sector, logical_sector, area, offset; + char checksum_onstack[max_t(size_t, HASH_MAX_DIGESTSIZE, MAX_TAG_SIZE)]; + struct page *page; + void *buffer; + + get_area_and_offset(ic, dio->range.logical_sector, &area, &offset); + dio->metadata_block = get_metadata_sector_and_offset(ic, area, offset, &dio->metadata_offset); + sector = get_data_sector(ic, area, offset); + logical_sector = dio->range.logical_sector; + + page = mempool_alloc(&ic->recheck_pool, GFP_NOIO); + buffer = page_to_virt(page); + + __bio_for_each_segment(bv, bio, iter, dio->bio_details.bi_iter) { + unsigned pos = 0; + + do { + char *mem; + int r; + struct dm_io_request io_req; + struct dm_io_region io_loc; + io_req.bi_opf = REQ_OP_READ; + io_req.mem.type = DM_IO_KMEM; + io_req.mem.ptr.addr = buffer; + io_req.notify.fn = NULL; + io_req.client = ic->io; + io_loc.bdev = ic->dev->bdev; + io_loc.sector = sector; + io_loc.count = ic->sectors_per_block; + + r = dm_io(&io_req, 1, &io_loc, NULL); + if (unlikely(r)) { + dio->bi_status = errno_to_blk_status(r); + goto free_ret; + } + + integrity_sector_checksum(ic, logical_sector, buffer, checksum_onstack); + r = dm_integrity_rw_tag(ic, checksum_onstack, &dio->metadata_block, &dio->metadata_offset, + ic->tag_size, TAG_CMP); + if (r) { + if (r > 0) { + DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx", bio->bi_bdev, logical_sector); + atomic64_inc(&ic->number_of_mismatches); + dm_audit_log_bio(DM_MSG_PREFIX, "integrity-checksum", bio, logical_sector, 0); + r = -EILSEQ; + } + dio->bi_status = errno_to_blk_status(r); + goto free_ret; + } + + mem = bvec_kmap_local(&bv); + memcpy(mem + pos, buffer, ic->sectors_per_block << SECTOR_SHIFT); + kunmap_local(mem); + + pos += ic->sectors_per_block << SECTOR_SHIFT; + sector += ic->sectors_per_block; + logical_sector += ic->sectors_per_block; + } while (pos < bv.bv_len); + } + +free_ret: + mempool_free(page, &ic->recheck_pool); +} + static void integrity_metadata(struct work_struct *w) { struct dm_integrity_io *dio = container_of(w, struct dm_integrity_io, work); @@ -1776,15 +1848,8 @@ again: checksums_ptr - checksums, dio->op == REQ_OP_READ ? TAG_CMP : TAG_WRITE); if (unlikely(r)) { if (r > 0) { - sector_t s; - - s = sector - ((r + ic->tag_size - 1) / ic->tag_size); - DMERR_LIMIT("%pg: Checksum failed at sector 0x%llx", - bio->bi_bdev, s); - r = -EILSEQ; - atomic64_inc(&ic->number_of_mismatches); - dm_audit_log_bio(DM_MSG_PREFIX, "integrity-checksum", - bio, s, 0); + integrity_recheck(dio); + goto skip_io; } if (likely(checksums != checksums_onstack)) kfree(checksums); @@ -4261,6 +4326,12 @@ static int dm_integrity_ctr(struct dm_ta goto bad; } + r = mempool_init_page_pool(&ic->recheck_pool, 1, 0); + if (r) { + ti->error = "Cannot allocate mempool"; + goto bad; + } + ic->metadata_wq = alloc_workqueue("dm-integrity-metadata", WQ_MEM_RECLAIM, METADATA_WORKQUEUE_MAX_ACTIVE); if (!ic->metadata_wq) { @@ -4609,6 +4680,7 @@ static void dm_integrity_dtr(struct dm_t kvfree(ic->bbs); if (ic->bufio) dm_bufio_client_destroy(ic->bufio); + mempool_exit(&ic->recheck_pool); mempool_exit(&ic->journal_io_mempool); if (ic->io) dm_io_client_destroy(ic->io); @@ -4661,7 +4733,7 @@ static void dm_integrity_dtr(struct dm_t static struct target_type integrity_target = { .name = "integrity", - .version = {1, 10, 0}, + .version = {1, 11, 0}, .module = THIS_MODULE, .features = DM_TARGET_SINGLETON | DM_TARGET_INTEGRITY, .ctr = dm_integrity_ctr, From patchwork Mon Feb 19 20:28:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 13563087 X-Patchwork-Delegate: snitzer@redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D76C9524BC for ; Mon, 19 Feb 2024 20:28:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374497; cv=none; b=iZhhn7JUWBd63+UEJg0+HTnIps2U2Flvx0DoFtDbcUPrjseWbNcdNLaQYD0mGKE6RJa8Kv1iptaTSEk6qm4sjNQp/5p8ISX6gHouS4mVrIiZJFh7zPdlmCZl4JJsj9FJY70lxCeQvNmFXNP4LpRnVrLqNQaukPX2y5UnuwM9oXk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374497; c=relaxed/simple; bh=iKOLAMrv4xQ5M41TgNeqm2wpflyVbx3KB7iUu8P5nLk=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=iR6pFEhXxcgBs8VTe0UHHN+eYEiMXYtFPMhoGgqh/Dbf5MBe0aGVLfMXO5Bp1v3BJAEmuMQ9AKGkDulfukS/Lv7yQUKV0z65wnL1UD6pCchraDCFo4/7zMLZHfLxOsxlE6zscg6AQY8FQ0kvFDzBFdv8Pn4XSPjJ8iEMZGu5blE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=F0E5rtl3; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="F0E5rtl3" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1708374492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dI0o3P5gx0q0niKI8uaAwxD49czrphIfu7Ddu29eyh0=; b=F0E5rtl3ImWtk31ZpoyMqLdi+pm3J8ZbT/KePgFzqIsbIImKfHaMazpdf+En4kE71z1U3I mKM/LJKk53Sj6JoC8VCtD2/jHFwiPnsKb68gxktTxa4FGOkv8QhIKFSNMlWdT8vceEx1Yy mDGUck7EpiYdaD095EEo/FpdE5vu+SQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-182-a2UQwJnpNOmK4eMBrps2kQ-1; Mon, 19 Feb 2024 15:28:09 -0500 X-MC-Unique: a2UQwJnpNOmK4eMBrps2kQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2165D85A58F for ; Mon, 19 Feb 2024 20:28:09 +0000 (UTC) Received: from file1-rdu.file-001.prod.rdu2.dc.redhat.com (unknown [10.11.5.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 17C48492BC6; Mon, 19 Feb 2024 20:28:09 +0000 (UTC) Received: by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix, from userid 12668) id 11E7630C1B8F; Mon, 19 Feb 2024 20:28:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix) with ESMTP id 1127F3FB4E; Mon, 19 Feb 2024 21:28:09 +0100 (CET) Date: Mon, 19 Feb 2024 21:28:09 +0100 (CET) From: Mikulas Patocka To: Mike Snitzer cc: dm-devel@lists.linux.dev Subject: [PATCH 2/5] dm-verity: recheck the hash after a failure In-Reply-To: Message-ID: <2eaf23b3-557f-89a5-48a3-4e45edd8cf2@redhat.com> References: Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.9 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com If a userspace process reads (with O_DIRECT) multiple blocks into the same buffer, dm-verity reports an error [1]. This commit fixes dm-verity, so that if hash verification fails, the data is read again into a kernel buffer (where userspace can't modify it) and the hash is rechecked. If the recheck succeeds, the content of the kernel buffer is copied into the user buffer; if the recheck fails, an error is reported. [1] https://people.redhat.com/~mpatocka/testcases/blk-auth-modify/read2.c Signed-off-by: Mikulas Patocka Cc: stable@vger.kernel.org --- drivers/md/dm-verity-target.c | 87 ++++++++++++++++++++++++++++++++++++++---- drivers/md/dm-verity.h | 6 ++ 2 files changed, 86 insertions(+), 7 deletions(-) Index: linux-2.6/drivers/md/dm-verity-target.c =================================================================== --- linux-2.6.orig/drivers/md/dm-verity-target.c 2024-02-19 17:22:37.000000000 +0100 +++ linux-2.6/drivers/md/dm-verity-target.c 2024-02-19 17:24:04.000000000 +0100 @@ -482,6 +482,62 @@ int verity_for_bv_block(struct dm_verity return 0; } +static int verity_recheck_copy(struct dm_verity *v, struct dm_verity_io *io, + u8 *data, size_t len) +{ + memcpy(data, io->recheck_buffer, len); + io->recheck_buffer += len; + + return 0; +} + +static int verity_recheck(struct dm_verity *v, struct dm_verity_io *io, + struct bvec_iter start, sector_t cur_block) +{ + struct page *page; + void *buffer; + int r; + struct dm_io_request io_req; + struct dm_io_region io_loc; + + page = mempool_alloc(&v->recheck_pool, GFP_NOIO); + buffer = page_to_virt(page); + + io_req.bi_opf = REQ_OP_READ; + io_req.mem.type = DM_IO_KMEM; + io_req.mem.ptr.addr = buffer; + io_req.notify.fn = NULL; + io_req.client = v->io; + io_loc.bdev = v->data_dev->bdev; + io_loc.sector = cur_block << (v->data_dev_block_bits - SECTOR_SHIFT); + io_loc.count = 1 << (v->data_dev_block_bits - SECTOR_SHIFT); + r = dm_io(&io_req, 1, &io_loc, NULL); + if (unlikely(r)) + goto free_ret; + + r = verity_hash(v, verity_io_hash_req(v, io), buffer, 1 << v->data_dev_block_bits, + verity_io_real_digest(v, io), true); + + if (unlikely(r)) + goto free_ret; + + if (memcmp(verity_io_real_digest(v, io), verity_io_want_digest(v, io), v->digest_size)) { + r = -EIO; + goto free_ret; + } + + io->recheck_buffer = buffer; + r = verity_for_bv_block(v, io, &start, verity_recheck_copy); + if (unlikely(r)) + goto free_ret; + + r = 0; +free_ret: + mempool_free(page, &v->recheck_pool); + + return r; +} + static int verity_bv_zero(struct dm_verity *v, struct dm_verity_io *io, u8 *data, size_t len) { @@ -508,9 +564,7 @@ static int verity_verify_io(struct dm_ve { bool is_zero; struct dm_verity *v = io->v; -#if defined(CONFIG_DM_VERITY_FEC) struct bvec_iter start; -#endif struct bvec_iter iter_copy; struct bvec_iter *iter; struct crypto_wait wait; @@ -561,10 +615,7 @@ static int verity_verify_io(struct dm_ve if (unlikely(r < 0)) return r; -#if defined(CONFIG_DM_VERITY_FEC) - if (verity_fec_is_enabled(v)) - start = *iter; -#endif + start = *iter; r = verity_for_io_block(v, io, iter, &wait); if (unlikely(r < 0)) return r; @@ -586,6 +637,10 @@ static int verity_verify_io(struct dm_ve * tasklet since it may sleep, so fallback to work-queue. */ return -EAGAIN; + } else if (verity_recheck(v, io, start, cur_block) == 0) { + if (v->validated_blocks) + set_bit(cur_block, v->validated_blocks); + continue; #if defined(CONFIG_DM_VERITY_FEC) } else if (verity_fec_decode(v, io, DM_VERITY_BLOCK_TYPE_DATA, cur_block, NULL, &start) == 0) { @@ -941,6 +996,10 @@ static void verity_dtr(struct dm_target if (v->verify_wq) destroy_workqueue(v->verify_wq); + mempool_exit(&v->recheck_pool); + if (v->io) + dm_io_client_destroy(v->io); + if (v->bufio) dm_bufio_client_destroy(v->bufio); @@ -1379,6 +1438,20 @@ static int verity_ctr(struct dm_target * } v->hash_blocks = hash_position; + r = mempool_init_page_pool(&v->recheck_pool, 1, 0); + if (unlikely(r)) { + ti->error = "Cannot allocate mempool"; + goto bad; + } + + v->io = dm_io_client_create(); + if (IS_ERR(v->io)) { + r = PTR_ERR(v->io); + v->io = NULL; + ti->error = "Cannot allocate dm io"; + goto bad; + } + v->bufio = dm_bufio_client_create(v->hash_dev->bdev, 1 << v->hash_dev_block_bits, 1, sizeof(struct buffer_aux), dm_bufio_alloc_callback, NULL, @@ -1486,7 +1559,7 @@ int dm_verity_get_root_digest(struct dm_ static struct target_type verity_target = { .name = "verity", .features = DM_TARGET_IMMUTABLE, - .version = {1, 9, 0}, + .version = {1, 10, 0}, .module = THIS_MODULE, .ctr = verity_ctr, .dtr = verity_dtr, Index: linux-2.6/drivers/md/dm-verity.h =================================================================== --- linux-2.6.orig/drivers/md/dm-verity.h 2024-02-19 17:22:37.000000000 +0100 +++ linux-2.6/drivers/md/dm-verity.h 2024-02-19 17:22:37.000000000 +0100 @@ -11,6 +11,7 @@ #ifndef DM_VERITY_H #define DM_VERITY_H +#include #include #include #include @@ -68,6 +69,9 @@ struct dm_verity { unsigned long *validated_blocks; /* bitset blocks validated */ char *signature_key_desc; /* signature keyring reference */ + + struct dm_io_client *io; + mempool_t recheck_pool; }; struct dm_verity_io { @@ -84,6 +88,8 @@ struct dm_verity_io { struct work_struct work; + char *recheck_buffer; + /* * Three variably-size fields follow this struct: * From patchwork Mon Feb 19 20:28:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 13563088 X-Patchwork-Delegate: snitzer@redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 239C2524B1 for ; Mon, 19 Feb 2024 20:28:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374532; cv=none; b=tx6uZ71UHoPOBdg4m2h1B+wXXxjNa6QPXtxVtKENbo6bXOH8zPUIVRHTXtRUl/UcYX08Dn6tOoq1gwIfkD4/LdwMsK4056NthBhdgtryb83tkzs7ZAZiw8f0GlI3kPRbPoYcA8yGjo6reIyELvavRZbKNWm22+TDSB0Q2p7vak8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374532; c=relaxed/simple; bh=wWjCnLolROVbQdhkDuavG2U/rqNADSbZT0/USP+jD/0=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=o37ugcwRs8itjRLIArTh40htPul5zr0eahXOGdHlhPoAyJmTeDawlFld/YI5l2MI43wek9JDfBonh4+IV/zvCcslqWefLHlLCUwgXrv0NSUZWAmfYBAsIW4U8pvrVYx+5giEmyU6v3a94DJJMzn0zq6XsJN5Jiq+43On3QaRbiY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=g3jKWAsG; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="g3jKWAsG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1708374530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=d92HVd2Av4o1o1NscjhvcRlI0+WRtb6I/jsVspmHbZA=; b=g3jKWAsGGEH/IawhRLMApVmSdRo4EeYLzVfOvggQYyHfXQr6LEjWcHbC2ijkNVBGrRp+xq YCzqPL0ZsYHutjdF87qCVAGQKNxW+oyFlBB3xvM7IUy/HM08RfuUhzoxrBYRhUmKO5SLrH +z2xW7z99BS1CAu0AN/YwtQlwEQ4vyI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-607-7EMsrZG0PzKDykRaTZLmcA-1; Mon, 19 Feb 2024 15:28:48 -0500 X-MC-Unique: 7EMsrZG0PzKDykRaTZLmcA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1AF8385A589 for ; Mon, 19 Feb 2024 20:28:48 +0000 (UTC) Received: from file1-rdu.file-001.prod.rdu2.dc.redhat.com (unknown [10.11.5.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 156002022AAC; Mon, 19 Feb 2024 20:28:48 +0000 (UTC) Received: by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix, from userid 12668) id 0D6E430C1B8F; Mon, 19 Feb 2024 20:28:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix) with ESMTP id 0C3203FB4E; Mon, 19 Feb 2024 21:28:48 +0100 (CET) Date: Mon, 19 Feb 2024 21:28:48 +0100 (CET) From: Mikulas Patocka To: Mike Snitzer cc: dm-devel@lists.linux.dev Subject: [PATCH 3/5] dm-verity: revert e9307e3deb52 ("dm verity: only copy bvec_iter in verity_verify_io if in_tasklet") In-Reply-To: Message-ID: <657d9cad-c5b7-2cef-bf5a-9fe3df46296@redhat.com> References: Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com The patch e9307e3deb52 was supposed to be an optimization, but it actually increases the size of the function verity_verify_io (by 112 bytes), so let's revert it. Signed-off-by: Mikulas Patocka --- drivers/md/dm-verity-target.c | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-) Index: linux-2.6/drivers/md/dm-verity-target.c =================================================================== --- linux-2.6.orig/drivers/md/dm-verity-target.c 2024-02-06 14:46:59.000000000 +0100 +++ linux-2.6/drivers/md/dm-verity-target.c 2024-02-06 14:50:50.000000000 +0100 @@ -565,22 +565,11 @@ static int verity_verify_io(struct dm_ve bool is_zero; struct dm_verity *v = io->v; struct bvec_iter start; - struct bvec_iter iter_copy; - struct bvec_iter *iter; + struct bvec_iter iter_copy = io->iter; struct crypto_wait wait; struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); unsigned int b; - if (static_branch_unlikely(&use_tasklet_enabled) && io->in_tasklet) { - /* - * Copy the iterator in case we need to restart - * verification in a work-queue. - */ - iter_copy = io->iter; - iter = &iter_copy; - } else - iter = &io->iter; - for (b = 0; b < io->n_blocks; b++) { int r; sector_t cur_block = io->block + b; @@ -588,7 +577,7 @@ static int verity_verify_io(struct dm_ve if (v->validated_blocks && bio->bi_status == BLK_STS_OK && likely(test_bit(cur_block, v->validated_blocks))) { - verity_bv_skip_block(v, io, iter); + verity_bv_skip_block(v, io, &iter_copy); continue; } @@ -603,7 +592,7 @@ static int verity_verify_io(struct dm_ve * If we expect a zero block, don't validate, just * return zeros. */ - r = verity_for_bv_block(v, io, iter, + r = verity_for_bv_block(v, io, &iter_copy, verity_bv_zero); if (unlikely(r < 0)) return r; @@ -615,8 +604,8 @@ static int verity_verify_io(struct dm_ve if (unlikely(r < 0)) return r; - start = *iter; - r = verity_for_io_block(v, io, iter, &wait); + start = iter_copy; + r = verity_for_io_block(v, io, &iter_copy, &wait); if (unlikely(r < 0)) return r; From patchwork Mon Feb 19 20:30:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 13563089 X-Patchwork-Delegate: snitzer@redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CFAE44C94 for ; Mon, 19 Feb 2024 20:30:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374615; cv=none; b=WUpBT9gMix+ONC8ZOWUUhiyMffDTwn3j4hafeG+M8TFifC0EmGines93W7ZbO1ki/nCpUo2dxvPYR2TameERYHC/4V2QUn8aFC9kfBQcO7DrB3/u/GG/akRgTTeY2fTmJdgqxV11wNMvyDIKhvOMZ1oF6agwyLcW011kUMrpeEI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374615; c=relaxed/simple; bh=7rlhbm+zccqHGK4ShnoV4ckZJecpZ34KqYaIdzRNIYE=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=pWV/YmrUVkJjbFPY/EFzgUb2/+QUJT0xiPfl9TT9ZcR1YF/Pwhs6VHCsqR6+E9WGnAnVzQSJm2vj3p7EYGuPaX/DZ9ILI9hK3slh5Lggbcehvw25/ucCcjex9qLA4Y30Ypilk1iObdu0JFM9nNoGp27vo2OV5aLm0TLipAr0Wsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Nx02f3Nc; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Nx02f3Nc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1708374612; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HXx4athjB8A8yEEpa4LwVYBisOfoAJyi/l4owabqxTU=; b=Nx02f3NcevnpcJxn3dtCDyqbXqylJV4l2yUyiNHFAwIk7uLwQZqIVy1oLgzu5nR+4386K5 n2ZflV8RKmMSf/yqda7ecPxCjFFIgUH6mOPtGeEQxIXUmq6dKRLJIgwvfAtJ4rxMsF4KBi yb6RxUFekzO2aVizf/1RcKbJkaDQYqs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-103-wa6uExKpNymaIypsctMysg-1; Mon, 19 Feb 2024 15:30:11 -0500 X-MC-Unique: wa6uExKpNymaIypsctMysg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B3D6C85A589 for ; Mon, 19 Feb 2024 20:30:10 +0000 (UTC) Received: from file1-rdu.file-001.prod.rdu2.dc.redhat.com (unknown [10.11.5.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AEE8F14F0; Mon, 19 Feb 2024 20:30:10 +0000 (UTC) Received: by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix, from userid 12668) id A834C30C1B8F; Mon, 19 Feb 2024 20:30:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix) with ESMTP id A76A43FB4E; Mon, 19 Feb 2024 21:30:10 +0100 (CET) Date: Mon, 19 Feb 2024 21:30:10 +0100 (CET) From: Mikulas Patocka To: Mike Snitzer cc: dm-devel@lists.linux.dev Subject: [PATCH 4/5] dm-crypt: don't modify the data when using authenticated encryption In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com It was said that authenticated encryption could produce invalid tag when the data that is being encrypted is modified [1]. So, fix this problem by copying the data into the clone bio first and then encrypt them inside the clone bio. This may reduce performance, but it is needed to prevent the user from corrupting the device by writing data with O_DIRECT and modifying them at the same time. [1] https://lore.kernel.org/all/20240207004723.GA35324@sol.localdomain/T/ Signed-off-by: Mikulas Patocka Cc: stable@vger.kernel.org --- drivers/md/dm-crypt.c | 6 ++++++ 1 file changed, 6 insertions(+) Index: linux-2.6/drivers/md/dm-crypt.c =================================================================== --- linux-2.6.orig/drivers/md/dm-crypt.c 2024-02-05 10:58:03.000000000 +0100 +++ linux-2.6/drivers/md/dm-crypt.c 2024-02-08 14:50:08.000000000 +0100 @@ -2071,6 +2071,12 @@ static void kcryptd_crypt_write_convert( io->ctx.bio_out = clone; io->ctx.iter_out = clone->bi_iter; + if (crypt_integrity_aead(cc)) { + bio_copy_data(clone, io->base_bio); + io->ctx.bio_in = clone; + io->ctx.iter_in = clone->bi_iter; + } + sector += bio_sectors(clone); crypt_inc_pending(io); From patchwork Mon Feb 19 20:31:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 13563090 X-Patchwork-Delegate: snitzer@redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DA0E3E470 for ; Mon, 19 Feb 2024 20:31:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374677; cv=none; b=M4w7YHn8lnY3+MayKsYw0Jy3wPUa1uz7FqD6s1Dr0wGqXUQuUInJI/Xtr18vB+jrggpl8Ce94O1Thp8Bl/71N5xDLsLeOcJsgxwr4xnZtuauAYSkn8qJ0e27sG0jsgWz2zab91CvJtU66+y5/1kPt0AHZBhWFF6NFIHKg4Y7xv8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708374677; c=relaxed/simple; bh=k9ibS4MjBKAt5EA77xj37GOB1GY+HKxuix8vHE4ecsY=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=sovSwARkKX5OB97zUpcC57X3VO+VdWZPJe2+Uzdr+g499ZI8/zw8Cct5oJ+N42+HtWMFTgKr+sMqMTXOK+X9phL2AkBX7+uH2/nUbwXcz3Mt8I9LtMl7nGAI2ky0yWc4h4MQ6RM047eI9v9LIpxkVraeD4bAOTXLApyrrdw/jgU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=imxBBDkZ; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="imxBBDkZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1708374674; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kAjtG3DynyKh4HULEJu2aSkVl6pZfRAdugHK9xWpBe0=; b=imxBBDkZpdv7uFR5V2BUJN0roQ1iqQS+JcDmQQabIDDx/HvuBFzlOQzJoA+Xv1tJo9eZSf BPOK1LfDXgUzDDQ43KG4cmdd3R+/cDDuNxaAcBn/iRAXWZkxr4lBp0d6QGFxUzxumyYWIt YafYg76bRT2Lhbub4BN7pbnlu1Tpjj8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-639-xqWIv7B-POqvxk9EDklnMg-1; Mon, 19 Feb 2024 15:31:12 -0500 X-MC-Unique: xqWIv7B-POqvxk9EDklnMg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D71341C04182 for ; Mon, 19 Feb 2024 20:31:11 +0000 (UTC) Received: from file1-rdu.file-001.prod.rdu2.dc.redhat.com (unknown [10.11.5.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D1DF0AD76; Mon, 19 Feb 2024 20:31:11 +0000 (UTC) Received: by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix, from userid 12668) id CA88730C1B8F; Mon, 19 Feb 2024 20:31:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix) with ESMTP id C97823FB4E; Mon, 19 Feb 2024 21:31:11 +0100 (CET) Date: Mon, 19 Feb 2024 21:31:11 +0100 (CET) From: Mikulas Patocka To: Mike Snitzer cc: dm-devel@lists.linux.dev Subject: [PATCH 5/5] dm-crypt: recheck the integrity tag after a failure In-Reply-To: Message-ID: <7641d5f-ebe-f7db-35b7-d64e6d65476@redhat.com> References: Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com If a userspace process reads (with O_DIRECT) multiple blocks into the same buffer, dm-crypt reports an authentication error [1]. The error is reported in a log and it may cause RAID leg being kicked out of the array. This commit fixes dm-crypt, so that if integrity verification fails, the data is read again into a kernel buffer (where userspace can't modify it) and the integrity tag is rechecked. If the recheck succeeds, the content of the kernel buffer is copied into the user buffer; if the recheck fails, an integrity error is reported. [1] https://people.redhat.com/~mpatocka/testcases/blk-auth-modify/read2.c Signed-off-by: Mikulas Patocka Cc: stable@vger.kernel.org --- drivers/md/dm-crypt.c | 91 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 74 insertions(+), 17 deletions(-) Index: linux-2.6/drivers/md/dm-crypt.c =================================================================== --- linux-2.6.orig/drivers/md/dm-crypt.c 2024-02-19 17:23:43.000000000 +0100 +++ linux-2.6/drivers/md/dm-crypt.c 2024-02-19 17:23:48.000000000 +0100 @@ -62,6 +62,8 @@ struct convert_context { struct skcipher_request *req; struct aead_request *req_aead; } r; + bool aead_recheck; + bool aead_failed; }; @@ -82,6 +84,8 @@ struct dm_crypt_io { blk_status_t error; sector_t sector; + struct bvec_iter saved_bi_iter; + struct rb_node rb_node; } CRYPTO_MINALIGN_ATTR; @@ -1370,10 +1374,13 @@ static int crypt_convert_block_aead(stru if (r == -EBADMSG) { sector_t s = le64_to_cpu(*sector); - DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu", - ctx->bio_in->bi_bdev, s); - dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead", - ctx->bio_in, s, 0); + ctx->aead_failed = true; + if (ctx->aead_recheck) { + DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu", + ctx->bio_in->bi_bdev, s); + dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead", + ctx->bio_in, s, 0); + } } if (!r && cc->iv_gen_ops && cc->iv_gen_ops->post) @@ -1757,6 +1764,8 @@ static void crypt_io_init(struct dm_cryp io->base_bio = bio; io->sector = sector; io->error = 0; + io->ctx.aead_recheck = false; + io->ctx.aead_failed = false; io->ctx.r.req = NULL; io->integrity_metadata = NULL; io->integrity_metadata_from_pool = false; @@ -1768,6 +1777,8 @@ static void crypt_inc_pending(struct dm_ atomic_inc(&io->io_pending); } +static void kcryptd_queue_read(struct dm_crypt_io *io); + /* * One of the bios was finished. Check for completion of * the whole request and correctly clean up the buffer. @@ -1781,6 +1792,15 @@ static void crypt_dec_pending(struct dm_ if (!atomic_dec_and_test(&io->io_pending)) return; + if (likely(!io->ctx.aead_recheck) && unlikely(io->ctx.aead_failed) && + cc->on_disk_tag_size && bio_data_dir(base_bio) == READ) { + io->ctx.aead_recheck = true; + io->ctx.aead_failed = false; + io->error = 0; + kcryptd_queue_read(io); + return; + } + if (io->ctx.r.req) crypt_free_req(cc, io->ctx.r.req, base_bio); @@ -1816,15 +1836,19 @@ static void crypt_endio(struct bio *clon struct dm_crypt_io *io = clone->bi_private; struct crypt_config *cc = io->cc; unsigned int rw = bio_data_dir(clone); - blk_status_t error; + blk_status_t error = clone->bi_status; + + if (io->ctx.aead_recheck && !error) { + kcryptd_queue_crypt(io); + return; + } /* * free the processed pages */ - if (rw == WRITE) + if (rw == WRITE || io->ctx.aead_recheck) crypt_free_buffer_pages(cc, clone); - error = clone->bi_status; bio_put(clone); if (rw == READ && !error) { @@ -1845,6 +1869,22 @@ static int kcryptd_io_read(struct dm_cry struct crypt_config *cc = io->cc; struct bio *clone; + if (io->ctx.aead_recheck) { + if (!(gfp & __GFP_DIRECT_RECLAIM)) + return 1; + crypt_inc_pending(io); + clone = crypt_alloc_buffer(io, io->base_bio->bi_iter.bi_size); + if (unlikely(!clone)) { + crypt_dec_pending(io); + return 1; + } + clone->bi_iter.bi_sector = cc->start + io->sector; + crypt_convert_init(cc, &io->ctx, clone, clone, io->sector); + io->saved_bi_iter = clone->bi_iter; + dm_submit_bio_remap(io->base_bio, clone); + return 0; + } + /* * We need the original biovec array in order to decrypt the whole bio * data *afterwards* -- thanks to immutable biovecs we don't need to @@ -2113,6 +2153,14 @@ dec: static void kcryptd_crypt_read_done(struct dm_crypt_io *io) { + if (io->ctx.aead_recheck) { + if (!io->error) { + io->ctx.bio_in->bi_iter = io->saved_bi_iter; + bio_copy_data(io->base_bio, io->ctx.bio_in); + } + crypt_free_buffer_pages(io->cc, io->ctx.bio_in); + bio_put(io->ctx.bio_in); + } crypt_dec_pending(io); } @@ -2142,11 +2190,17 @@ static void kcryptd_crypt_read_convert(s crypt_inc_pending(io); - crypt_convert_init(cc, &io->ctx, io->base_bio, io->base_bio, - io->sector); + if (io->ctx.aead_recheck) { + io->ctx.cc_sector = io->sector + cc->iv_offset; + r = crypt_convert(cc, &io->ctx, + test_bit(DM_CRYPT_NO_READ_WORKQUEUE, &cc->flags), true); + } else { + crypt_convert_init(cc, &io->ctx, io->base_bio, io->base_bio, + io->sector); - r = crypt_convert(cc, &io->ctx, - test_bit(DM_CRYPT_NO_READ_WORKQUEUE, &cc->flags), true); + r = crypt_convert(cc, &io->ctx, + test_bit(DM_CRYPT_NO_READ_WORKQUEUE, &cc->flags), true); + } /* * Crypto API backlogged the request, because its queue was full * and we're in softirq context, so continue from a workqueue @@ -2188,10 +2242,13 @@ static void kcryptd_async_done(void *dat if (error == -EBADMSG) { sector_t s = le64_to_cpu(*org_sector_of_dmreq(cc, dmreq)); - DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu", - ctx->bio_in->bi_bdev, s); - dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead", - ctx->bio_in, s, 0); + ctx->aead_failed = true; + if (ctx->aead_recheck) { + DMERR_LIMIT("%pg: INTEGRITY AEAD ERROR, sector %llu", + ctx->bio_in->bi_bdev, s); + dm_audit_log_bio(DM_MSG_PREFIX, "integrity-aead", + ctx->bio_in, s, 0); + } io->error = BLK_STS_PROTECTION; } else if (error < 0) io->error = BLK_STS_IOERR; @@ -3116,7 +3173,7 @@ static int crypt_ctr_optional(struct dm_ sval = strchr(opt_string + strlen("integrity:"), ':') + 1; if (!strcasecmp(sval, "aead")) { set_bit(CRYPT_MODE_INTEGRITY_AEAD, &cc->cipher_flags); - } else if (strcasecmp(sval, "none")) { + } else if (strcasecmp(sval, "none")) { ti->error = "Unknown integrity profile"; return -EINVAL; } @@ -3645,7 +3702,7 @@ static void crypt_io_hints(struct dm_tar static struct target_type crypt_target = { .name = "crypt", - .version = {1, 24, 0}, + .version = {1, 25, 0}, .module = THIS_MODULE, .ctr = crypt_ctr, .dtr = crypt_dtr,