From patchwork Fri Jun 10 19:58:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 515CBC43334 for ; Fri, 10 Jun 2022 20:21:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346385AbiFJUVU (ORCPT ); Fri, 10 Jun 2022 16:21:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344532AbiFJUVT (ORCPT ); Fri, 10 Jun 2022 16:21:19 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1BA2F2FAC1D for ; Fri, 10 Jun 2022 13:21:17 -0700 (PDT) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AHOMP8003663 for ; Fri, 10 Jun 2022 13:21:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=qTth+ppG1ZjwXraXMR+U8AMElWzxev/HoVAAn9Cuo0g=; b=eEWccvzobcg0vZw/OguU4JlSyQCcm2H9YJ4Uh40cFBtk6auUnyjt95EUpW/F0Aza8R1Y 4t3yZUkDcqYSFCWZAP8hIVmFV5iELmjqbvca7BR2/4PYP7jawS11+5TT3diWiNMPiMLw lqjwfW2ahm3nidn08hQ2mKYRq/t1pRcMizE= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmab0h84p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:21:16 -0700 Received: from twshared5131.09.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:21:12 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 467E84E9D68A; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv6 01/11] block: fix infinite loop for invalid zone append Date: Fri, 10 Jun 2022 12:58:20 -0700 Message-ID: <20220610195830.3574005-2-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: uyLLLrvoR3Ab0YT7T2R4Ax5KBG4zYSD_ X-Proofpoint-GUID: uyLLLrvoR3Ab0YT7T2R4Ax5KBG4zYSD_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Returning 0 early from __bio_iov_append_get_pages() for the max_append_sectors warning just creates an infinite loop since 0 means success, and the bio will never fill from the unadvancing iov_iter. We could turn the return into an error value, but it will already be turned into an error value later on, so just remove the warning. Clearly no one ever hit it anyway. Fixes: 0512a75b98f84 ("block: Introduce REQ_OP_ZONE_APPEND") Signed-off-by: Keith Busch Reviewed-by: Damien Le Moal Reviewed-by: Christoph Hellwig Reviewed-by: Johannes Thumshirn --- block/bio.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/block/bio.c b/block/bio.c index f92d0223247b..d481d5e4fe47 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1229,9 +1229,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) size_t offset; int ret = 0; - if (WARN_ON_ONCE(!max_append_sectors)) - return 0; - /* * Move page array up in the allocated memory for the bio vecs as far as * possible so that we can start filling biovecs from the beginning From patchwork Fri Jun 10 19:58:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BC0CC433EF for ; Fri, 10 Jun 2022 20:16:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347418AbiFJUQN (ORCPT ); Fri, 10 Jun 2022 16:16:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350511AbiFJUQL (ORCPT ); Fri, 10 Jun 2022 16:16:11 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 939FC24A6A6 for ; Fri, 10 Jun 2022 13:16:10 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AIB8Qb032034 for ; Fri, 10 Jun 2022 13:16:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=v9HSzt/DU0sJkVqXpTJOEZLAe15Mipuuo7k8GXKR8W4=; b=GSKF+o9km8xqOmibihfinbz5uQz+D6TH2oOI7wJdR6UWLfsX85smqDcJGvJ4jEmyJe5n cWD0jiTwpeUckcyckZQMQqzdtsXpH7Sz3idz1RqD90AtCtYveiY2tJQbWVhL98GWZm3M ux/AUzitiPPjS9lscaLamJb0aZFjzCOmgTA= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmb0s8qvx-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:16:10 -0700 Received: from twshared8508.05.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:16:09 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 89A794E9D691; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv6 02/11] block/bio: remove duplicate append pages code Date: Fri, 10 Jun 2022 12:58:21 -0700 Message-ID: <20220610195830.3574005-3-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: _TRl5_cmBXpjIh8QRwjDO1pES8YUK09W X-Proofpoint-ORIG-GUID: _TRl5_cmBXpjIh8QRwjDO1pES8YUK09W X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch The getting pages setup for zone append and normal IO are identical. Use common code for each. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- block/bio.c | 102 ++++++++++++++++++++++------------------------------ 1 file changed, 42 insertions(+), 60 deletions(-) diff --git a/block/bio.c b/block/bio.c index d481d5e4fe47..5618c6a4b3a3 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1159,6 +1159,37 @@ static void bio_put_pages(struct page **pages, size_t size, size_t off) put_page(pages[i]); } +static int bio_iov_add_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) +{ + bool same_page = false; + + if (!__bio_try_merge_page(bio, page, len, offset, &same_page)) { + if (WARN_ON_ONCE(bio_full(bio, len))) + return -EINVAL; + __bio_add_page(bio, page, len, offset); + return 0; + } + + if (same_page) + put_page(page); + return 0; +} + +static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) +{ + struct request_queue *q = bdev_get_queue(bio->bi_bdev); + bool same_page = false; + + if (bio_add_hw_page(q, bio, page, len, offset, + queue_max_zone_append_sectors(q), &same_page) != len) + return -EINVAL; + if (same_page) + put_page(page); + return 0; +} + #define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *)) /** @@ -1177,7 +1208,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; - bool same_page = false; ssize_t size, left; unsigned len, i; size_t offset; @@ -1186,7 +1216,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * Move page array up in the allocated memory for the bio vecs as far as * possible so that we can start filling biovecs from the beginning * without overwriting the temporary page array. - */ + */ BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); @@ -1196,18 +1226,18 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) for (left = size, i = 0; left > 0; left -= len, i++) { struct page *page = pages[i]; + int ret; len = min_t(size_t, PAGE_SIZE - offset, left); + if (bio_op(bio) == REQ_OP_ZONE_APPEND) + ret = bio_iov_add_zone_append_page(bio, page, len, + offset); + else + ret = bio_iov_add_page(bio, page, len, offset); - if (__bio_try_merge_page(bio, page, len, offset, &same_page)) { - if (same_page) - put_page(page); - } else { - if (WARN_ON_ONCE(bio_full(bio, len))) { - bio_put_pages(pages + i, left, offset); - return -EINVAL; - } - __bio_add_page(bio, page, len, offset); + if (ret) { + bio_put_pages(pages + i, left, offset); + return ret; } offset = 0; } @@ -1216,51 +1246,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } -static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) -{ - unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt; - unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; - struct request_queue *q = bdev_get_queue(bio->bi_bdev); - unsigned int max_append_sectors = queue_max_zone_append_sectors(q); - struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; - struct page **pages = (struct page **)bv; - ssize_t size, left; - unsigned len, i; - size_t offset; - int ret = 0; - - /* - * Move page array up in the allocated memory for the bio vecs as far as - * possible so that we can start filling biovecs from the beginning - * without overwriting the temporary page array. - */ - BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); - pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); - - size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); - if (unlikely(size <= 0)) - return size ? size : -EFAULT; - - for (left = size, i = 0; left > 0; left -= len, i++) { - struct page *page = pages[i]; - bool same_page = false; - - len = min_t(size_t, PAGE_SIZE - offset, left); - if (bio_add_hw_page(q, bio, page, len, offset, - max_append_sectors, &same_page) != len) { - bio_put_pages(pages + i, left, offset); - ret = -EINVAL; - break; - } - if (same_page) - put_page(page); - offset = 0; - } - - iov_iter_advance(iter, size - left); - return ret; -} - /** * bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to @@ -1295,10 +1280,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) } do { - if (bio_op(bio) == REQ_OP_ZONE_APPEND) - ret = __bio_iov_append_get_pages(bio, iter); - else - ret = __bio_iov_iter_get_pages(bio, iter); + ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); /* don't account direct I/O as memory stall */ From patchwork Fri Jun 10 19:58:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BA4CC43334 for ; Fri, 10 Jun 2022 20:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245697AbiFJUEQ (ORCPT ); Fri, 10 Jun 2022 16:04:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344427AbiFJUEP (ORCPT ); Fri, 10 Jun 2022 16:04:15 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E8FA28710 for ; Fri, 10 Jun 2022 13:04:13 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AIDr76028846 for ; Fri, 10 Jun 2022 13:04:12 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=d8bkh7bfZZY5WjJ1g5p81fAjYDMdOKOPAhoGcAryaQs=; b=kH9v/ayu679VGaCZ+C4IvFjZt/07EaKj4vRvCHMau5dISJI6AY4GAnsiCwvwwY3NHgXy Tx8HdLP79eCJ3hsE/UMFk6wQmSrDWm28Txay4ArVEAeznAu/CMNKzJLc+OJDxAbmzXnR TQadrCvQLuqT1F0VBQ+GIShLSuoHGanKD6E= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmb208m54-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:04:12 -0700 Received: from twshared5413.23.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:04:11 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 99F384E9D696; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv6 03/11] block: export dma_alignment attribute Date: Fri, 10 Jun 2022 12:58:22 -0700 Message-ID: <20220610195830.3574005-4-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: bj3HofPq1eYezXSAinWTXD3xmO1yuJqb X-Proofpoint-ORIG-GUID: bj3HofPq1eYezXSAinWTXD3xmO1yuJqb X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch User space may want to know how to align their buffers to avoid bouncing. Export the queue attribute. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- Documentation/ABI/stable/sysfs-block | 9 +++++++++ block/blk-sysfs.c | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index e8797cd09aff..cd14ecb3c9a5 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -260,6 +260,15 @@ Description: for discards, and don't read this file. +What: /sys/block//queue/dma_alignment +Date: May 2022 +Contact: linux-block@vger.kernel.org +Description: + Reports the alignment that user space addresses must have to be + used for raw block device access with O_DIRECT and other driver + specific passthrough mechanisms. + + What: /sys/block//queue/fua Date: May 2018 Contact: linux-block@vger.kernel.org diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 88bd41d4cb59..14607565d781 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -274,6 +274,11 @@ static ssize_t queue_virt_boundary_mask_show(struct request_queue *q, char *page return queue_var_show(q->limits.virt_boundary_mask, page); } +static ssize_t queue_dma_alignment_show(struct request_queue *q, char *page) +{ + return queue_var_show(queue_dma_alignment(q), page); +} + #define QUEUE_SYSFS_BIT_FNS(name, flag, neg) \ static ssize_t \ queue_##name##_show(struct request_queue *q, char *page) \ @@ -606,6 +611,7 @@ QUEUE_RO_ENTRY(queue_dax, "dax"); QUEUE_RW_ENTRY(queue_io_timeout, "io_timeout"); QUEUE_RW_ENTRY(queue_wb_lat, "wbt_lat_usec"); QUEUE_RO_ENTRY(queue_virt_boundary_mask, "virt_boundary_mask"); +QUEUE_RO_ENTRY(queue_dma_alignment, "dma_alignment"); #ifdef CONFIG_BLK_DEV_THROTTLING_LOW QUEUE_RW_ENTRY(blk_throtl_sample_time, "throttle_sample_time"); @@ -667,6 +673,7 @@ static struct attribute *queue_attrs[] = { &blk_throtl_sample_time_entry.attr, #endif &queue_virt_boundary_mask_entry.attr, + &queue_dma_alignment_entry.attr, NULL, }; From patchwork Fri Jun 10 19:58:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E95FC433EF for ; Fri, 10 Jun 2022 20:28:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347665AbiFJU26 (ORCPT ); Fri, 10 Jun 2022 16:28:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350972AbiFJU2j (ORCPT ); Fri, 10 Jun 2022 16:28:39 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12E3B302887 for ; Fri, 10 Jun 2022 13:28:13 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AIDr5e028807 for ; Fri, 10 Jun 2022 13:28:12 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=Vn6X0NSrBCyy75YvHAJe+5TwuMczOHtCmNh2r6npjvQ=; b=gfl9ocP7iOrYLEoTx6jCtY6uLlCSynOE7FrOGM3DDr56hWN/DmmH3pRw8UOeb9ChewvV iXDA0J8osLKsjpn3n1PrDXwaQogujvDBR5IVyeEcM7Q40G59AaO2yu09hqM548BMj+gu 4VHX+SwkzQIzYgaVhYWwqxxVUkQjshn0DLc= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmb208rpu-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:28:12 -0700 Received: from twshared10560.18.frc3.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:28:10 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id AA3AB4E9D69A; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv6 04/11] block: introduce bdev_dma_alignment helper Date: Fri, 10 Jun 2022 12:58:23 -0700 Message-ID: <20220610195830.3574005-5-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: cikizQl5o-RUgvaWPK3XsEdz49L4g4IX X-Proofpoint-ORIG-GUID: cikizQl5o-RUgvaWPK3XsEdz49L4g4IX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Preparing for upcoming dma_alignment users that have a block_device, but don't need the request_queue. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- include/linux/blkdev.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 608d577734c2..ab7e6aa17954 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1366,6 +1366,11 @@ static inline int queue_dma_alignment(const struct request_queue *q) return q ? q->dma_alignment : 511; } +static inline unsigned int bdev_dma_alignment(struct block_device *bdev) +{ + return queue_dma_alignment(bdev_get_queue(bdev)); +} + static inline int blk_rq_aligned(struct request_queue *q, unsigned long addr, unsigned int len) { From patchwork Fri Jun 10 19:58:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFD64CCA47B for ; Fri, 10 Jun 2022 20:22:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244658AbiFJUWP (ORCPT ); Fri, 10 Jun 2022 16:22:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344532AbiFJUWO (ORCPT ); Fri, 10 Jun 2022 16:22:14 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C35A73002C1 for ; Fri, 10 Jun 2022 13:22:11 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AIB4xS031951 for ; Fri, 10 Jun 2022 13:22:11 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=a1ESS9FvEbxK8WmdGc+88RFDsifxDVAiztMf4TMM3sc=; b=c7PBurL3cb+9y3immdgkxzdwXC4jNCxcRquUyKEukaB0z/B/jog+pUCdiB+gwbvyjJ75 kGc9rtPv9pRFBe07yF+sXxiFzh10TMUqnlnOsbnjmnuaxsLP75jgEY+WvrvDdBZH+JPf uzfJVlgyxUONKyr4O2izm6FXL0/JsK0ylxM= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmb0s8s22-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:22:11 -0700 Received: from twshared0725.22.frc3.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:22:10 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id C0A084E9D6A1; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv6 05/11] block: add a helper function for dio alignment Date: Fri, 10 Jun 2022 12:58:24 -0700 Message-ID: <20220610195830.3574005-6-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: ecI7NIbYDI4A3w9psRbjXPJTaIhLHy4S X-Proofpoint-ORIG-GUID: ecI7NIbYDI4A3w9psRbjXPJTaIhLHy4S X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch This will make it easier to add more complex acceptable alignment criteria in the future. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- block/fops.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/block/fops.c b/block/fops.c index d6b3276a6c68..9d32df6fc315 100644 --- a/block/fops.c +++ b/block/fops.c @@ -42,6 +42,13 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb) return op; } +static bool blkdev_dio_unaligned(struct block_device *bdev, loff_t pos, + struct iov_iter *iter) +{ + return ((pos | iov_iter_alignment(iter)) & + (bdev_logical_block_size(bdev) - 1)); +} + #define DIO_INLINE_BIO_VECS 4 static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, @@ -54,8 +61,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, struct bio bio; ssize_t ret; - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) + if (blkdev_dio_unaligned(bdev, pos, iter)) return -EINVAL; if (nr_pages <= DIO_INLINE_BIO_VECS) @@ -173,8 +179,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, loff_t pos = iocb->ki_pos; int ret = 0; - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) + if (blkdev_dio_unaligned(bdev, pos, iter)) return -EINVAL; if (iocb->ki_flags & IOCB_ALLOC_CACHE) @@ -298,8 +303,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, loff_t pos = iocb->ki_pos; int ret = 0; - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) + if (blkdev_dio_unaligned(bdev, pos, iter)) return -EINVAL; if (iocb->ki_flags & IOCB_ALLOC_CACHE) From patchwork Fri Jun 10 19:58:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCDA4C43334 for ; Fri, 10 Jun 2022 20:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347094AbiFJUKP (ORCPT ); Fri, 10 Jun 2022 16:10:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347206AbiFJUKO (ORCPT ); Fri, 10 Jun 2022 16:10:14 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C41023F222 for ; Fri, 10 Jun 2022 13:10:13 -0700 (PDT) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AIAHeN020085 for ; Fri, 10 Jun 2022 13:10:13 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=mlVah37siRWUWBy0LAmeGrNqNHKmkneld6bcywwyZNw=; b=XVtUQEGF87zrS1oOcqaWmHPRxNMJniZ9tdOWqFKowFkODVkvbKdg39TEWsuVYZHuBwaU ycjRfRfinI27x/Pa65F2318wWjpkyU1LEP4EQEWfKSQy06t8b4iki1giYooFNCn7ZxCx c65DfK1eNkeonpkKk6k7b8Q+HUjCaHEY2+M= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmb0hgprc-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:10:13 -0700 Received: from twshared10560.18.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:10:11 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id DF25F4E9D6A4; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv6 06/11] block/merge: count bytes instead of sectors Date: Fri, 10 Jun 2022 12:58:25 -0700 Message-ID: <20220610195830.3574005-7-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: J9VHSdKzGFLL9pNhrv13XbW8lRmEbltD X-Proofpoint-ORIG-GUID: J9VHSdKzGFLL9pNhrv13XbW8lRmEbltD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Individual bv_len's may not be a sector size. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig Reviewed-by: Johannes Thumshirn --- block/blk-merge.c | 41 ++++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/block/blk-merge.c b/block/blk-merge.c index 7771dacc99cb..3874619ba136 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -201,11 +201,11 @@ static inline unsigned get_max_segment_size(const struct request_queue *q, * @nsegs: [in,out] Number of segments in the bio being built. Incremented * by the number of segments from @bv that may be appended to that * bio without exceeding @max_segs - * @sectors: [in,out] Number of sectors in the bio being built. Incremented - * by the number of sectors from @bv that may be appended to that - * bio without exceeding @max_sectors + * @bytes: [in,out] Number of bytes in the bio being built. Incremented + * by the number of bytes from @bv that may be appended to that + * bio without exceeding @max_bytes * @max_segs: [in] upper bound for *@nsegs - * @max_sectors: [in] upper bound for *@sectors + * @max_bytes: [in] upper bound for *@bytes * * When splitting a bio, it can happen that a bvec is encountered that is too * big to fit in a single segment and hence that it has to be split in the @@ -216,10 +216,10 @@ static inline unsigned get_max_segment_size(const struct request_queue *q, */ static bool bvec_split_segs(const struct request_queue *q, const struct bio_vec *bv, unsigned *nsegs, - unsigned *sectors, unsigned max_segs, - unsigned max_sectors) + unsigned *bytes, unsigned max_segs, + unsigned max_bytes) { - unsigned max_len = (min(max_sectors, UINT_MAX >> 9) - *sectors) << 9; + unsigned max_len = min(max_bytes, UINT_MAX) - *bytes; unsigned len = min(bv->bv_len, max_len); unsigned total_len = 0; unsigned seg_size = 0; @@ -237,7 +237,7 @@ static bool bvec_split_segs(const struct request_queue *q, break; } - *sectors += total_len >> 9; + *bytes += total_len; /* tell the caller to split the bvec if it is too big to fit */ return len > 0 || bv->bv_len > max_len; @@ -269,8 +269,8 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, { struct bio_vec bv, bvprv, *bvprvp = NULL; struct bvec_iter iter; - unsigned nsegs = 0, sectors = 0; - const unsigned max_sectors = get_max_io_size(q, bio); + unsigned nsegs = 0, bytes = 0; + const unsigned max_bytes = get_max_io_size(q, bio) << 9; const unsigned max_segs = queue_max_segments(q); bio_for_each_bvec(bv, bio, iter) { @@ -282,12 +282,12 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, goto split; if (nsegs < max_segs && - sectors + (bv.bv_len >> 9) <= max_sectors && + bytes + bv.bv_len <= max_bytes && bv.bv_offset + bv.bv_len <= PAGE_SIZE) { nsegs++; - sectors += bv.bv_len >> 9; - } else if (bvec_split_segs(q, &bv, &nsegs, §ors, max_segs, - max_sectors)) { + bytes += bv.bv_len; + } else if (bvec_split_segs(q, &bv, &nsegs, &bytes, max_segs, + max_bytes)) { goto split; } @@ -300,13 +300,20 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, split: *segs = nsegs; + /* + * Individual bvecs might not be logical block aligned. Round down the + * split size so that each bio is properly block size aligned, even if + * we do not use the full hardware limits. + */ + bytes = ALIGN_DOWN(bytes, queue_logical_block_size(q)); + /* * Bio splitting may cause subtle trouble such as hang when doing sync * iopoll in direct IO routine. Given performance gain of iopoll for * big IO can be trival, disable iopoll when split needed. */ bio_clear_polled(bio); - return bio_split(bio, sectors, GFP_NOIO, bs); + return bio_split(bio, bytes >> SECTOR_SHIFT, GFP_NOIO, bs); } /** @@ -375,7 +382,7 @@ EXPORT_SYMBOL(blk_queue_split); unsigned int blk_recalc_rq_segments(struct request *rq) { unsigned int nr_phys_segs = 0; - unsigned int nr_sectors = 0; + unsigned int bytes = 0; struct req_iterator iter; struct bio_vec bv; @@ -398,7 +405,7 @@ unsigned int blk_recalc_rq_segments(struct request *rq) } rq_for_each_bvec(bv, rq, iter) - bvec_split_segs(rq->q, &bv, &nr_phys_segs, &nr_sectors, + bvec_split_segs(rq->q, &bv, &nr_phys_segs, &bytes, UINT_MAX, UINT_MAX); return nr_phys_segs; } From patchwork Fri Jun 10 19:58:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2BE8C433EF for ; Fri, 10 Jun 2022 20:16:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347052AbiFJUQH (ORCPT ); Fri, 10 Jun 2022 16:16:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350511AbiFJUQG (ORCPT ); Fri, 10 Jun 2022 16:16:06 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A25F0248244 for ; Fri, 10 Jun 2022 13:16:05 -0700 (PDT) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 25AHfkCM023139 for ; Fri, 10 Jun 2022 13:16:05 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=yZwAcMb7nvm5kT10B/5We7Ift1xV1Z3biqKS4IGAik0=; b=R1g4AndrsLF0QN5ADX8A51Nc86xdhtJWrXWSICpv58MHs4FJ4rRl9lDMKKRNDK3u5/g8 RyQJ4p5LYyvkLfT6lda5QSJJbesVsDGaGDDv5rl8qWo+NK5EnJxUDkgAG5YG1gPkayg7 mq7ojV2cfpDedZF7Evugh7ygJd47cGu7S2U= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net (PPS) with ESMTPS id 3gmak510n2-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:16:04 -0700 Received: from twshared18317.08.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:16:04 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 1FEAC4E9D6A8; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch , Alexander Viro Subject: [PATCHv6 08/11] iov: introduce iov_iter_aligned Date: Fri, 10 Jun 2022 12:58:27 -0700 Message-ID: <20220610195830.3574005-9-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: tQZbNZv-ZDJo5ew4L4B2vmbPgenVrWDn X-Proofpoint-ORIG-GUID: tQZbNZv-ZDJo5ew4L4B2vmbPgenVrWDn X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch The existing iov_iter_alignment() function returns the logical OR of address and length. For cases where address and length need to be considered separately, introduce a helper function that a caller can specificy length and address masks that indicate if the iov is unaligned. Cc: Alexander Viro Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- include/linux/uio.h | 2 + lib/iov_iter.c | 92 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 739285fe5a2f..34ba4a731179 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -219,6 +219,8 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i); #endif size_t iov_iter_zero(size_t bytes, struct iov_iter *); +bool iov_iter_is_aligned(const struct iov_iter *i, unsigned addr_mask, + unsigned len_mask); unsigned long iov_iter_alignment(const struct iov_iter *i); unsigned long iov_iter_gap_alignment(const struct iov_iter *i); void iov_iter_init(struct iov_iter *i, unsigned int direction, const struct iovec *iov, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6dd5330f7a99..a39b24496878 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1268,6 +1268,98 @@ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count) } EXPORT_SYMBOL(iov_iter_discard); +static bool iov_iter_aligned_iovec(const struct iov_iter *i, unsigned addr_mask, + unsigned len_mask) +{ + size_t size = i->count; + size_t skip = i->iov_offset; + unsigned k; + + for (k = 0; k < i->nr_segs; k++, skip = 0) { + size_t len = i->iov[k].iov_len - skip; + + if (len > size) + len = size; + if (len & len_mask) + return false; + if ((unsigned long)(i->iov[k].iov_base + skip) & addr_mask) + return false; + + size -= len; + if (!size) + break; + } + return true; +} + +static bool iov_iter_aligned_bvec(const struct iov_iter *i, unsigned addr_mask, + unsigned len_mask) +{ + size_t size = i->count; + unsigned skip = i->iov_offset; + unsigned k; + + for (k = 0; k < i->nr_segs; k++, skip = 0) { + size_t len = i->bvec[k].bv_len - skip; + + if (len > size) + len = size; + if (len & len_mask) + return false; + if ((unsigned long)(i->bvec[k].bv_offset + skip) & addr_mask) + return false; + + size -= len; + if (!size) + break; + } + return true; +} + +/** + * iov_iter_is_aligned() - Check if the addresses and lengths of each segments + * are aligned to the parameters. + * + * @i: &struct iov_iter to restore + * @addr_mask: bit mask to check against the iov element's addresses + * @len_mask: bit mask to check against the iov element's lengths + * + * Return: false if any addresses or lengths intersect with the provided masks + */ +bool iov_iter_is_aligned(const struct iov_iter *i, unsigned addr_mask, + unsigned len_mask) +{ + if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) + return iov_iter_aligned_iovec(i, addr_mask, len_mask); + + if (iov_iter_is_bvec(i)) + return iov_iter_aligned_bvec(i, addr_mask, len_mask); + + if (iov_iter_is_pipe(i)) { + unsigned int p_mask = i->pipe->ring_size - 1; + size_t size = i->count; + + if (size & len_mask) + return false; + if (size && allocated(&i->pipe->bufs[i->head & p_mask])) { + if (i->iov_offset & addr_mask) + return false; + } + + return true; + } + + if (iov_iter_is_xarray(i)) { + if (i->count & len_mask) + return false; + if ((i->xarray_start + i->iov_offset) & addr_mask) + return false; + } + + return true; +} +EXPORT_SYMBOL_GPL(iov_iter_is_aligned); + static unsigned long iov_iter_alignment_iovec(const struct iov_iter *i) { unsigned long res = 0; From patchwork Fri Jun 10 19:58:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FEE0C433EF for ; Fri, 10 Jun 2022 20:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347077AbiFJUKP (ORCPT ); Fri, 10 Jun 2022 16:10:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347094AbiFJUKN (ORCPT ); Fri, 10 Jun 2022 16:10:13 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 654FC666A5 for ; Fri, 10 Jun 2022 13:10:10 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AIB3ol031875 for ; Fri, 10 Jun 2022 13:10:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=dywOri+/cY1xPB0YNJx6g470GDKJiIgQUSlE0U+ycFg=; b=FN15DiGAcxYLaVcAAl0WunUuMNYlPCsVNqnyh1Apm38KQVujBdl9cP5dj7LskcHJE4mc uAs7i+kyiPu5FiGP1mbAvtbCAZQSFu3uSgwDaAeLO8GsKCR3UeaDJ39sdJomscE2UVLv l7VbT41oOvLtzJDzXvxQ1sRh9hyiHfMYRbs= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmb0s8pq3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:10:10 -0700 Received: from twshared14577.08.ash8.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:10:09 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 2C2244E9D6A9; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv6 09/11] block: introduce bdev_iter_is_aligned helper Date: Fri, 10 Jun 2022 12:58:28 -0700 Message-ID: <20220610195830.3574005-10-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: v9pXwPqdYCqWf_xb0MaAP1gfpWsh6G5A X-Proofpoint-ORIG-GUID: v9pXwPqdYCqWf_xb0MaAP1gfpWsh6G5A X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Provide a convenient function for this repeatable coding pattern. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- include/linux/blkdev.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ab7e6aa17954..fb5c177708d5 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1371,6 +1371,13 @@ static inline unsigned int bdev_dma_alignment(struct block_device *bdev) return queue_dma_alignment(bdev_get_queue(bdev)); } +static inline bool bdev_iter_is_aligned(struct block_device *bdev, + struct iov_iter *iter) +{ + return iov_iter_is_aligned(iter, bdev_dma_alignment(bdev), + bdev_logical_block_size(bdev) - 1); +} + static inline int blk_rq_aligned(struct request_queue *q, unsigned long addr, unsigned int len) { From patchwork Fri Jun 10 19:58:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CDE6C433EF for ; Fri, 10 Jun 2022 20:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344427AbiFJUEQ (ORCPT ); Fri, 10 Jun 2022 16:04:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346801AbiFJUEP (ORCPT ); Fri, 10 Jun 2022 16:04:15 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A68D928732 for ; Fri, 10 Jun 2022 13:04:14 -0700 (PDT) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 25AHLH0P003566 for ; Fri, 10 Jun 2022 13:04:14 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=7uKXzg/OPyXABdABOidNCjnrvjXhXTUMSABSngag/3w=; b=SFVIqev8kpSEzX+JXUynTupOi/zwkrLNaKrcB9F30a3AYKQzM/Ds++cj4AS3OrpQg3Hp /ACIj9D+NMmAz4ZyyMJoEHIDgHs/yI4KAs2YsCltn9asi+dm8F2YNbOpUnyzfcbO32eS z7oVTKbTuXcxyhC9kdnfW4fQ0eW9sC4ClkE= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3gma9e94wb-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:04:13 -0700 Received: from twshared14818.18.frc3.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:04:12 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 364FA4E9D6AB; Fri, 10 Jun 2022 12:58:31 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv6 10/11] block: relax direct io memory alignment Date: Fri, 10 Jun 2022 12:58:29 -0700 Message-ID: <20220610195830.3574005-11-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: ZwHld_SnbRQs4iyX7KS0Q3edRNHbKwkZ X-Proofpoint-GUID: ZwHld_SnbRQs4iyX7KS0Q3edRNHbKwkZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size. User space can discover the alignment requirements from the dma_alignment queue attribute. User space can specify any hardware compatible DMA offset for each segment, but every segment length is still required to be a multiple of the block size. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- block/bio.c | 9 +++++++++ block/fops.c | 4 ++-- include/linux/blkdev.h | 5 +++++ 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/block/bio.c b/block/bio.c index 5618c6a4b3a3..551f1d12208b 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1220,7 +1220,16 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); + /* + * Each segment in the iov is required to be a block size multiple. + * However, we may not be able to get the entire segment if it spans + * more pages than bi_max_vecs allows, so we have to ALIGN_DOWN the + * result to ensure the bio's total size is correct. The remainder of + * the iov data will be picked up in the next bio iteration. + */ size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); + if (size > 0) + size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); if (unlikely(size <= 0)) return size ? size : -EFAULT; diff --git a/block/fops.c b/block/fops.c index 9d32df6fc315..86d3cab9bf93 100644 --- a/block/fops.c +++ b/block/fops.c @@ -45,8 +45,8 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb) static bool blkdev_dio_unaligned(struct block_device *bdev, loff_t pos, struct iov_iter *iter) { - return ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)); + return pos & (bdev_logical_block_size(bdev) - 1) || + !bdev_iter_is_aligned(bdev, iter); } #define DIO_INLINE_BIO_VECS 4 diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index fb5c177708d5..914c613d81da 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -425,6 +425,11 @@ struct request_queue { unsigned long nr_requests; /* Max # of requests */ unsigned int dma_pad_mask; + /* + * Drivers that set dma_alignment to less than 511 must be prepared to + * handle individual bvec's that are not a multiple of a SECTOR_SIZE + * due to possible offsets. + */ unsigned int dma_alignment; #ifdef CONFIG_BLK_INLINE_ENCRYPTION From patchwork Fri Jun 10 19:58:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12878058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B77DC433EF for ; Fri, 10 Jun 2022 20:16:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350564AbiFJUQQ (ORCPT ); Fri, 10 Jun 2022 16:16:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350528AbiFJUQP (ORCPT ); Fri, 10 Jun 2022 16:16:15 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DA5024C0B9 for ; Fri, 10 Jun 2022 13:16:14 -0700 (PDT) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25AHidpd024989 for ; Fri, 10 Jun 2022 13:16:14 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=ngiycs0SqGNoBcBOOiS6n513hBNhAYc94F0OBqdAAPA=; b=NqOkwff1ndriV/TqQJAtJRzR/JpvvR4lbX+fxMfx67z+bs2ijggeFi+39k+wAQGAWNL4 0gOQgflYbrkdH1971CCxabjXgApJatfZLuB38WHbBwvotbjy77Yi0m31qTPtFQ5SJelR Rlr1taJtGb4I6ZY3wNiRa8R9DZEri3nmduE= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3gmam8ryq4-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 10 Jun 2022 13:16:14 -0700 Received: from twshared5131.09.ash9.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Fri, 10 Jun 2022 13:16:12 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 408614E9D6AC; Fri, 10 Jun 2022 12:58:32 -0700 (PDT) From: Keith Busch To: , , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv6 11/11] iomap: add support for dma aligned direct-io Date: Fri, 10 Jun 2022 12:58:30 -0700 Message-ID: <20220610195830.3574005-12-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220610195830.3574005-1-kbusch@fb.com> References: <20220610195830.3574005-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: YpZOQ_p8Tfhh3irUaxmMRrgJyP946SqS X-Proofpoint-ORIG-GUID: YpZOQ_p8Tfhh3irUaxmMRrgJyP946SqS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.517,FMLib:17.11.64.514 definitions=2022-06-10_08,2022-06-09_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Use the address alignment requirements from the block_device for direct io instead of requiring addresses be aligned to the block size. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- fs/iomap/direct-io.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 370c3241618a..5d098adba443 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -242,7 +242,6 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, struct inode *inode = iter->inode; unsigned int blkbits = blksize_bits(bdev_logical_block_size(iomap->bdev)); unsigned int fs_block_size = i_blocksize(inode), pad; - unsigned int align = iov_iter_alignment(dio->submit.iter); loff_t length = iomap_length(iter); loff_t pos = iter->pos; unsigned int bio_opf; @@ -253,7 +252,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, size_t copied = 0; size_t orig_count; - if ((pos | length | align) & ((1 << blkbits) - 1)) + if ((pos | length) & ((1 << blkbits) - 1) || + !bdev_iter_is_aligned(iomap->bdev, dio->submit.iter)) return -EINVAL; if (iomap->type == IOMAP_UNWRITTEN) {