From patchwork Thu May 26 01:06:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90F2CC433FE for ; Thu, 26 May 2022 01:06:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241026AbiEZBGj (ORCPT ); Wed, 25 May 2022 21:06:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233854AbiEZBGg (ORCPT ); Wed, 25 May 2022 21:06:36 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64DB89156B for ; Wed, 25 May 2022 18:06:35 -0700 (PDT) Received: from pps.filterd (m0109333.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGtdWU011318 for ; Wed, 25 May 2022 18:06:35 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=chofrzRtFagHUfFlePvR1tikCZVtDw9mOGJhX8Mi6kI=; b=ReXw0BazdvpA+dHc8KPzG0ZWotYCo5CRXRwFlhqT50Z4FQuFD5Jdeeb5X3vqrJrU5ich SMa64y0IMKhiG6vluKdHYbOK/t4HL6VkKYFDXmqxU3Oo02xUmE/HjQLvBo/I6fiH76VL J4YDs8bvj2I5jdEWKqSb0Lp9i1RJgRxyWqE= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g93uj23be-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:06:34 -0700 Received: from twshared10560.18.frc3.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:06:34 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id E3B9345C0BC8; Wed, 25 May 2022 18:06:19 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv4 1/9] block: fix infiniate loop for invalid zone append Date: Wed, 25 May 2022 18:06:05 -0700 Message-ID: <20220526010613.4016118-2-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: oh4n5R_v625QnRsqS2pxIeDq9WanKta1 X-Proofpoint-ORIG-GUID: oh4n5R_v625QnRsqS2pxIeDq9WanKta1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Returning 0 early from __bio_iov_append_get_pages() for the max_append_sectors warning just creates an infinite loop since 0 means success, and the bio will never fill from the unadvancing iov_iter. We could turn the return into an error value, but it will already be turned into an error value later on, so just remove the warning. Clearly no one ever hit it anyway. Fixes: 0512a75b98f84 ("block: Introduce REQ_OP_ZONE_APPEND") Signed-off-by: Keith Busch Reviewed-by: Damien Le Moal Reviewed-by: Christoph Hellwig --- block/bio.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/block/bio.c b/block/bio.c index a3893d80dccc..e249f6414fd5 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1228,9 +1228,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) size_t offset; int ret = 0; - if (WARN_ON_ONCE(!max_append_sectors)) - return 0; - /* * Move page array up in the allocated memory for the bio vecs as far as * possible so that we can start filling biovecs from the beginning From patchwork Thu May 26 01:06:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02E4FC433F5 for ; Thu, 26 May 2022 01:08:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238879AbiEZBIf (ORCPT ); Wed, 25 May 2022 21:08:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232046AbiEZBId (ORCPT ); Wed, 25 May 2022 21:08:33 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BB5C9156B for ; Wed, 25 May 2022 18:08:31 -0700 (PDT) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGthmS031857 for ; Wed, 25 May 2022 18:08:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=IPGZEoZLlJcqR4y6vi40ehjfP/MqoJW85cjDrHKckc4=; b=FrT4tTxGTITL9IHqw2rpyYrDPfPw4oqxYhIl1hSGIx0lJshz9qRcb2MC/UAt4+86Hobi U7oQqzJUbuLDHrPzb7NSMZhr+AGpZR67neq5mHS3HzijCJGkx3aSdi+pCCGvtr2kqIlb HcarAlMDkMPahCCeDQSHtTirsxGjvLp6+s8= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g9bky765d-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:30 -0700 Received: from twshared19572.14.frc2.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:29 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 3EF5045C0BCB; Wed, 25 May 2022 18:06:19 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv4 2/9] block/bio: remove duplicate append pages code Date: Wed, 25 May 2022 18:06:06 -0700 Message-ID: <20220526010613.4016118-3-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: c_s6v7QMD4BHHSjIbDYi2IrFj3JpJO8S X-Proofpoint-ORIG-GUID: c_s6v7QMD4BHHSjIbDYi2IrFj3JpJO8S X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch The getting pages setup for zone append and normal IO are identical. Use common code for each. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- block/bio.c | 102 ++++++++++++++++++++++------------------------------ 1 file changed, 42 insertions(+), 60 deletions(-) diff --git a/block/bio.c b/block/bio.c index e249f6414fd5..55d2a9c4e312 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1158,6 +1158,37 @@ static void bio_put_pages(struct page **pages, size_t size, size_t off) put_page(pages[i]); } +static int bio_iov_add_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) +{ + bool same_page = false; + + if (!__bio_try_merge_page(bio, page, len, offset, &same_page)) { + if (WARN_ON_ONCE(bio_full(bio, len))) + return -EINVAL; + __bio_add_page(bio, page, len, offset); + return 0; + } + + if (same_page) + put_page(page); + return 0; +} + +static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) +{ + struct request_queue *q = bdev_get_queue(bio->bi_bdev); + bool same_page = false; + + if (bio_add_hw_page(q, bio, page, len, offset, + queue_max_zone_append_sectors(q), &same_page) != len) + return -EINVAL; + if (same_page) + put_page(page); + return 0; +} + #define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *)) /** @@ -1176,7 +1207,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; - bool same_page = false; ssize_t size, left; unsigned len, i; size_t offset; @@ -1185,7 +1215,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * Move page array up in the allocated memory for the bio vecs as far as * possible so that we can start filling biovecs from the beginning * without overwriting the temporary page array. - */ + */ BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); @@ -1195,18 +1225,18 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) for (left = size, i = 0; left > 0; left -= len, i++) { struct page *page = pages[i]; + int ret; len = min_t(size_t, PAGE_SIZE - offset, left); + if (bio_op(bio) == REQ_OP_ZONE_APPEND) + ret = bio_iov_add_zone_append_page(bio, page, len, + offset); + else + ret = bio_iov_add_page(bio, page, len, offset); - if (__bio_try_merge_page(bio, page, len, offset, &same_page)) { - if (same_page) - put_page(page); - } else { - if (WARN_ON_ONCE(bio_full(bio, len))) { - bio_put_pages(pages + i, left, offset); - return -EINVAL; - } - __bio_add_page(bio, page, len, offset); + if (ret) { + bio_put_pages(pages + i, left, offset); + return ret; } offset = 0; } @@ -1215,51 +1245,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } -static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) -{ - unsigned short nr_pages = bio->bi_max_vecs - bio->bi_vcnt; - unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; - struct request_queue *q = bdev_get_queue(bio->bi_bdev); - unsigned int max_append_sectors = queue_max_zone_append_sectors(q); - struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; - struct page **pages = (struct page **)bv; - ssize_t size, left; - unsigned len, i; - size_t offset; - int ret = 0; - - /* - * Move page array up in the allocated memory for the bio vecs as far as - * possible so that we can start filling biovecs from the beginning - * without overwriting the temporary page array. - */ - BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); - pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); - - size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); - if (unlikely(size <= 0)) - return size ? size : -EFAULT; - - for (left = size, i = 0; left > 0; left -= len, i++) { - struct page *page = pages[i]; - bool same_page = false; - - len = min_t(size_t, PAGE_SIZE - offset, left); - if (bio_add_hw_page(q, bio, page, len, offset, - max_append_sectors, &same_page) != len) { - bio_put_pages(pages + i, left, offset); - ret = -EINVAL; - break; - } - if (same_page) - put_page(page); - offset = 0; - } - - iov_iter_advance(iter, size - left); - return ret; -} - /** * bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to @@ -1294,10 +1279,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) } do { - if (bio_op(bio) == REQ_OP_ZONE_APPEND) - ret = __bio_iov_append_get_pages(bio, iter); - else - ret = __bio_iov_iter_get_pages(bio, iter); + ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); /* don't account direct I/O as memory stall */ From patchwork Thu May 26 01:06:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87FB3C433F5 for ; Thu, 26 May 2022 01:08:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245509AbiEZBIk (ORCPT ); Wed, 25 May 2022 21:08:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242436AbiEZBIj (ORCPT ); Wed, 25 May 2022 21:08:39 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4882691575 for ; Wed, 25 May 2022 18:08:38 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24Q0QNXc009810 for ; Wed, 25 May 2022 18:08:37 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=dAu9R2sAQLSaRAT82clt0hL+Poo0w/bpDIcs6PokFXc=; b=jwUVoEs8b9vq4ZdOUa8GND0rVjsNKS1EMNsaayhRXZFiEsCgQbgMF3UXs92j3rfRQgK1 mFYkxK6XLuhd+Dcop+OlKqp0ezcQ594VnWypQq9dPtzkVDIIsjBL4YbWXoZn8VM7BGxH y9t5TCwqdTnAvY/Fk29B5YlZjJPOH2GXuWY= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g93tvt478-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:37 -0700 Received: from twshared14577.08.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::d) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:36 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 9C8E545C0BD3; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv4 3/9] block: export dma_alignment attribute Date: Wed, 25 May 2022 18:06:07 -0700 Message-ID: <20220526010613.4016118-4-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: tG05R-nnBxdv834mS0fTolZwouyfETsd X-Proofpoint-ORIG-GUID: tG05R-nnBxdv834mS0fTolZwouyfETsd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch User space may want to know how to align their buffers to avoid bouncing. Export the queue attribute. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- Documentation/ABI/stable/sysfs-block | 9 +++++++++ block/blk-sysfs.c | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index e8797cd09aff..08f6d00df138 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -142,6 +142,15 @@ Description: Default value of this file is '1'(on). +What: /sys/block//queue/dma_alignment +Date: May 2022 +Contact: linux-block@vger.kernel.org +Description: + Reports the alignment that user space addresses must have to be + used for raw block device access with O_DIRECT and other driver + specific passthrough mechanisms. + + What: /sys/block//queue/chunk_sectors Date: September 2016 Contact: Hannes Reinecke diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 88bd41d4cb59..14607565d781 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -274,6 +274,11 @@ static ssize_t queue_virt_boundary_mask_show(struct request_queue *q, char *page return queue_var_show(q->limits.virt_boundary_mask, page); } +static ssize_t queue_dma_alignment_show(struct request_queue *q, char *page) +{ + return queue_var_show(queue_dma_alignment(q), page); +} + #define QUEUE_SYSFS_BIT_FNS(name, flag, neg) \ static ssize_t \ queue_##name##_show(struct request_queue *q, char *page) \ @@ -606,6 +611,7 @@ QUEUE_RO_ENTRY(queue_dax, "dax"); QUEUE_RW_ENTRY(queue_io_timeout, "io_timeout"); QUEUE_RW_ENTRY(queue_wb_lat, "wbt_lat_usec"); QUEUE_RO_ENTRY(queue_virt_boundary_mask, "virt_boundary_mask"); +QUEUE_RO_ENTRY(queue_dma_alignment, "dma_alignment"); #ifdef CONFIG_BLK_DEV_THROTTLING_LOW QUEUE_RW_ENTRY(blk_throtl_sample_time, "throttle_sample_time"); @@ -667,6 +673,7 @@ static struct attribute *queue_attrs[] = { &blk_throtl_sample_time_entry.attr, #endif &queue_virt_boundary_mask_entry.attr, + &queue_dma_alignment_entry.attr, NULL, }; From patchwork Thu May 26 01:06:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C040C433EF for ; Thu, 26 May 2022 01:08:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345478AbiEZBIr (ORCPT ); Wed, 25 May 2022 21:08:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344998AbiEZBIo (ORCPT ); Wed, 25 May 2022 21:08:44 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 799B691575 for ; Wed, 25 May 2022 18:08:40 -0700 (PDT) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGtaeS017219 for ; Wed, 25 May 2022 18:08:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=faOcvTN3625CjqrjeCHz0BkjmPq+BgAMSxF65DUtUoM=; b=ZUpKSw8rCE6MPXiGTPTJPPU8XDO0Ne1F4L6GDaIsmadJ1NjR9nkjtYHClx15hh7DGG9b 8o40iuc3sOxnNDZWEqGStdeUyClvNRNpmc1N3aZopB/TUnRt5jbXhmQAxmf9hQL0BbP7 JqUY/hgL0JlUim8a3yx6RZigPDfvMtJ6/dE= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g93vsa50u-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:40 -0700 Received: from twshared19572.14.frc2.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:39 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id DDB4045C0BD9; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch , Johannes Thumshirn Subject: [PATCHv4 4/9] block: introduce bdev_dma_alignment helper Date: Wed, 25 May 2022 18:06:08 -0700 Message-ID: <20220526010613.4016118-5-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: i2KVInlaklorXO5beZ7JL-hDxwjJ3TAC X-Proofpoint-GUID: i2KVInlaklorXO5beZ7JL-hDxwjJ3TAC X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Preparing for upcoming dma_alignment users that have a block_device, but don't need the request_queue. Signed-off-by: Keith Busch Reviewed-by: Johannes Thumshirn Reviewed-by: Christoph Hellwig --- include/linux/blkdev.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 5bdf2ac9142c..834b981ef01b 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1365,6 +1365,11 @@ static inline int queue_dma_alignment(const struct request_queue *q) return q ? q->dma_alignment : 511; } +static inline unsigned int bdev_dma_alignment(struct block_device *bdev) +{ + return queue_dma_alignment(bdev_get_queue(bdev)); +} + static inline int blk_rq_aligned(struct request_queue *q, unsigned long addr, unsigned int len) { From patchwork Thu May 26 01:06:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E80EC433EF for ; Thu, 26 May 2022 01:06:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229451AbiEZBGk (ORCPT ); Wed, 25 May 2022 21:06:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239244AbiEZBGj (ORCPT ); Wed, 25 May 2022 21:06:39 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71AFC8FD68 for ; Wed, 25 May 2022 18:06:38 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGth3M009996 for ; Wed, 25 May 2022 18:06:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=NDVPtsohLwJbs3chctEtWIn1Z10WtLG6eN0XAr5wzj8=; b=XHN+y8mvAlzSc8wJsZvMts2ARx95UAC/VrBMemjI6ca3QPggRWpK1DbOM4Jcem3/XZOJ nfKX0BZLQGm/5EVYh9cPpVvL2JvqsAxQYEXhrSpPiYI4CJgXt1hAFWapZpHXhtm9ZBV+ ZgfMrsoCnxVyPIAREIfO2M0lNBl4Si5/1l8= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g93upj24m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:06:38 -0700 Received: from twshared35748.07.ash9.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:06:37 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id EDFBD45C0BDB; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv4 5/9] block: add a helper function for dio alignment Date: Wed, 25 May 2022 18:06:09 -0700 Message-ID: <20220526010613.4016118-6-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 2xhAkvd4MMN4BsIPI0pOBy83kLHn-QaT X-Proofpoint-ORIG-GUID: 2xhAkvd4MMN4BsIPI0pOBy83kLHn-QaT X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch This will make it easier to add more complex acceptable alignment criteria in the future. Signed-off-by: Keith Busch --- block/fops.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/block/fops.c b/block/fops.c index b9b83030e0df..bd6c2e13a4e3 100644 --- a/block/fops.c +++ b/block/fops.c @@ -42,6 +42,16 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb) return op; } +static int blkdev_dio_aligned(struct block_device *bdev, loff_t pos, + struct iov_iter *iter) +{ + if ((pos | iov_iter_alignment(iter)) & + (bdev_logical_block_size(bdev) - 1)) + return -EINVAL; + + return 0; +} + #define DIO_INLINE_BIO_VECS 4 static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, @@ -54,9 +64,9 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, struct bio bio; ssize_t ret; - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) - return -EINVAL; + ret = blkdev_dio_aligned(bdev, pos, iter); + if (ret) + return ret; if (nr_pages <= DIO_INLINE_BIO_VECS) vecs = inline_vecs; @@ -171,11 +181,11 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bool is_read = (iov_iter_rw(iter) == READ), is_sync; unsigned int opf = is_read ? REQ_OP_READ : dio_bio_write_op(iocb); loff_t pos = iocb->ki_pos; - int ret = 0; + int ret; - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) - return -EINVAL; + ret = blkdev_dio_aligned(bdev, pos, iter); + if (ret) + return ret; if (iocb->ki_flags & IOCB_ALLOC_CACHE) opf |= REQ_ALLOC_CACHE; @@ -296,11 +306,11 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, struct blkdev_dio *dio; struct bio *bio; loff_t pos = iocb->ki_pos; - int ret = 0; + int ret; - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) - return -EINVAL; + ret = blkdev_dio_aligned(bdev, pos, iter); + if (ret) + return ret; if (iocb->ki_flags & IOCB_ALLOC_CACHE) opf |= REQ_ALLOC_CACHE; From patchwork Thu May 26 01:06:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B341C4332F for ; Thu, 26 May 2022 01:08:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344668AbiEZBIq (ORCPT ); Wed, 25 May 2022 21:08:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345478AbiEZBIo (ORCPT ); Wed, 25 May 2022 21:08:44 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1994492D2E for ; Wed, 25 May 2022 18:08:41 -0700 (PDT) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGtkXQ009118 for ; Wed, 25 May 2022 18:08:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=lU1zOubyJNqvc/wlpjU381aeR9tLvbgn9Af6496EpNg=; b=ZAiX821yr7YfPbCeHhKvSOrF6ORgWvWalJiFZLN+xAgAu+3bAioCC1R4B5W237WNPGfy Fk9VV+4S4RMdin35VM/GGF9mDZxQiLT2wJoeBeaq2GhoYj9x7QkXBTtUTtOJoGjbzrLa spxpRyNN+gwnn6WT3h1GgkrovzSmejD/8tY= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g93vut4cw-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:40 -0700 Received: from twshared26317.07.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:38 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 0E2B845C0BDC; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv4 6/9] block/merge: count bytes instead of sectors Date: Wed, 25 May 2022 18:06:10 -0700 Message-ID: <20220526010613.4016118-7-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: Tm7FeWpX9Ho0I9eLb1Lc-BAQa4s1yjqb X-Proofpoint-ORIG-GUID: Tm7FeWpX9Ho0I9eLb1Lc-BAQa4s1yjqb X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Individual bv_len's may not be a sector size. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- v3->v4: Use sector shift Add comment explaing the ALIGN_DOWN block/blk-merge.c | 41 ++++++++++++++++++++++++----------------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/block/blk-merge.c b/block/blk-merge.c index 7771dacc99cb..9ff0cb9e4840 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -201,11 +201,11 @@ static inline unsigned get_max_segment_size(const struct request_queue *q, * @nsegs: [in,out] Number of segments in the bio being built. Incremented * by the number of segments from @bv that may be appended to that * bio without exceeding @max_segs - * @sectors: [in,out] Number of sectors in the bio being built. Incremented - * by the number of sectors from @bv that may be appended to that - * bio without exceeding @max_sectors + * @bytes: [in,out] Number of bytes in the bio being built. Incremented + * by the number of bytes from @bv that may be appended to that + * bio without exceeding @max_bytes * @max_segs: [in] upper bound for *@nsegs - * @max_sectors: [in] upper bound for *@sectors + * @max_bytes: [in] upper bound for *@bytes * * When splitting a bio, it can happen that a bvec is encountered that is too * big to fit in a single segment and hence that it has to be split in the @@ -216,10 +216,10 @@ static inline unsigned get_max_segment_size(const struct request_queue *q, */ static bool bvec_split_segs(const struct request_queue *q, const struct bio_vec *bv, unsigned *nsegs, - unsigned *sectors, unsigned max_segs, - unsigned max_sectors) + unsigned *bytes, unsigned max_segs, + unsigned max_bytes) { - unsigned max_len = (min(max_sectors, UINT_MAX >> 9) - *sectors) << 9; + unsigned max_len = min(max_bytes, UINT_MAX) - *bytes; unsigned len = min(bv->bv_len, max_len); unsigned total_len = 0; unsigned seg_size = 0; @@ -237,7 +237,7 @@ static bool bvec_split_segs(const struct request_queue *q, break; } - *sectors += total_len >> 9; + *bytes += total_len; /* tell the caller to split the bvec if it is too big to fit */ return len > 0 || bv->bv_len > max_len; @@ -269,8 +269,8 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, { struct bio_vec bv, bvprv, *bvprvp = NULL; struct bvec_iter iter; - unsigned nsegs = 0, sectors = 0; - const unsigned max_sectors = get_max_io_size(q, bio); + unsigned nsegs = 0, bytes = 0; + const unsigned max_bytes = get_max_io_size(q, bio) << 9; const unsigned max_segs = queue_max_segments(q); bio_for_each_bvec(bv, bio, iter) { @@ -282,12 +282,12 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, goto split; if (nsegs < max_segs && - sectors + (bv.bv_len >> 9) <= max_sectors && + bytes + bv.bv_len <= max_bytes && bv.bv_offset + bv.bv_len <= PAGE_SIZE) { nsegs++; - sectors += bv.bv_len >> 9; - } else if (bvec_split_segs(q, &bv, &nsegs, §ors, max_segs, - max_sectors)) { + bytes += bv.bv_len; + } else if (bvec_split_segs(q, &bv, &nsegs, &bytes, max_segs, + max_bytes)) { goto split; } @@ -300,13 +300,20 @@ static struct bio *blk_bio_segment_split(struct request_queue *q, split: *segs = nsegs; + /* + * Individual bvecs may not be logical block aligned, so round down to + * that size to ensure both sides of the split bio are appropriately + * sized. + */ + bytes = ALIGN_DOWN(bytes, queue_logical_block_size(q)); + /* * Bio splitting may cause subtle trouble such as hang when doing sync * iopoll in direct IO routine. Given performance gain of iopoll for * big IO can be trival, disable iopoll when split needed. */ bio_clear_polled(bio); - return bio_split(bio, sectors, GFP_NOIO, bs); + return bio_split(bio, bytes >> SECTOR_SHIFT, GFP_NOIO, bs); } /** @@ -375,7 +382,7 @@ EXPORT_SYMBOL(blk_queue_split); unsigned int blk_recalc_rq_segments(struct request *rq) { unsigned int nr_phys_segs = 0; - unsigned int nr_sectors = 0; + unsigned int bytes = 0; struct req_iterator iter; struct bio_vec bv; @@ -398,7 +405,7 @@ unsigned int blk_recalc_rq_segments(struct request *rq) } rq_for_each_bvec(bv, rq, iter) - bvec_split_segs(rq->q, &bv, &nr_phys_segs, &nr_sectors, + bvec_split_segs(rq->q, &bv, &nr_phys_segs, &bytes, UINT_MAX, UINT_MAX); return nr_phys_segs; } From patchwork Thu May 26 01:06:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCAC2C433FE for ; Thu, 26 May 2022 01:08:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343841AbiEZBIp (ORCPT ); Wed, 25 May 2022 21:08:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344668AbiEZBIo (ORCPT ); Wed, 25 May 2022 21:08:44 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E92C92D05 for ; Wed, 25 May 2022 18:08:39 -0700 (PDT) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 24PGtasN012117 for ; Wed, 25 May 2022 18:08:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=ng5jvbQ8e5kkQ3+8JRxl9hl6c4zipP7NfhMRhPIwcAQ=; b=G0/oah+kRKu/W1WggEFAxp6KmjshoKODoYEYfUvKn7tRqDwhgev6URHqwRJBe579q7ki y0DAKmYrS+WTbbHut6WbhRVixUIS4FrbsehOxsPjGx5oVKBlaMP9K4OyV1wMlR5SkmF0 NZyOIYaQkpEF7CXYy/AOIj18STttQBC7NfU= Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3g9qtub5x7-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:38 -0700 Received: from twshared10276.08.ash9.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:21d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:37 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 1A68045C0BDE; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv4 7/9] block/bounce: count bytes instead of sectors Date: Wed, 25 May 2022 18:06:11 -0700 Message-ID: <20220526010613.4016118-8-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 5B4RBcIp4hl1xTI7XJR10-hrbQwW1qs- X-Proofpoint-ORIG-GUID: 5B4RBcIp4hl1xTI7XJR10-hrbQwW1qs- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Individual bv_len's may not be a sector size. Signed-off-by: Keith Busch Reviewed-by: Damien Le Moal Reviewed-by: Pankaj Raghav --- v3->v4: Use sector shift Add comment explaing the ALIGN_DOWN Use unsigned int type for counting bytes block/bounce.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/block/bounce.c b/block/bounce.c index 8f7b6fe3b4db..f6ae21ec2a70 100644 --- a/block/bounce.c +++ b/block/bounce.c @@ -205,19 +205,25 @@ void __blk_queue_bounce(struct request_queue *q, struct bio **bio_orig) int rw = bio_data_dir(*bio_orig); struct bio_vec *to, from; struct bvec_iter iter; - unsigned i = 0; + unsigned i = 0, bytes = 0; bool bounce = false; - int sectors = 0; + int sectors; bio_for_each_segment(from, *bio_orig, iter) { if (i++ < BIO_MAX_VECS) - sectors += from.bv_len >> 9; + bytes += from.bv_len; if (PageHighMem(from.bv_page)) bounce = true; } if (!bounce) return; + /* + * If the original has more than BIO_MAX_VECS biovecs, the total bytes + * may not be block size aligned. Align down to ensure both sides of + * the split bio are appropriately sized. + */ + sectors = ALIGN_DOWN(bytes, queue_logical_block_size(q)) >> SECTOR_SHIFT; if (sectors < bio_sectors(*bio_orig)) { bio = bio_split(*bio_orig, sectors, GFP_NOIO, &bounce_bio_split); bio_chain(bio, *bio_orig); From patchwork Thu May 26 01:06:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BC9DC433EF for ; Thu, 26 May 2022 01:09:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345556AbiEZBJG (ORCPT ); Wed, 25 May 2022 21:09:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345553AbiEZBIu (ORCPT ); Wed, 25 May 2022 21:08:50 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF68392D2E for ; Wed, 25 May 2022 18:08:48 -0700 (PDT) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGterv008742 for ; Wed, 25 May 2022 18:08:48 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=p1SrztScHuNblAyNzaIrLrE3P0hdGIoUW1/0kG6YeG0=; b=bI4l3PPlztGf+5lncfJTmG775gE70vfmcBpB/sEWiibua7XR+31Mhg/W+4lDpnmlJA3k XP14kJxvoBYPhP9RVMdaLL2S4Nn86NOmN0pY7lAU3cQfIPd488F0uKzP6vHj3qEUAXUm tVNHNOAmxJr7td0Lb7PTzGkOADbJyNDlfwI= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g93vut4d3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:48 -0700 Received: from twshared10276.08.ash9.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:46 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 269D045C0BDF; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv4 8/9] block: relax direct io memory alignment Date: Wed, 25 May 2022 18:06:12 -0700 Message-ID: <20220526010613.4016118-9-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: mzkDhkfj7cpc8CR3kb8OAXcqFPlavjFz X-Proofpoint-ORIG-GUID: mzkDhkfj7cpc8CR3kb8OAXcqFPlavjFz X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Use the address alignment requirements from the hardware for direct io instead of requiring addresses be aligned to the block size. User space can discover the alignment requirements from the dma_alignment queue attribute. User space can specify any hardware compatible DMA offset for each segment, but every segment length is still required to be a multiple of the block size. Signed-off-by: Keith Busch Reviewed-by: Damien Le Moal --- block/bio.c | 12 ++++++++++++ block/fops.c | 14 +++++++++++--- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/block/bio.c b/block/bio.c index 55d2a9c4e312..c492881959d1 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1219,7 +1219,19 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); + /* + * Each segment in the iov is required to be a block size multiple. + * However, we may not be able to get the entire segment if it spans + * more pages than bi_max_vecs allows, so we have to ALIGN_DOWN the + * result to ensure the bio's total size is correct. The remainder of + * the iov data will be picked up in the next bio iteration. + * + * If the result is ever 0, that indicates the iov fails the segment + * size requirement and is an error. + */ size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); + if (size > 0) + size = ALIGN_DOWN(size, bdev_logical_block_size(bio->bi_bdev)); if (unlikely(size <= 0)) return size ? size : -EFAULT; diff --git a/block/fops.c b/block/fops.c index bd6c2e13a4e3..6ecbccc552b9 100644 --- a/block/fops.c +++ b/block/fops.c @@ -45,10 +45,10 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb) static int blkdev_dio_aligned(struct block_device *bdev, loff_t pos, struct iov_iter *iter) { - if ((pos | iov_iter_alignment(iter)) & - (bdev_logical_block_size(bdev) - 1)) + if ((pos | iov_iter_count(iter)) & (bdev_logical_block_size(bdev) - 1)) + return -EINVAL; + if (iov_iter_alignment(iter) & bdev_dma_alignment(bdev)) return -EINVAL; - return 0; } @@ -88,6 +88,10 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, bio.bi_ioprio = iocb->ki_ioprio; ret = bio_iov_iter_get_pages(&bio, iter); + + /* check if iov is not aligned */ + if (unlikely(!ret && iov_iter_count(iter))) + ret = -EINVAL; if (unlikely(ret)) goto out; ret = bio.bi_iter.bi_size; @@ -333,6 +337,10 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, bio_iov_bvec_set(bio, iter); } else { ret = bio_iov_iter_get_pages(bio, iter); + + /* check if iov is not aligned */ + if (unlikely(!ret && iov_iter_count(iter))) + ret = -EINVAL; if (unlikely(ret)) { bio_put(bio); return ret; From patchwork Thu May 26 01:06:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 12861940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57643C433FE for ; Thu, 26 May 2022 01:08:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344932AbiEZBIs (ORCPT ); Wed, 25 May 2022 21:08:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232046AbiEZBIo (ORCPT ); Wed, 25 May 2022 21:08:44 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76F119344D for ; Wed, 25 May 2022 18:08:41 -0700 (PDT) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 24PGtaTI031464 for ; Wed, 25 May 2022 18:08:40 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=rloVwIFmva/Mw8+w//LHl9STezmnJzPW30nB4MxWxeg=; b=LMX7bFMBChiFKnvLc7zxxRioAWJzpqw1184RAgHobRms8gu6TiqAFnL9jgNIx+jUQ1ja 4yqnxP7GkIDRIkWDmsQ/CyASx91wMmSabPLUdP2wNw90w9d4cCzaa5xGbTQ5ahtI1QLq vEC/FltPS68w6mnSSVGEi5IObPvyCxZRZ8g= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3g9bky7667-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 25 May 2022 18:08:40 -0700 Received: from twshared19572.14.frc2.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.28; Wed, 25 May 2022 18:08:39 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 30C3145C0BE0; Wed, 25 May 2022 18:06:20 -0700 (PDT) From: Keith Busch To: , CC: , Kernel Team , , , , , , Keith Busch Subject: [PATCHv4 9/9] fs: add support for dma aligned direct-io Date: Wed, 25 May 2022 18:06:13 -0700 Message-ID: <20220526010613.4016118-10-kbusch@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526010613.4016118-1-kbusch@fb.com> References: <20220526010613.4016118-1-kbusch@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: e10VpbaO4oRVwn1h1rR9EVxbWAcggz3P X-Proofpoint-ORIG-GUID: e10VpbaO4oRVwn1h1rR9EVxbWAcggz3P X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.874,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-25_07,2022-05-25_02,2022-02-23_01 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Keith Busch Use the address alignment requirements from the hardware for direct io instead of requiring addresses be aligned to the block size. Signed-off-by: Keith Busch Reviewed-by: Damien Le Moal --- fs/direct-io.c | 11 +++++++---- fs/iomap/direct-io.c | 3 ++- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/fs/direct-io.c b/fs/direct-io.c index 840752006f60..64cc176be60c 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -1131,7 +1131,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, struct dio_submit sdio = { 0, }; struct buffer_head map_bh = { 0, }; struct blk_plug plug; - unsigned long align = offset | iov_iter_alignment(iter); + unsigned long align = iov_iter_alignment(iter); /* * Avoid references to bdev if not absolutely needed to give @@ -1165,11 +1165,14 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, goto fail_dio; } - if (align & blocksize_mask) { - if (bdev) + if ((offset | align) & blocksize_mask) { + if (bdev) { blkbits = blksize_bits(bdev_logical_block_size(bdev)); + if (align & bdev_dma_alignment(bdev)) + goto fail_dio; + } blocksize_mask = (1 << blkbits) - 1; - if (align & blocksize_mask) + if ((offset | count) & blocksize_mask) goto fail_dio; } diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 80f9b047aa1b..0256d28baa8e 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -244,7 +244,8 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, size_t copied = 0; size_t orig_count; - if ((pos | length | align) & ((1 << blkbits) - 1)) + if ((pos | length) & ((1 << blkbits) - 1) || + align & bdev_dma_alignment(iomap->bdev)) return -EINVAL; if (iomap->type == IOMAP_UNWRITTEN) {