From patchwork Wed May 4 16:23:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12838160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4867FC433F5 for ; Wed, 4 May 2022 16:23:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353402AbiEDQ12 (ORCPT ); Wed, 4 May 2022 12:27:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232646AbiEDQ1Y (ORCPT ); Wed, 4 May 2022 12:27:24 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC20D45AC4; Wed, 4 May 2022 09:23:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=NUkZqAWhGiclxUCt/aQAB87+vXSMe5Vh5EDsntJyPOk=; b=4VEjFxqLQHvU9DpjO0J9wc1EXG M6LY4JQcVEAQaGuBqIrAbCH36KjTyphQpgST0pMt45NtNwJxeWosI/bo1FM0VnXFrnAtM6VjPyM5G 4qt3HszxC+Eic7M5D70DV3Pg0uuhA6i1SZrOpLfdH1LMwKxvRJaW5t1hhg0S68JQFTne/TdnPwGzk kMqmqHuvLW44s+L4uZKMQNYnbbqOPU2NJNI7D1CWINk9WDwl8uuCZ3bVBrENN7lswmbOeN8YfANEy Y6DmYpL9sn3cUBA/7nhQkYKRYQSZd2HD7S3tTJczYEech4VvJq7XQQV0TeHt0hq+Rm2dNzOOlmPhL uwL9mQkg==; Received: from [8.34.116.185] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmHmx-00BeD5-4E; Wed, 04 May 2022 16:23:43 +0000 From: Christoph Hellwig To: Chris Mason , Josef Bacik , David Sterba , "Darrick J. Wong" Cc: linux-btrfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 1/5] iomap: allow the file system to provide a bio_set for direct I/O Date: Wed, 4 May 2022 09:23:38 -0700 Message-Id: <20220504162342.573651-2-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220504162342.573651-1-hch@lst.de> References: <20220504162342.573651-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Allow the file system to provide a specific bio_set for allocating direct I/O bios. This will allow file systems that use the ->submit_io hook to stash away additional information for file system use. To make use of this additional space for information in the completion path, the file system needs to override the ->bi_end_io callback and then call back into iomap, so export iomap_dio_bio_end_io for that. Signed-off-by: Christoph Hellwig Reviewed-by: Darrick J. Wong --- fs/iomap/direct-io.c | 18 ++++++++++++++---- include/linux/iomap.h | 3 +++ 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index b08f5dc31780d..15929690d89e3 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -51,6 +51,15 @@ struct iomap_dio { }; }; +static struct bio *iomap_dio_alloc_bio(const struct iomap_iter *iter, + struct iomap_dio *dio, unsigned short nr_vecs, unsigned int opf) +{ + if (dio->dops && dio->dops->bio_set) + return bio_alloc_bioset(iter->iomap.bdev, nr_vecs, opf, + GFP_KERNEL, dio->dops->bio_set); + return bio_alloc(iter->iomap.bdev, nr_vecs, opf, GFP_KERNEL); +} + static void iomap_dio_submit_bio(const struct iomap_iter *iter, struct iomap_dio *dio, struct bio *bio, loff_t pos) { @@ -144,7 +153,7 @@ static inline void iomap_dio_set_error(struct iomap_dio *dio, int ret) cmpxchg(&dio->error, 0, ret); } -static void iomap_dio_bio_end_io(struct bio *bio) +void iomap_dio_bio_end_io(struct bio *bio) { struct iomap_dio *dio = bio->bi_private; bool should_dirty = (dio->flags & IOMAP_DIO_DIRTY); @@ -176,16 +185,17 @@ static void iomap_dio_bio_end_io(struct bio *bio) bio_put(bio); } } +EXPORT_SYMBOL_GPL(iomap_dio_bio_end_io); static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, loff_t pos, unsigned len) { struct inode *inode = file_inode(dio->iocb->ki_filp); struct page *page = ZERO_PAGE(0); - int flags = REQ_SYNC | REQ_IDLE; struct bio *bio; - bio = bio_alloc(iter->iomap.bdev, 1, REQ_OP_WRITE | flags, GFP_KERNEL); + bio = iomap_dio_alloc_bio(iter, dio, 1, + REQ_OP_WRITE | REQ_SYNC | REQ_IDLE); fscrypt_set_bio_crypt_ctx(bio, inode, pos >> inode->i_blkbits, GFP_KERNEL); bio->bi_iter.bi_sector = iomap_sector(&iter->iomap, pos); @@ -311,7 +321,7 @@ static loff_t iomap_dio_bio_iter(const struct iomap_iter *iter, goto out; } - bio = bio_alloc(iomap->bdev, nr_pages, bio_opf, GFP_KERNEL); + bio = iomap_dio_alloc_bio(iter, dio, nr_pages, bio_opf); fscrypt_set_bio_crypt_ctx(bio, inode, pos >> inode->i_blkbits, GFP_KERNEL); bio->bi_iter.bi_sector = iomap_sector(iomap, pos); diff --git a/include/linux/iomap.h b/include/linux/iomap.h index b76f0dd149fb4..a5483020dad41 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -320,6 +320,8 @@ struct iomap_dio_ops { unsigned flags); void (*submit_io)(const struct iomap_iter *iter, struct bio *bio, loff_t file_offset); + + struct bio_set *bio_set; }; /* @@ -349,6 +351,7 @@ struct iomap_dio *__iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, const struct iomap_ops *ops, const struct iomap_dio_ops *dops, unsigned int dio_flags, size_t done_before); ssize_t iomap_dio_complete(struct iomap_dio *dio); +void iomap_dio_bio_end_io(struct bio *bio); #ifdef CONFIG_SWAP struct file; From patchwork Wed May 4 16:23:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12838159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A32EC433FE for ; Wed, 4 May 2022 16:23:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353391AbiEDQ10 (ORCPT ); Wed, 4 May 2022 12:27:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234286AbiEDQ1Y (ORCPT ); Wed, 4 May 2022 12:27:24 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA05646675; Wed, 4 May 2022 09:23:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=i27+VtCIOe04rf/aY8+YzgwW8t+4wpVcGI5t6v0kSec=; b=lrXyuWIsSlTKa0Dea/OPSIqX+i yLL0F9h9Wb7hXWkJj4YhMh2JOTUmC1IHlKSsm8d0ny66A094kEu5tEev7HwmJPRJrDiwCzKnYFC9p 5y/Gbyxbe7WsSr1ks7PyUKtxKwrV54XXXeJarLkQi6kHIzZBtwrvroj8UheZHMEcGvzFJRrs2t5dC Ug2yk032vVk9ldqnPsMWdixb1/RU7N4UHjKIrp1YhG3EKtzuRKlEYaTx5rmKhK6b17pYRtjLvc4aa ITkCWqQPWvdkdoSXPSJqmN4/PakyI0ze1BmAkgHiXI8tnjvxp+IAIHhy5tzmioWzJWAiOuVfnMRzj D9FgCdZA==; Received: from [8.34.116.185] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmHmx-00BeDF-PG; Wed, 04 May 2022 16:23:43 +0000 From: Christoph Hellwig To: Chris Mason , Josef Bacik , David Sterba , "Darrick J. Wong" Cc: linux-btrfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 2/5] iomap: add per-iomap_iter private data Date: Wed, 4 May 2022 09:23:39 -0700 Message-Id: <20220504162342.573651-3-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220504162342.573651-1-hch@lst.de> References: <20220504162342.573651-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Allow the file system to keep state for all iterations. For now only wire it up for direct I/O as there is an immediate need for it there. Signed-off-by: Christoph Hellwig --- fs/iomap/direct-io.c | 8 ++++++++ include/linux/iomap.h | 1 + 2 files changed, 9 insertions(+) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 15929690d89e3..355abe2eacc6a 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -520,6 +520,14 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, dio->submit.waiter = current; dio->submit.poll_bio = NULL; + /* + * Transfer the private data that was passed by the caller to the + * iomap_iter, and clear it in the iocb, as iocb->private will be + * used for polled bio completion later. + */ + iomi.private = iocb->private; + WRITE_ONCE(iocb->private, NULL); + if (iov_iter_rw(iter) == READ) { if (iomi.pos >= dio->i_size) goto out_free_dio; diff --git a/include/linux/iomap.h b/include/linux/iomap.h index a5483020dad41..109c055865f73 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -188,6 +188,7 @@ struct iomap_iter { unsigned flags; struct iomap iomap; struct iomap srcmap; + void *private; }; int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops); From patchwork Wed May 4 16:23:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12838161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A21F7C43217 for ; Wed, 4 May 2022 16:23:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353397AbiEDQ1a (ORCPT ); Wed, 4 May 2022 12:27:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351186AbiEDQ1Y (ORCPT ); Wed, 4 May 2022 12:27:24 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8988D4667A; Wed, 4 May 2022 09:23:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=fBVrbe6Bwv56qkohYH6YihFUc4A7+M1nHINpHwxi6xU=; b=rJaED6EOzmgD7azz0LYKszTuG7 dA4UKGo0pqAyX3jJWNAyPyTUBV14rTvUnxbxpJOZvFoZ0lASJSqUhlcoYX/WLHdFKQJ2oW+Qs0Sp6 CPaeX0HMxCYhEBJcvBAmPD4/yvQQ4aywd9CpHnzAMPur2I0GQlYcIKfyevpCrqGVA9Mag9td1Qh+c LSwRELykidOkWmc38Ln/CZED2wp6+ejvnIDP2LOm5TRhdIAExLvWirgQZ2OrTMiTEMsYTShQmPRAh PYcblJ2uGXKS0jyftOYNs8IZiG+NsGmXJZWB9O7VZKUjdUTnzeERJ9Ri7p5xWcDlDtQi5u0I3s6AT xDT9ANZg==; Received: from [8.34.116.185] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmHmy-00BeDT-DM; Wed, 04 May 2022 16:23:44 +0000 From: Christoph Hellwig To: Chris Mason , Josef Bacik , David Sterba , "Darrick J. Wong" Cc: linux-btrfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 3/5] btrfs: add a btrfs_dio_rw wrapper Date: Wed, 4 May 2022 09:23:40 -0700 Message-Id: <20220504162342.573651-4-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220504162342.573651-1-hch@lst.de> References: <20220504162342.573651-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Add a wrapper around iomap_dio_rw that keeps the direct I/O internals isolated in inode.c. Signed-off-by: Christoph Hellwig --- fs/btrfs/ctree.h | 5 +++-- fs/btrfs/file.c | 6 ++---- fs/btrfs/inode.c | 11 +++++++++-- 3 files changed, 14 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 6e939bf01dcc3..aa6e71fdc72b9 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -3358,9 +3358,10 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, ssize_t btrfs_do_encoded_write(struct kiocb *iocb, struct iov_iter *from, const struct btrfs_ioctl_encoded_io_args *encoded); +ssize_t btrfs_dio_rw(struct kiocb *iocb, struct iov_iter *iter, + size_t done_before); + extern const struct dentry_operations btrfs_dentry_operations; -extern const struct iomap_ops btrfs_dio_iomap_ops; -extern const struct iomap_dio_ops btrfs_dio_ops; /* Inode locking type flags, by default the exclusive lock is taken */ #define BTRFS_ILOCK_SHARED (1U << 0) diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index b64fb93d90469..46c2baa8fdf54 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -1929,8 +1929,7 @@ static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from) */ again: from->nofault = true; - err = iomap_dio_rw(iocb, from, &btrfs_dio_iomap_ops, &btrfs_dio_ops, - IOMAP_DIO_PARTIAL, written); + err = btrfs_dio_rw(iocb, from, written); from->nofault = false; /* No increment (+=) because iomap returns a cumulative value. */ @@ -3693,8 +3692,7 @@ static ssize_t btrfs_direct_read(struct kiocb *iocb, struct iov_iter *to) */ pagefault_disable(); to->nofault = true; - ret = iomap_dio_rw(iocb, to, &btrfs_dio_iomap_ops, &btrfs_dio_ops, - IOMAP_DIO_PARTIAL, read); + ret = btrfs_dio_rw(iocb, to, read); to->nofault = false; pagefault_enable(); diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index b42d6e7e4049f..cdf96a2472821 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8155,15 +8155,22 @@ static void btrfs_submit_direct(const struct iomap_iter *iter, btrfs_dio_private_put(dip); } -const struct iomap_ops btrfs_dio_iomap_ops = { +static const struct iomap_ops btrfs_dio_iomap_ops = { .iomap_begin = btrfs_dio_iomap_begin, .iomap_end = btrfs_dio_iomap_end, }; -const struct iomap_dio_ops btrfs_dio_ops = { +static const struct iomap_dio_ops btrfs_dio_ops = { .submit_io = btrfs_submit_direct, }; +ssize_t btrfs_dio_rw(struct kiocb *iocb, struct iov_iter *iter, + size_t done_before) +{ + return iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops, + IOMAP_DIO_PARTIAL, done_before); +} + static int btrfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, u64 start, u64 len) { From patchwork Wed May 4 16:23:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12838162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57C18C4332F for ; Wed, 4 May 2022 16:23:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353413AbiEDQ1b (ORCPT ); Wed, 4 May 2022 12:27:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353395AbiEDQ11 (ORCPT ); Wed, 4 May 2022 12:27:27 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 567F34617E; Wed, 4 May 2022 09:23:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=P59pyQBqBoF17d2W6JJgn//aSw8c2p3w7OtvRpfEhsQ=; b=COYWQjJcNP9HMTXTKeN4Fl5rLf qMMUlC7kox0OnEihMCI4DSG9D294Kik6ZGKkSyjE8mgfUS/pftaUY6Ah7kKMO8OpfmJws46iqZG2I x9SYfbomKduQ6a3RZuT/Xtth7w92iymPwNv2gaMiNnR5WiKpqW3Jkl4lNE1qkZwXbpiEf8gXQG/oa WIPzmspHnwWaE1s5YIJRe28d/A1bZUdQJYk2GwzYHfY/FCa6omkmoorcrVIT0DxI53+Ic9X4GcWRf mj3d+VAwoRZW5HycNdTfLlzuoYWkBz4XGM3kum+F4Ug9RhHpTy+oNAL7MFRfti9twe26SmNKgs+XG ZMGb9BMg==; Received: from [8.34.116.185] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmHn0-00BeE4-2A; Wed, 04 May 2022 16:23:46 +0000 From: Christoph Hellwig To: Chris Mason , Josef Bacik , David Sterba , "Darrick J. Wong" Cc: linux-btrfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 4/5] btrfs: allocate dio_data on stack Date: Wed, 4 May 2022 09:23:41 -0700 Message-Id: <20220504162342.573651-5-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220504162342.573651-1-hch@lst.de> References: <20220504162342.573651-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Make use of the new iomap_iter->private field to avoid a memory allocation per iomap range. Signed-off-by: Christoph Hellwig --- fs/btrfs/inode.c | 33 +++++++++++---------------------- 1 file changed, 11 insertions(+), 22 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index cdf96a2472821..6ecff32c1fc22 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -7546,10 +7546,11 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start, loff_t length, unsigned int flags, struct iomap *iomap, struct iomap *srcmap) { + struct iomap_iter *iter = container_of(iomap, struct iomap_iter, iomap); struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); struct extent_map *em; struct extent_state *cached_state = NULL; - struct btrfs_dio_data *dio_data = NULL; + struct btrfs_dio_data *dio_data = iter->private; u64 lockstart, lockend; const bool write = !!(flags & IOMAP_WRITE); int ret = 0; @@ -7595,17 +7596,7 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start, } } - if (flags & IOMAP_NOWAIT) { - dio_data = kzalloc(sizeof(*dio_data), GFP_NOWAIT); - if (!dio_data) - return -EAGAIN; - } else { - dio_data = kzalloc(sizeof(*dio_data), GFP_NOFS); - if (!dio_data) - return -ENOMEM; - } - - iomap->private = dio_data; + memset(dio_data, 0, sizeof(*dio_data)); /* * We always try to allocate data space and must do it before locking @@ -7769,23 +7760,22 @@ static int btrfs_dio_iomap_begin(struct inode *inode, loff_t start, extent_changeset_free(dio_data->data_reserved); } - kfree(dio_data); - return ret; } static int btrfs_dio_iomap_end(struct inode *inode, loff_t pos, loff_t length, ssize_t written, unsigned int flags, struct iomap *iomap) { - int ret = 0; - struct btrfs_dio_data *dio_data = iomap->private; + struct iomap_iter *iter = container_of(iomap, struct iomap_iter, iomap); + struct btrfs_dio_data *dio_data = iter->private; size_t submitted = dio_data->submitted; const bool write = !!(flags & IOMAP_WRITE); + int ret = 0; if (!write && (iomap->type == IOMAP_HOLE)) { /* If reading from a hole, unlock and return */ unlock_extent(&BTRFS_I(inode)->io_tree, pos, pos + length - 1); - goto out; + return 0; } if (submitted < length) { @@ -7802,10 +7792,6 @@ static int btrfs_dio_iomap_end(struct inode *inode, loff_t pos, loff_t length, if (write) extent_changeset_free(dio_data->data_reserved); -out: - kfree(dio_data); - iomap->private = NULL; - return ret; } @@ -8041,7 +8027,7 @@ static void btrfs_submit_direct(const struct iomap_iter *iter, int ret; blk_status_t status; struct btrfs_io_geometry geom; - struct btrfs_dio_data *dio_data = iter->iomap.private; + struct btrfs_dio_data *dio_data = iter->private; struct extent_map *em = NULL; dip = btrfs_create_dio_private(dio_bio, inode, file_offset); @@ -8167,6 +8153,9 @@ static const struct iomap_dio_ops btrfs_dio_ops = { ssize_t btrfs_dio_rw(struct kiocb *iocb, struct iov_iter *iter, size_t done_before) { + struct btrfs_dio_data data; + + iocb->private = &data; return iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops, IOMAP_DIO_PARTIAL, done_before); } From patchwork Wed May 4 16:23:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 12838163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF29FC433FE for ; Wed, 4 May 2022 16:23:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353418AbiEDQ1c (ORCPT ); Wed, 4 May 2022 12:27:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353398AbiEDQ11 (ORCPT ); Wed, 4 May 2022 12:27:27 -0400 Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6DC04667E; Wed, 4 May 2022 09:23:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=CmNIU4tdYylQH7CvbeH+XEIPPG8YbbseW7lFLeYKIJU=; b=qv2PNagW2oYK1Yr+Nl2MUf/vHY IjinyxPbZuja+3qoPZl2KkUGzgr/bT+yrvqYV4bxRnTNwk/hP+XdA//csnZhXDmkGlPY9a+OTHv8z 7iOJtGlfFGgiaCcLxWEv4j88kCMGTgcRe2ONc6Rhq9b4Oy4+2jt8OJZu21Xv0FBqQFeJmx6aDu4qs hxWjFDTfO7ySYObYq8gKeg0zWf5iGlaAFK3tBVYoFmBEEZTG+OyXjcW3b/Tci70E8v7O8fwqnl34x cKCKwEAD9oHrAxVWSMrg/E/4Fv1d/KQO+qmj/I36pv1st4gTKb61/KaOnSYaHjaSlCa5t0FHtp0Ga ApArqhfQ==; Received: from [8.34.116.185] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmHn0-00BeEG-O7; Wed, 04 May 2022 16:23:46 +0000 From: Christoph Hellwig To: Chris Mason , Josef Bacik , David Sterba , "Darrick J. Wong" Cc: linux-btrfs@vger.kernel.org, linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 5/5] btrfs: allocate the btrfs_dio_private as part of the iomap dio bio Date: Wed, 4 May 2022 09:23:42 -0700 Message-Id: <20220504162342.573651-6-hch@lst.de> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220504162342.573651-1-hch@lst.de> References: <20220504162342.573651-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org Create a new bio_set that contains all the per-bio private data needed by btrfs for direct I/O and tell the iomap code to use that instead of separately allocation the btrfs_dio_private structure. Signed-off-by: Christoph Hellwig --- fs/btrfs/btrfs_inode.h | 25 ---------- fs/btrfs/ctree.h | 1 - fs/btrfs/inode.c | 108 ++++++++++++++++++++--------------------- 3 files changed, 53 insertions(+), 81 deletions(-) diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h index 32131a5d321b3..33811e896623f 100644 --- a/fs/btrfs/btrfs_inode.h +++ b/fs/btrfs/btrfs_inode.h @@ -395,31 +395,6 @@ static inline bool btrfs_inode_can_compress(const struct btrfs_inode *inode) return true; } -struct btrfs_dio_private { - struct inode *inode; - - /* - * Since DIO can use anonymous page, we cannot use page_offset() to - * grab the file offset, thus need a dedicated member for file offset. - */ - u64 file_offset; - u64 disk_bytenr; - /* Used for bio::bi_size */ - u32 bytes; - - /* - * References to this structure. There is one reference per in-flight - * bio plus one while we're still setting up. - */ - refcount_t refs; - - /* dio_bio came from fs/direct-io.c */ - struct bio *dio_bio; - - /* Array of checksums */ - u8 csums[]; -}; - /* * btrfs_inode_item stores flags in a u64, btrfs_inode stores them in two * separate u32s. These two functions convert between the two representations. diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index aa6e71fdc72b9..fa64323c453f5 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -3217,7 +3217,6 @@ int btrfs_del_orphan_item(struct btrfs_trans_handle *trans, int btrfs_find_orphan_item(struct btrfs_root *root, u64 offset); /* file-item.c */ -struct btrfs_dio_private; int btrfs_del_csums(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 bytenr, u64 len); blk_status_t btrfs_lookup_bio_sums(struct inode *inode, struct bio *bio, u8 *dst); diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 6ecff32c1fc22..f0c74bd1e6c19 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -68,6 +68,31 @@ struct btrfs_dio_data { bool nocow_done; }; +struct btrfs_dio_private { + struct inode *inode; + + /* + * Since DIO can use anonymous page, we cannot use page_offset() to + * grab the file offset, thus need a dedicated member for file offset. + */ + u64 file_offset; + /* Used for bio::bi_size */ + u32 bytes; + + /* + * References to this structure. There is one reference per in-flight + * bio plus one while we're still setting up. + */ + refcount_t refs; + + /* Array of checksums */ + u8 *csums; + + struct bio bio; +}; + +static struct bio_set btrfs_dio_bioset; + struct btrfs_rename_ctx { /* Output field. Stores the index number of the old directory entry. */ u64 index; @@ -7804,19 +7829,19 @@ static void btrfs_dio_private_put(struct btrfs_dio_private *dip) if (!refcount_dec_and_test(&dip->refs)) return; - if (btrfs_op(dip->dio_bio) == BTRFS_MAP_WRITE) { + if (btrfs_op(&dip->bio) == BTRFS_MAP_WRITE) { __endio_write_update_ordered(BTRFS_I(dip->inode), dip->file_offset, dip->bytes, - !dip->dio_bio->bi_status); + !dip->bio.bi_status); } else { unlock_extent(&BTRFS_I(dip->inode)->io_tree, dip->file_offset, dip->file_offset + dip->bytes - 1); } - bio_endio(dip->dio_bio); - kfree(dip); + kfree(dip->csums); + bio_endio(&dip->bio); } static void submit_dio_repair_bio(struct inode *inode, struct bio *bio, @@ -7918,7 +7943,7 @@ static void btrfs_end_dio_bio(struct bio *bio) err = btrfs_check_read_dio_bio(dip, bbio, !err); if (err) - dip->dio_bio->bi_status = err; + dip->bio.bi_status = err; btrfs_record_physical_zoned(dip->inode, bbio->file_offset, bio); @@ -7973,50 +7998,16 @@ static inline blk_status_t btrfs_submit_dio_bio(struct bio *bio, return ret; } -/* - * If this succeeds, the btrfs_dio_private is responsible for cleaning up locked - * or ordered extents whether or not we submit any bios. - */ -static struct btrfs_dio_private *btrfs_create_dio_private(struct bio *dio_bio, - struct inode *inode, - loff_t file_offset) -{ - const bool write = (btrfs_op(dio_bio) == BTRFS_MAP_WRITE); - const bool csum = !(BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM); - size_t dip_size; - struct btrfs_dio_private *dip; - - dip_size = sizeof(*dip); - if (!write && csum) { - struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); - size_t nblocks; - - nblocks = dio_bio->bi_iter.bi_size >> fs_info->sectorsize_bits; - dip_size += fs_info->csum_size * nblocks; - } - - dip = kzalloc(dip_size, GFP_NOFS); - if (!dip) - return NULL; - - dip->inode = inode; - dip->file_offset = file_offset; - dip->bytes = dio_bio->bi_iter.bi_size; - dip->disk_bytenr = dio_bio->bi_iter.bi_sector << 9; - dip->dio_bio = dio_bio; - refcount_set(&dip->refs, 1); - return dip; -} - static void btrfs_submit_direct(const struct iomap_iter *iter, struct bio *dio_bio, loff_t file_offset) { + struct btrfs_dio_private *dip = + container_of(dio_bio, struct btrfs_dio_private, bio); struct inode *inode = iter->inode; const bool write = (btrfs_op(dio_bio) == BTRFS_MAP_WRITE); struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); const bool raid56 = (btrfs_data_alloc_profile(fs_info) & BTRFS_BLOCK_GROUP_RAID56_MASK); - struct btrfs_dio_private *dip; struct bio *bio; u64 start_sector; int async_submit = 0; @@ -8030,24 +8021,24 @@ static void btrfs_submit_direct(const struct iomap_iter *iter, struct btrfs_dio_data *dio_data = iter->private; struct extent_map *em = NULL; - dip = btrfs_create_dio_private(dio_bio, inode, file_offset); - if (!dip) { - if (!write) { - unlock_extent(&BTRFS_I(inode)->io_tree, file_offset, - file_offset + dio_bio->bi_iter.bi_size - 1); - } - dio_bio->bi_status = BLK_STS_RESOURCE; - bio_endio(dio_bio); - return; - } + dip->inode = inode; + dip->file_offset = file_offset; + dip->bytes = dio_bio->bi_iter.bi_size; + refcount_set(&dip->refs, 1); + dip->csums = NULL; - if (!write) { + if (!write && !(BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM)) { /* * Load the csums up front to reduce csum tree searches and * contention when submitting bios. - * - * If we have csums disabled this will do nothing. */ + status = BLK_STS_RESOURCE; + dip->csums = kzalloc(fs_info->csum_size * + (dio_bio->bi_iter.bi_size >> fs_info->sectorsize_bits), + GFP_NOFS); + if (!dip) + goto out_err; + status = btrfs_lookup_bio_sums(inode, dio_bio, dip->csums); if (status != BLK_STS_OK) goto out_err; @@ -8137,7 +8128,7 @@ static void btrfs_submit_direct(const struct iomap_iter *iter, out_err_em: free_extent_map(em); out_err: - dip->dio_bio->bi_status = status; + dio_bio->bi_status = status; btrfs_dio_private_put(dip); } @@ -8148,6 +8139,7 @@ static const struct iomap_ops btrfs_dio_iomap_ops = { static const struct iomap_dio_ops btrfs_dio_ops = { .submit_io = btrfs_submit_direct, + .bio_set = &btrfs_dio_bioset, }; ssize_t btrfs_dio_rw(struct kiocb *iocb, struct iov_iter *iter, @@ -8970,6 +8962,7 @@ void __cold btrfs_destroy_cachep(void) * destroy cache. */ rcu_barrier(); + bioset_exit(&btrfs_dio_bioset); kmem_cache_destroy(btrfs_inode_cachep); kmem_cache_destroy(btrfs_trans_handle_cachep); kmem_cache_destroy(btrfs_path_cachep); @@ -9010,6 +9003,11 @@ int __init btrfs_init_cachep(void) if (!btrfs_free_space_bitmap_cachep) goto fail; + if (bioset_init(&btrfs_dio_bioset, BIO_POOL_SIZE, + offsetof(struct btrfs_dio_private, bio), + BIOSET_NEED_BVECS)) + goto fail; + return 0; fail: btrfs_destroy_cachep();