From patchwork Sun Sep 8 23:18:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Bobrowski X-Patchwork-Id: 11137111 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F78F14ED for ; Sun, 8 Sep 2019 23:19:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2F472216C8 for ; Sun, 8 Sep 2019 23:19:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=mbobrowski-org.20150623.gappssmtp.com header.i=@mbobrowski-org.20150623.gappssmtp.com header.b="MMhE+g1E" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731247AbfIHXTD (ORCPT ); Sun, 8 Sep 2019 19:19:03 -0400 Received: from mail-pl1-f194.google.com ([209.85.214.194]:34766 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726080AbfIHXTD (ORCPT ); Sun, 8 Sep 2019 19:19:03 -0400 Received: by mail-pl1-f194.google.com with SMTP id d3so5746785plr.1 for ; Sun, 08 Sep 2019 16:18:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mbobrowski-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=6uMksyYoQE2pF7FGUiDsdEdyfVo+lbVfSO3ifjA8fXY=; b=MMhE+g1E1klM3GkMUOXoVuIrg5F35jG9Fp0fFFgF8T1WvGR8mKBuORKcSbOf6JZ9fi 5JZVDzs3CXlPPTVKejo301V6FkbOyxCwekmAT8TvcqFY8+i3VG7SkpIKoaBFii2tsaNf WEbIxgMIN5+jxpBGfhh+qHECzLIzU52RO36ePGtC/lBR9XsuwJj2ED5cHuDa8FkqY9aB AAZ/HK4ruGNY73iWuBQLk7dKBdjpTl1EmhJ8cw3fd6F/yetcOjbdm2VKzD6wtsnxDvzI hV3TVDBBE+3KB2r5hV/nLsmCKt0etczoBW2z1KZ1MZOcLErOS0QEbRzSsaVXCRlwNlbu 2KUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=6uMksyYoQE2pF7FGUiDsdEdyfVo+lbVfSO3ifjA8fXY=; b=gCemEBEZNmcgZxsk50tmY5iMMf01snfAlBGT4UqH552CMNWuk8q/cXKKmRhNdLJpbr KBS/7lKFkPMMYnicsmttnWOTMLt7hxZAxBU4+ZirqQNHH+90qL+IzUKLm6R3075k9x++ mPJym5XncQKnuiLnxt3A+5Pkid9rSonl66NgwfNdUKYnzM451M8MvXzX+XMu+eTTHsxq ef6+hBxH/PhJMBbOErRFba8npytIXI3EbqdGYKn61cOCgWor73NRIVjAkbaNQI6lEj3X ZFwbRQ6tmePhhVb6HkBdePORJAQpIzh+XZckB0iKM6ysX2uGD/SKM4PMhDXDKeui0NES T5yA== X-Gm-Message-State: APjAAAXJDeh+U5oMZS5rAqI0FNvZKXJ9aszCY5UQ0yRexL/gcDKUfW39 Y12rNtjejBFh2s7Djd0aKDmH X-Google-Smtp-Source: APXvYqx8az6HHPPG8qGquQzVzZroFH+w7Yb0fkUIJ0hdikaAfDhkQYunKjRl6y+shJ06O8LY/q9x/w== X-Received: by 2002:a17:902:8e82:: with SMTP id bg2mr21041767plb.323.1567984737618; Sun, 08 Sep 2019 16:18:57 -0700 (PDT) Received: from bobrowski ([110.232.114.101]) by smtp.gmail.com with ESMTPSA id g8sm10832669pgk.1.2019.09.08.16.18.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 16:18:57 -0700 (PDT) Date: Mon, 9 Sep 2019 09:18:51 +1000 From: Matthew Bobrowski To: tytso@mit.edu, jack@suse.cz, adilger.kernel@dilger.ca Cc: linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, david@fromorbit.com, hch@infradead.org, darrick.wong@oracle.com Subject: [PATCH v2 1/6] ext4: introduce direct IO read path using iomap infrastructure Message-ID: <75a6ead09a10e362526a849af482510a0090f82a.1567978633.git.mbobrowski@mbobrowski.org> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch introduces a new direct IO read path that makes use of the iomap infrastructure. The new function ext4_dio_read_iter() is responsible for calling into the iomap infrastructure via iomap_dio_rw(). If the inode in question does not pass preliminary checks in ext4_dio_checks(), then we simply fallback to buffered IO and take that path to fulfil the request. It's imperative that we drop the IOCB_DIRECT flag from iocb->ki_flags in order to prevent generic_file_read_iter() from trying to take the direct IO code path again. Signed-off-by: Matthew Bobrowski Reviewed-by: Ritesh Harjani --- fs/ext4/file.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 70b0438dbc94..e52e3928dc25 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -34,6 +34,53 @@ #include "xattr.h" #include "acl.h" +static bool ext4_dio_checks(struct inode *inode) +{ +#if IS_ENABLED(CONFIG_FS_ENCRYPTION) + if (IS_ENCRYPTED(inode)) + return false; +#endif + if (ext4_should_journal_data(inode)) + return false; + if (ext4_has_inline_data(inode)) + return false; + return true; +} + +static ssize_t ext4_dio_read_iter(struct kiocb *iocb, struct iov_iter *to) +{ + ssize_t ret; + struct inode *inode = file_inode(iocb->ki_filp); + + /* + * Get exclusion from truncate and other inode operations. + */ + if (!inode_trylock_shared(inode)) { + if (iocb->ki_flags & IOCB_NOWAIT) + return -EAGAIN; + inode_lock_shared(inode); + } + + if (!ext4_dio_checks(inode)) { + inode_unlock_shared(inode); + /* + * Fallback to buffered IO if the operation being + * performed on the inode is not supported by direct + * IO. The IOCB_DIRECT flag flags needs to be cleared + * here to ensure that the direct IO code path within + * generic_file_read_iter() is not taken again. + */ + iocb->ki_flags &= ~IOCB_DIRECT; + return generic_file_read_iter(iocb, to); + } + + ret = iomap_dio_rw(iocb, to, &ext4_iomap_ops, NULL); + inode_unlock_shared(inode); + + file_accessed(iocb->ki_filp); + return ret; +} + #ifdef CONFIG_FS_DAX static ssize_t ext4_dax_read_iter(struct kiocb *iocb, struct iov_iter *to) { @@ -64,16 +111,19 @@ static ssize_t ext4_dax_read_iter(struct kiocb *iocb, struct iov_iter *to) static ssize_t ext4_file_read_iter(struct kiocb *iocb, struct iov_iter *to) { - if (unlikely(ext4_forced_shutdown(EXT4_SB(file_inode(iocb->ki_filp)->i_sb)))) + struct inode *inode = file_inode(iocb->ki_filp); + + if (unlikely(ext4_forced_shutdown(EXT4_SB(inode->i_sb)))) return -EIO; if (!iov_iter_count(to)) return 0; /* skip atime */ -#ifdef CONFIG_FS_DAX - if (IS_DAX(file_inode(iocb->ki_filp))) + if (IS_DAX(inode)) return ext4_dax_read_iter(iocb, to); -#endif + + if (iocb->ki_flags & IOCB_DIRECT) + return ext4_dio_read_iter(iocb, to); return generic_file_read_iter(iocb, to); } From patchwork Sun Sep 8 23:19:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Bobrowski X-Patchwork-Id: 11137113 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7702B186E for ; Sun, 8 Sep 2019 23:19:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5672E20828 for ; Sun, 8 Sep 2019 23:19:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=mbobrowski-org.20150623.gappssmtp.com header.i=@mbobrowski-org.20150623.gappssmtp.com header.b="Hc5ap/DF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731257AbfIHXTX (ORCPT ); Sun, 8 Sep 2019 19:19:23 -0400 Received: from mail-pf1-f194.google.com ([209.85.210.194]:40383 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731201AbfIHXTX (ORCPT ); Sun, 8 Sep 2019 19:19:23 -0400 Received: by mail-pf1-f194.google.com with SMTP id x127so7979946pfb.7 for ; Sun, 08 Sep 2019 16:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mbobrowski-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=NSL01mkUl8fEtBlCwOw2cj9yasSdvgdfrSqSeiKCTmQ=; b=Hc5ap/DFW2n2quEwo6i8ElkD/wvYVSJ8INTCfC7QW5Bf44rLm2V5h+dYg5hL4mGxGb Mi8nkEoY63QYWPZF6RXIKvcH+VzMRCj9yw7lEwFyecgDn783uehOx2sJpsbRsxAuHrLZ hq2h1gbcvQyrEPLXiBplmLGJFQv3Bw6OxmRaWdym4nXkz0E7DJO0EfWibAEJYKkyhHO+ lR8Uysl1P0WxbiC5jRxHCZNNAqsdjlcPPsj2Zu/PIq/M/X2P4PlIyh/eVIK/KJfnrVXS XYuDfH9f4GZHfYydFN2oUORE+E+FHYqFqCK7LReN/XQ6UcU4zavtevFQmVkgUNs6ajuy nG4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=NSL01mkUl8fEtBlCwOw2cj9yasSdvgdfrSqSeiKCTmQ=; b=JsC4R7TBBRc5ifziXz3kXzU/vaP5TVcfM98MktRzr/HuM7IRsQF4x8FpC+jyClyUe9 PXgHxNkza13EXFbBoU8LoLadTsSFriZefAv9oW6yOb/NGRjnZiMk4epQNDalR/veTU1R AGz0wGtxyNv94wH1cauf2egMfCrqIsHWTJ/8g4iDFWBVYB9vuAh4J02v2fUeYekMm/+9 JK/rAqnaB8ka0ivph7fD+wUklx8o9R7GB9Jq63zi688CZLLFv2ebJxZ7+saSuyAVOO9v pLTTyJwrPB2bWDtnU3wkwqE8dK/O53bnFb1HHCz53UMtiKOgfow+WofzsrBn3bbd6kmd EoxA== X-Gm-Message-State: APjAAAXW7j4zW0sDSSL0l9wF5aOkR1vrA/yuG/+AGlSnipSXDHUBzJ8H PqYAwNiWaQ1WSzt6j13iA2gdTtIFwlNc X-Google-Smtp-Source: APXvYqx+Wqr5mMuBQ66NOiGFoN5UlNdtQAbjv38+YJ/rfLZIyIz7GlmCQW9jjzaJL0D+zx0sFf3WpA== X-Received: by 2002:a63:f907:: with SMTP id h7mr18832410pgi.418.1567984762022; Sun, 08 Sep 2019 16:19:22 -0700 (PDT) Received: from bobrowski ([110.232.114.101]) by smtp.gmail.com with ESMTPSA id u7sm10507007pgr.94.2019.09.08.16.19.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 16:19:21 -0700 (PDT) Date: Mon, 9 Sep 2019 09:19:15 +1000 From: Matthew Bobrowski To: tytso@mit.edu, jack@suse.cz, adilger.kernel@dilger.ca Cc: linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, david@fromorbit.com, hch@infradead.org, darrick.wong@oracle.com Subject: [PATCH v2 2/6] ext4: move inode extension/truncate code out from ext4_iomap_end() Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org In preparation for implementing the iomap direct IO write path modifications, the inode extension/truncate code needs to be moved out from ext4_iomap_end(). For direct IO, if the current code remained within ext4_iomap_end() it would behave incorrectly. If we update the inode size prior to converting unwritten extents we run the risk of allowing a racing direct IO read operation to find unwritten extents before they are converted. The inode extension/truncate code has been moved out into a new helper ext4_handle_inode_extension(). This helper has been designed so that it can be used by both DAX and direct IO paths in the instance that the result of the write is extending the inode. Signed-off-by: Matthew Bobrowski --- fs/ext4/file.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++- fs/ext4/inode.c | 48 +------------------------ 2 files changed, 93 insertions(+), 48 deletions(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index e52e3928dc25..8e586198f6e6 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -33,6 +33,7 @@ #include "ext4_jbd2.h" #include "xattr.h" #include "acl.h" +#include "truncate.h" static bool ext4_dio_checks(struct inode *inode) { @@ -233,12 +234,91 @@ static ssize_t ext4_write_checks(struct kiocb *iocb, struct iov_iter *from) return iov_iter_count(from); } +static int ext4_handle_inode_extension(struct inode *inode, loff_t offset, + ssize_t len, size_t count) +{ + handle_t *handle; + bool truncate = false; + ext4_lblk_t written_blk, end_blk; + int ret = 0, blkbits = inode->i_blkbits; + + handle = ext4_journal_start(inode, EXT4_HT_INODE, 2); + if (IS_ERR(handle)) { + ret = PTR_ERR(handle); + goto orphan_del; + } + + if (ext4_update_inode_size(inode, offset + len)) + ext4_mark_inode_dirty(handle, inode); + + /* + * We may need truncate allocated but not written blocks + * beyond EOF. + */ + written_blk = ALIGN(offset + len, 1 << blkbits); + end_blk = ALIGN(offset + len + count, 1 << blkbits); + if (written_blk < end_blk && ext4_can_truncate(inode)) + truncate = true; + + /* + * Remove the inode from the orphan list if it has been + * extended and everything went OK. + */ + if (!truncate && inode->i_nlink) + ext4_orphan_del(handle, inode); + ext4_journal_stop(handle); + + if (truncate) { + ext4_truncate_failed_write(inode); +orphan_del: + /* + * If the truncate operation failed early the inode + * may still be on the orphan list. In that case, we + * need try remove the inode from the linked list in + * memory. + */ + if (inode->i_nlink) + ext4_orphan_del(NULL, inode); + } + return ret; +} + +/* + * The inode may have been placed onto the orphan list or has had + * blocks allocated beyond EOF as a result of an extension. We need to + * ensure that any necessary cleanup routines are performed if the + * error path has been taken for a write. + */ +static int ext4_handle_failed_inode_extension(struct inode *inode, loff_t size) +{ + int ret = 0; + handle_t *handle; + + if (size > i_size_read(inode)) + ext4_truncate_failed_write(inode); + + if (!list_empty(&EXT4_I(inode)->i_orphan)) { + handle = ext4_journal_start(inode, EXT4_HT_INODE, 2); + if (IS_ERR(handle)) { + if (inode->i_nlink) + ext4_orphan_del(NULL, inode); + return PTR_ERR(handle); + } + if (inode->i_nlink) + ext4_orphan_del(handle, inode); + ext4_journal_stop(handle); + } + return ret; +} + #ifdef CONFIG_FS_DAX static ssize_t ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from) { - struct inode *inode = file_inode(iocb->ki_filp); ssize_t ret; + int error = 0; + loff_t offset; + struct inode *inode = file_inode(iocb->ki_filp); if (!inode_trylock(inode)) { if (iocb->ki_flags & IOCB_NOWAIT) @@ -255,7 +335,18 @@ ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from) if (ret) goto out; + offset = iocb->ki_pos; ret = dax_iomap_rw(iocb, from, &ext4_iomap_ops); + if (ret > 0 && iocb->ki_pos > i_size_read(inode)) + error = ext4_handle_inode_extension(inode, offset, ret, + iov_iter_count(from)); + + if (ret < 0) + error = ext4_handle_failed_inode_extension(inode, + iocb->ki_pos); + + if (error) + ret = error; out: inode_unlock(inode); if (ret > 0) diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 420fe3deed39..761ce6286b05 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -3601,53 +3601,7 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length, static int ext4_iomap_end(struct inode *inode, loff_t offset, loff_t length, ssize_t written, unsigned flags, struct iomap *iomap) { - int ret = 0; - handle_t *handle; - int blkbits = inode->i_blkbits; - bool truncate = false; - - if (!(flags & IOMAP_WRITE) || (flags & IOMAP_FAULT)) - return 0; - - handle = ext4_journal_start(inode, EXT4_HT_INODE, 2); - if (IS_ERR(handle)) { - ret = PTR_ERR(handle); - goto orphan_del; - } - if (ext4_update_inode_size(inode, offset + written)) - ext4_mark_inode_dirty(handle, inode); - /* - * We may need to truncate allocated but not written blocks beyond EOF. - */ - if (iomap->offset + iomap->length > - ALIGN(inode->i_size, 1 << blkbits)) { - ext4_lblk_t written_blk, end_blk; - - written_blk = (offset + written) >> blkbits; - end_blk = (offset + length) >> blkbits; - if (written_blk < end_blk && ext4_can_truncate(inode)) - truncate = true; - } - /* - * Remove inode from orphan list if we were extending a inode and - * everything went fine. - */ - if (!truncate && inode->i_nlink && - !list_empty(&EXT4_I(inode)->i_orphan)) - ext4_orphan_del(handle, inode); - ext4_journal_stop(handle); - if (truncate) { - ext4_truncate_failed_write(inode); -orphan_del: - /* - * If truncate failed early the inode might still be on the - * orphan list; we need to make sure the inode is removed from - * the orphan list in that case. - */ - if (inode->i_nlink) - ext4_orphan_del(NULL, inode); - } - return ret; + return 0; } const struct iomap_ops ext4_iomap_ops = { From patchwork Sun Sep 8 23:19:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Bobrowski X-Patchwork-Id: 11137115 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A794417E6 for ; Sun, 8 Sep 2019 23:19:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7DDCC20828 for ; Sun, 8 Sep 2019 23:19:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=mbobrowski-org.20150623.gappssmtp.com header.i=@mbobrowski-org.20150623.gappssmtp.com header.b="F01PsGlp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731278AbfIHXTi (ORCPT ); Sun, 8 Sep 2019 19:19:38 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:38619 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731260AbfIHXTi (ORCPT ); Sun, 8 Sep 2019 19:19:38 -0400 Received: by mail-pg1-f195.google.com with SMTP id d10so6704934pgo.5 for ; Sun, 08 Sep 2019 16:19:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mbobrowski-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=gx5XXdzXQF7G+1VrbudI1A9EwDDJ+tyVBcppt9K0Szs=; b=F01PsGlpkKUdf7ARLGoLElKXYgoOD/rlV0S5n7k8NS7EP45gvPuJjctcl+F0L4k9bt pDQXERm5AzCRpD3dA1v0uoQIDzBWCX1AGcdXTMQ2swDhFSRUaF8LIthjXc+fZIig+zLQ s/7z5bNVbrZwqKM7RAAQG1VijKSr88f27jHgnsbY0tUZfcsG5/SBvOqpzcNzXQsn8XB/ J0MXTuCwsEmPEBxptasCFIhg26bDPdRLBq4eFmmB8B0TL4o7ap7flYLw3uwnqls7GuxD RgtfGkyD0IuiBJ/B8ZSqxJlUlH9WpzpR5SxzDVfsxeA6+rArw29HoA72JuYPlezWpxCG +EkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=gx5XXdzXQF7G+1VrbudI1A9EwDDJ+tyVBcppt9K0Szs=; b=Pt9ULKZZk9fxc9u+6WsC8dAVTHx+ejFxCbUthMMQjJZjkUp+oGAf2pzWLIGxd2opfD /wQDDo3jKnmjVhhhlupi3L0HHmllBBoQwqKVjoCdiKt0FYwrXrYAVUdfpGaTZwzsKe3I m9dvr/CC3MvJrwnbnYIpp0/WHQZggqKDC6/4ku/agKFzmvAAN5Ex4t+PFalyJdePEfAi pWu7db4YG/w6oPlHnqGuHzRUlsUJxrXRLAZprRDXQNavojg4fw23fFE5gi7mUKxt1k5C L9y/AdRMQ9rhozsMHXspxSpmCyf7SAQu39va1xUhChv60FnAorBIi2snWlyrG/S12NVf ES9w== X-Gm-Message-State: APjAAAVI5OYM3RYRt9O6/UM1ZYefK+aLT/zsyTQbTLcPuNtqDU6SjNW6 gWMIx0sDSucvg+OxXqsANsBZ X-Google-Smtp-Source: APXvYqz7aWiON36Pum31Iaiaxq6K6qf3NBcQmcHZejX3zQ68ZdEeugxZEoHLn+LUsyAMIyFjFw0/qg== X-Received: by 2002:a65:5cca:: with SMTP id b10mr19318386pgt.365.1567984777787; Sun, 08 Sep 2019 16:19:37 -0700 (PDT) Received: from bobrowski ([110.232.114.101]) by smtp.gmail.com with ESMTPSA id u5sm12572075pfl.25.2019.09.08.16.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 16:19:37 -0700 (PDT) Date: Mon, 9 Sep 2019 09:19:31 +1000 From: Matthew Bobrowski To: tytso@mit.edu, jack@suse.cz, adilger.kernel@dilger.ca Cc: linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, david@fromorbit.com, hch@infradead.org, darrick.wong@oracle.com Subject: [PATCH v2 3/6] iomap: modify ->end_io() calling convention Message-ID: <8368d2ea5f2e80fed7fbba3685b0d3c1e5ff742a.1567978633.git.mbobrowski@mbobrowski.org> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch modifies the calling convention for the iomap ->end_io() callback. Rather than passing either dio->error or dio->size as the 'size' argument, we instead pass both dio->error and dio->size values separately. In the instance that an error occurred during a write, we currently cannot determine whether any blocks have been allocated beyond the current EOF and data has subsequently been written to these blocks within the ->end_io() callback. As a result, we cannot judge whether we should take the truncate failed write path. Having both dio->error and dio->size will allow us to perform such checks within this callback. Signed-off-by: Matthew Bobrowski --- fs/iomap/direct-io.c | 9 +++------ fs/xfs/xfs_file.c | 8 +++++--- include/linux/iomap.h | 4 ++-- 3 files changed, 10 insertions(+), 11 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 10517cea9682..2ccf1c6460d4 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -77,13 +77,10 @@ static ssize_t iomap_dio_complete(struct iomap_dio *dio) loff_t offset = iocb->ki_pos; ssize_t ret; - if (dio->end_io) { - ret = dio->end_io(iocb, - dio->error ? dio->error : dio->size, - dio->flags); - } else { + if (dio->end_io) + ret = dio->end_io(iocb, dio->size, dio->error, dio->flags); + else ret = dio->error; - } if (likely(!ret)) { ret = dio->size; diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 28101bbc0b78..d49759008c54 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -369,21 +369,23 @@ static int xfs_dio_write_end_io( struct kiocb *iocb, ssize_t size, + int error, unsigned flags) { struct inode *inode = file_inode(iocb->ki_filp); struct xfs_inode *ip = XFS_I(inode); loff_t offset = iocb->ki_pos; unsigned int nofs_flag; - int error = 0; trace_xfs_end_io_direct_write(ip, offset, size); if (XFS_FORCED_SHUTDOWN(ip->i_mount)) return -EIO; - if (size <= 0) - return size; + if (error) + return error; + if (!size) + return 0; /* * Capture amount written on completion as we can't reliably account diff --git a/include/linux/iomap.h b/include/linux/iomap.h index bc499ceae392..d983cdcf2e72 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -188,8 +188,8 @@ sector_t iomap_bmap(struct address_space *mapping, sector_t bno, */ #define IOMAP_DIO_UNWRITTEN (1 << 0) /* covers unwritten extent(s) */ #define IOMAP_DIO_COW (1 << 1) /* covers COW extent(s) */ -typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t ret, - unsigned flags); +typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t size, + int error, unsigned int flags); ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, const struct iomap_ops *ops, iomap_dio_end_io_t end_io); int iomap_dio_iopoll(struct kiocb *kiocb, bool spin); From patchwork Sun Sep 8 23:19:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Bobrowski X-Patchwork-Id: 11137117 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99E8D14ED for ; Sun, 8 Sep 2019 23:19:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 78C7420828 for ; Sun, 8 Sep 2019 23:19:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=mbobrowski-org.20150623.gappssmtp.com header.i=@mbobrowski-org.20150623.gappssmtp.com header.b="VX1ufSX+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731288AbfIHXTv (ORCPT ); Sun, 8 Sep 2019 19:19:51 -0400 Received: from mail-pf1-f196.google.com ([209.85.210.196]:46331 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731260AbfIHXTv (ORCPT ); Sun, 8 Sep 2019 19:19:51 -0400 Received: by mail-pf1-f196.google.com with SMTP id q5so7965132pfg.13 for ; Sun, 08 Sep 2019 16:19:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mbobrowski-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=BEDKpXPdUTV7XE4Q14Y66bFncWGnnigD83Lkh90WyW8=; b=VX1ufSX+EkshMc6s83N6qaf+btBEf9yhfbW6461tj53vckr/OoZpr8badf1c6bVRW2 Xy4o50K2g/HgazBt17HSZx2wlmvSA+iDTiixxTMb88DQgpdWJ1zGSpY8ovSwe2fKg2mM 8g0CwKKXWpzS+GGk7KjRMb/HnGsU2At8MlqcSRPE6bxllUHdnQiAf7ZGiAaTsIx0xsGj JMK9/lFKyEjE9wfOqSxxyTu3W7n9yxZGa5FD1c6rQaqgQaQDQY+Vn+o7XNj9r2+XpBAL gWsfOhupO5gXSKP4xgGU1dCQ9tjrtZT+Oh2KfPucABgzCcf6zXbuAq0Ys/h6fENTWdVm Nhkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=BEDKpXPdUTV7XE4Q14Y66bFncWGnnigD83Lkh90WyW8=; b=Mhh/qkh8cIJYR+u+QOdHBC/WBVSJtrQQalIKz/TGrF+NdKk0xfZzZapMW8/NlwJAXO UsyT5Kf8L993DjPGAdVQZ3aFtbmiIcSJ03oC9QxRgjfiykLQxvAalSE3ZQt7VAWZ3T3/ wBaLJcwbKxBWbLYrs6+OASorqFzExh1hv+4ymN2aY1977qhHdtLKa/GHskLZxKt7HY86 ueof3FP5uamlE3XbwaGBRCW5mRQdZqvBD/t+i9LTV4XjNK1WCWzZUn20X0VUnVMj8xC/ 2scA6P+R1lfTy1Ah6WqZ12ieW8YMYYtasDWFBtgWHOAzlqwQfzEDh6CgNtJ/b89+uDzF bnbw== X-Gm-Message-State: APjAAAXeIIyVZKkdBBgtftAIpHfZOdQQToF5pYhu5z22SywPjExGCHrt iPva2n1pE9mz1XTzqaXrTIgN X-Google-Smtp-Source: APXvYqy3mQm3z49ZC/NdLW7xH3ra53grqxkIsg20iZa9ZuVw7JEBHIfynKTyoqo6TLwR4fEcaiuMBg== X-Received: by 2002:a17:90a:8c16:: with SMTP id a22mr13087982pjo.111.1567984790208; Sun, 08 Sep 2019 16:19:50 -0700 (PDT) Received: from bobrowski ([110.232.114.101]) by smtp.gmail.com with ESMTPSA id v18sm10904747pgl.87.2019.09.08.16.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 16:19:49 -0700 (PDT) Date: Mon, 9 Sep 2019 09:19:44 +1000 From: Matthew Bobrowski To: tytso@mit.edu, jack@suse.cz, adilger.kernel@dilger.ca Cc: linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, david@fromorbit.com, hch@infradead.org, darrick.wong@oracle.com Subject: [PATCH v2 4/6] ext4: reorder map.m_flags checks in ext4_iomap_begin() Message-ID: <38c2c1dd6f62f82e485b1a767ddeb49606439d67.1567978633.git.mbobrowski@mbobrowski.org> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For iomap direct IO write code path changes, we need to accommodate for the case where the block mapping flags passed to ext4_map_blocks() will result in m_flags having both EXT4_MAP_MAPPED and EXT4_MAP_UNWRITTEN bits set. In order for the allocated unwritten extents to be converted properly in the end_io handler, iomap->type must be set to IOMAP_UNWRITTEN, so we need to reshuffle the conditional statement in order to achieve this. This change is a no-op for DAX code path as the block mapping flag passed to ext4_map_blocks() when IS_DAX(inode) never results in EXT4_MAP_MAPPED and EXT4_MAP_UNWRITTEN being set at once. Signed-off-by: Matthew Bobrowski Reviewed-by: Ritesh Harjani --- fs/ext4/inode.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 761ce6286b05..efb184928e51 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -3581,10 +3581,21 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length, iomap->type = delalloc ? IOMAP_DELALLOC : IOMAP_HOLE; iomap->addr = IOMAP_NULL_ADDR; } else { - if (map.m_flags & EXT4_MAP_MAPPED) { - iomap->type = IOMAP_MAPPED; - } else if (map.m_flags & EXT4_MAP_UNWRITTEN) { + /* + * Flags passed to ext4_map_blocks() for direct IO + * writes can result in m_flags having both + * EXT4_MAP_MAPPED and EXT4_MAP_UNWRITTEN bits set. In + * order for allocated unwritten extents to be + * converted to written extents in the end_io handler + * correctly, we need to ensure that the iomap->type + * is also set appropriately in that case. Thus, we + * need to check whether EXT4_MAP_UNWRITTEN is set + * first. + */ + if (map.m_flags & EXT4_MAP_UNWRITTEN) { iomap->type = IOMAP_UNWRITTEN; + } else if (map.m_flags & EXT4_MAP_MAPPED) { + iomap->type = IOMAP_MAPPED; } else { WARN_ON_ONCE(1); return -EIO; From patchwork Sun Sep 8 23:19:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Bobrowski X-Patchwork-Id: 11137119 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7989A14ED for ; Sun, 8 Sep 2019 23:20:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 33E29216C8 for ; Sun, 8 Sep 2019 23:20:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=mbobrowski-org.20150623.gappssmtp.com header.i=@mbobrowski-org.20150623.gappssmtp.com header.b="aRUJnTyT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731282AbfIHXUD (ORCPT ); Sun, 8 Sep 2019 19:20:03 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:45231 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731245AbfIHXUC (ORCPT ); Sun, 8 Sep 2019 19:20:02 -0400 Received: by mail-pf1-f193.google.com with SMTP id y72so7955482pfb.12 for ; Sun, 08 Sep 2019 16:20:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mbobrowski-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=ZXe8GCRxPXTndFWuYeSWB+6dNeh/wQOQbmrtOTcAGj8=; b=aRUJnTyTOdGpYdf5q51USGqiRac27Am5gV26rMbNCYSAIEofNGYw+ppJ7JgtfVSmVQ ztBImzq2/yGgedfUuxMY6gq6ESOQmvgamcUnKbyJ2sCri3i2XyF0NL/wIbsIQ95veTzn foGGYLXsXaBb4LHv93dPscw5wG8ODcNXEEbRSycesEpQTM9uMGsz+QFnDjRt4hSR9HEf jyNl2lxUUS7iW/uwSBj3xczdXn2WVrkgL7s+B+wsdV55XSJ2NM9LS2RyD7Fzn4mf6u2O RcfZYE+aU2iEu+d2fB2fJDlEUy4DPQWAD+rasfTCggBB3wCtW3j2JUlXBNiuESTIOwIh xPpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=ZXe8GCRxPXTndFWuYeSWB+6dNeh/wQOQbmrtOTcAGj8=; b=LSOQwS+Y1xQL5kdA4frL0ngezRxMra2DxUlpzOp85ccFSAkOfxWQ1dVCWfzAFvQTUz Tp3zvezBD0AjMJV34ysFmktGEX8pzzJMZ8nrTtuB6w7EcvVpWvCEcn4ft7TzCcVw8m6d LlJNsEYOaFlGoJoxXE39Hbb6Q5ODmEKVXAhwuYJZMSwtLZCw4o8E4B26wy3h2CdXF2Qs EsdoR1VBvGsIT93jkyYF3jX8atITqmI+fgvOAf7iRiT2qlWrcxi8CvmGVOUmS8hAs2Q1 OTLzkJn8LtnSMX5Q36ZeCcrfbSjXQaQxLDO2s5QH25CkT9Afo/pc5vwpIMHBJPpApKN4 NSsQ== X-Gm-Message-State: APjAAAVawl6iGF7Fsa+gbyTPDBR/K+sP8lolF7UsxCYWFFxEOwiC5KpQ 6eQkfCaKUHiq/LkPXMlzCmbU X-Google-Smtp-Source: APXvYqzmi1IdmMHCJgeLG02tJ2axSk8U5rWDzebNs1HQFVl16EVwRIpz++mi8+H0nuTzSFvxO2E52A== X-Received: by 2002:a65:534c:: with SMTP id w12mr10249267pgr.51.1567984801761; Sun, 08 Sep 2019 16:20:01 -0700 (PDT) Received: from bobrowski ([110.232.114.101]) by smtp.gmail.com with ESMTPSA id z189sm13760624pfb.137.2019.09.08.16.19.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 16:20:01 -0700 (PDT) Date: Mon, 9 Sep 2019 09:19:55 +1000 From: Matthew Bobrowski To: tytso@mit.edu, jack@suse.cz, adilger.kernel@dilger.ca Cc: linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, david@fromorbit.com, hch@infradead.org, darrick.wong@oracle.com Subject: [PATCH v2 5/6] ext4: introduce direct IO write path using iomap infrastructure Message-ID: <7c2f0ee02b2659d5a45f3e30dbee66b443b5ea0a.1567978633.git.mbobrowski@mbobrowski.org> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch introduces a new direct IO write code path implementation that makes use of the iomap infrastructure. All direct IO write operations are now passed from the ->write_iter() callback to the new function ext4_dio_write_iter(). This function is responsible for calling into iomap infrastructure via iomap_dio_rw(). Snippets of the direct IO code from within ext4_file_write_iter(), such as checking whether the IO request is unaligned asynchronous IO, or whether it will ber overwriting allocated and initialized blocks has been moved out and into ext4_dio_write_iter(). The block mapping flags that are passed to ext4_map_blocks() from within ext4_dio_get_block() and friends have effectively been taken out and introduced within the ext4_iomap_begin(). If ext4_map_blocks() happens to have instantiated blocks beyond the i_size, then we attempt to place the inode onto the orphan list. Despite being able to perform i_size extension checking earlier on in the direct IO code path, it makes most sense to perform this bit post successful block allocation. The ->end_io() callback ext4_dio_write_end_io() is responsible for removing the inode from the orphan list and determining if we should truncate a failed write in the case of an error. We also convert a range of unwritten extents to written if IOMAP_DIO_UNWRITTEN is set and perform the necessary i_size/i_disksize extension if the iocb->ki_pos + dio->size > i_size_read(inode). In the instance of a short write, we fallback to buffered IO and complete whatever is left the 'iter'. Any blocks that may have been allocated in preparation for direct IO will be reused by buffered IO, so there's no issue with leaving allocated blocks beyond EOF. Signed-off-by: Matthew Bobrowski Reviewed-by: Ritesh Harjani Reviewed-by: Ritesh Harjani --- fs/ext4/file.c | 219 +++++++++++++++++++++++++++++++++--------------- fs/ext4/inode.c | 57 ++++++++++--- 2 files changed, 198 insertions(+), 78 deletions(-) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 8e586198f6e6..bf22425a6a6f 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -29,6 +29,7 @@ #include #include #include +#include #include "ext4.h" #include "ext4_jbd2.h" #include "xattr.h" @@ -217,6 +218,14 @@ static ssize_t ext4_write_checks(struct kiocb *iocb, struct iov_iter *from) if (ret <= 0) return ret; + ret = file_remove_privs(iocb->ki_filp); + if (ret) + return 0; + + ret = file_update_time(iocb->ki_filp); + if (ret) + return 0; + if (unlikely(IS_IMMUTABLE(inode))) return -EPERM; @@ -234,6 +243,34 @@ static ssize_t ext4_write_checks(struct kiocb *iocb, struct iov_iter *from) return iov_iter_count(from); } +static ssize_t ext4_buffered_write_iter(struct kiocb *iocb, + struct iov_iter *from) +{ + ssize_t ret; + struct inode *inode = file_inode(iocb->ki_filp); + + if (iocb->ki_flags & IOCB_NOWAIT) + return -EOPNOTSUPP; + + if (!inode_trylock(inode)) + inode_lock(inode); + + ret = ext4_write_checks(iocb, from); + if (ret <= 0) + goto out; + + current->backing_dev_info = inode_to_bdi(inode); + ret = generic_perform_write(iocb->ki_filp, from, iocb->ki_pos); + current->backing_dev_info = NULL; +out: + inode_unlock(inode); + if (likely(ret > 0)) { + iocb->ki_pos += ret; + ret = generic_write_sync(iocb, ret); + } + return ret; +} + static int ext4_handle_inode_extension(struct inode *inode, loff_t offset, ssize_t len, size_t count) { @@ -311,6 +348,118 @@ static int ext4_handle_failed_inode_extension(struct inode *inode, loff_t size) return ret; } +/* + * For a write that extends the inode size, ext4_dio_write_iter() will + * wait for the write to complete. Consequently, operations performed + * within this function are still covered by the inode_lock(). + */ +static int ext4_dio_write_end_io(struct kiocb *iocb, ssize_t size, int error, + unsigned int flags) +{ + int ret = 0; + loff_t offset = iocb->ki_pos; + struct inode *inode = file_inode(iocb->ki_filp); + + if (error) { + ret = ext4_handle_failed_inode_extension(inode, offset + size); + return ret ? ret : error; + } + + if (flags & IOMAP_DIO_UNWRITTEN) { + ret = ext4_convert_unwritten_extents(NULL, inode, + offset, size); + if (ret) + return ret; + } + + if (offset + size > i_size_read(inode)) { + ret = ext4_handle_inode_extension(inode, offset, size, 0); + if (ret) + return ret; + } + return ret; +} + +static ssize_t ext4_dio_write_iter(struct kiocb *iocb, struct iov_iter *from) +{ + ssize_t ret; + loff_t offset = iocb->ki_pos; + size_t count = iov_iter_count(from); + struct inode *inode = file_inode(iocb->ki_filp); + bool extend = false, overwrite = false, unaligned_aio = false; + + if (!inode_trylock(inode)) { + if (iocb->ki_flags & IOCB_NOWAIT) + return -EAGAIN; + inode_lock(inode); + } + + if (!ext4_dio_checks(inode)) { + inode_unlock(inode); + /* + * Fallback to buffered IO if the operation on the + * inode is not supported by direct IO. + */ + return ext4_buffered_write_iter(iocb, from); + } + + ret = ext4_write_checks(iocb, from); + if (ret <= 0) { + inode_unlock(inode); + return ret; + } + + /* + * Unaligned direct AIO must be serialized among each other as + * the zeroing of partial blocks of two competing unaligned + * AIOs can result in data corruption. + */ + if (ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS) && + !is_sync_kiocb(iocb) && ext4_unaligned_aio(inode, from, offset)) { + unaligned_aio = true; + inode_dio_wait(inode); + } + + /* + * Determine whether the IO operation will overwrite allocated + * and initialized blocks. If so, check to see whether it is + * possible to take the dioread_nolock path. + */ + if (!unaligned_aio && ext4_overwrite_io(inode, offset, count) && + ext4_should_dioread_nolock(inode)) { + overwrite = true; + downgrade_write(&inode->i_rwsem); + } + + if (offset + count > i_size_read(inode) || + offset + count > EXT4_I(inode)->i_disksize) { + ext4_update_i_disksize(inode, inode->i_size); + extend = true; + } + + ret = iomap_dio_rw(iocb, from, &ext4_iomap_ops, ext4_dio_write_end_io); + + /* + * Unaligned direct AIO must be the only IO in flight or else + * any overlapping aligned IO after unaligned IO might result + * in data corruption. We also need to wait here in the case + * where the inode is being extended so that inode extension + * routines in ext4_dio_write_end_io() are covered by the + * inode_lock(). + */ + if (ret == -EIOCBQUEUED && (unaligned_aio || extend)) + inode_dio_wait(inode); + + if (overwrite) + inode_unlock_shared(inode); + else + inode_unlock(inode); + + if (ret >= 0 && iov_iter_count(from)) + return ext4_buffered_write_iter(iocb, from); + return ret; +} + #ifdef CONFIG_FS_DAX static ssize_t ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from) @@ -325,15 +474,10 @@ ext4_dax_write_iter(struct kiocb *iocb, struct iov_iter *from) return -EAGAIN; inode_lock(inode); } + ret = ext4_write_checks(iocb, from); if (ret <= 0) goto out; - ret = file_remove_privs(iocb->ki_filp); - if (ret) - goto out; - ret = file_update_time(iocb->ki_filp); - if (ret) - goto out; offset = iocb->ki_pos; ret = dax_iomap_rw(iocb, from, &ext4_iomap_ops); @@ -359,73 +503,16 @@ static ssize_t ext4_file_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct inode *inode = file_inode(iocb->ki_filp); - int o_direct = iocb->ki_flags & IOCB_DIRECT; - int unaligned_aio = 0; - int overwrite = 0; - ssize_t ret; if (unlikely(ext4_forced_shutdown(EXT4_SB(inode->i_sb)))) return -EIO; -#ifdef CONFIG_FS_DAX if (IS_DAX(inode)) return ext4_dax_write_iter(iocb, from); -#endif - if (!o_direct && (iocb->ki_flags & IOCB_NOWAIT)) - return -EOPNOTSUPP; - if (!inode_trylock(inode)) { - if (iocb->ki_flags & IOCB_NOWAIT) - return -EAGAIN; - inode_lock(inode); - } - - ret = ext4_write_checks(iocb, from); - if (ret <= 0) - goto out; - - /* - * Unaligned direct AIO must be serialized among each other as zeroing - * of partial blocks of two competing unaligned AIOs can result in data - * corruption. - */ - if (o_direct && ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS) && - !is_sync_kiocb(iocb) && - ext4_unaligned_aio(inode, from, iocb->ki_pos)) { - unaligned_aio = 1; - ext4_unwritten_wait(inode); - } - - iocb->private = &overwrite; - /* Check whether we do a DIO overwrite or not */ - if (o_direct && !unaligned_aio) { - if (ext4_overwrite_io(inode, iocb->ki_pos, iov_iter_count(from))) { - if (ext4_should_dioread_nolock(inode)) - overwrite = 1; - } else if (iocb->ki_flags & IOCB_NOWAIT) { - ret = -EAGAIN; - goto out; - } - } - - ret = __generic_file_write_iter(iocb, from); - /* - * Unaligned direct AIO must be the only IO in flight. Otherwise - * overlapping aligned IO after unaligned might result in data - * corruption. - */ - if (ret == -EIOCBQUEUED && unaligned_aio) - ext4_unwritten_wait(inode); - inode_unlock(inode); - - if (ret > 0) - ret = generic_write_sync(iocb, ret); - - return ret; - -out: - inode_unlock(inode); - return ret; + if (iocb->ki_flags & IOCB_DIRECT) + return ext4_dio_write_iter(iocb, from); + return ext4_buffered_write_iter(iocb, from); } #ifdef CONFIG_FS_DAX diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index efb184928e51..f52ad3065236 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -3513,11 +3513,13 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length, } } } else if (flags & IOMAP_WRITE) { - int dio_credits; handle_t *handle; - int retries = 0; + int dio_credits, retries = 0, m_flags = 0; - /* Trim mapping request to maximum we can map at once for DIO */ + /* + * Trim mapping request to maximum we can map at once + * for DIO. + */ if (map.m_len > DIO_MAX_BLOCKS) map.m_len = DIO_MAX_BLOCKS; dio_credits = ext4_chunk_trans_blocks(inode, map.m_len); @@ -3533,8 +3535,30 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length, if (IS_ERR(handle)) return PTR_ERR(handle); - ret = ext4_map_blocks(handle, inode, &map, - EXT4_GET_BLOCKS_CREATE_ZERO); + /* + * DAX and direct IO are the only two operations that + * are currently supported with IOMAP_WRITE. + */ + WARN_ON(!IS_DAX(inode) && !(flags & IOMAP_DIRECT)); + if (IS_DAX(inode)) + m_flags = EXT4_GET_BLOCKS_CREATE_ZERO; + else if (round_down(offset, i_blocksize(inode)) >= + i_size_read(inode)) + m_flags = EXT4_GET_BLOCKS_CREATE; + else if (ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)) + m_flags = EXT4_GET_BLOCKS_IO_CREATE_EXT; + + ret = ext4_map_blocks(handle, inode, &map, m_flags); + + /* + * We cannot fill holes in indirect tree based inodes + * as that could expose stale data in the case of a + * crash. Use the magic error code to fallback to + * buffered IO. + */ + if (!m_flags && !ret) + ret = -ENOTBLK; + if (ret < 0) { ext4_journal_stop(handle); if (ret == -ENOSPC && @@ -3544,13 +3568,14 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length, } /* - * If we added blocks beyond i_size, we need to make sure they - * will get truncated if we crash before updating i_size in - * ext4_iomap_end(). For faults we don't need to do that (and - * even cannot because for orphan list operations inode_lock is - * required) - if we happen to instantiate block beyond i_size, - * it is because we race with truncate which has already added - * the inode to the orphan list. + * If we added blocks beyond i_size, we need to make + * sure they will get truncated if we crash before + * updating the i_size. For faults we don't need to do + * that (and even cannot because for orphan list + * operations inode_lock is required) - if we happen + * to instantiate block beyond i_size, it is because + * we race with truncate which has already added the + * inode to the orphan list. */ if (!(flags & IOMAP_FAULT) && first_block + map.m_len > (i_size_read(inode) + (1 << blkbits) - 1) >> blkbits) { @@ -3612,6 +3637,14 @@ static int ext4_iomap_begin(struct inode *inode, loff_t offset, loff_t length, static int ext4_iomap_end(struct inode *inode, loff_t offset, loff_t length, ssize_t written, unsigned flags, struct iomap *iomap) { + /* + * Check to see whether an error occurred while writing data + * out to allocated blocks. If so, return the magic error code + * so that we fallback to buffered IO and reuse the blocks + * that were allocated in preparation for the direct IO write. + */ + if (flags & IOMAP_DIRECT && written == 0) + return -ENOTBLK; return 0; } From patchwork Sun Sep 8 23:20:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Bobrowski X-Patchwork-Id: 11137121 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 33748912 for ; Sun, 8 Sep 2019 23:20:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ECB1F218AE for ; Sun, 8 Sep 2019 23:20:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=mbobrowski-org.20150623.gappssmtp.com header.i=@mbobrowski-org.20150623.gappssmtp.com header.b="ffKY96pY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731299AbfIHXUO (ORCPT ); Sun, 8 Sep 2019 19:20:14 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:33080 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731252AbfIHXUO (ORCPT ); Sun, 8 Sep 2019 19:20:14 -0400 Received: by mail-pg1-f195.google.com with SMTP id n190so6721569pgn.0 for ; Sun, 08 Sep 2019 16:20:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mbobrowski-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Tr2zH7micHSXUeeeM7DiKZNyewx1wfbg0fdC6H8DW7M=; b=ffKY96pYazV0mdsKjw5Pvj1amVHUmcHb+AIKzJif0Uy6PLpoD3lP0YS5LAvNmo2J/2 4Q/t/sDGiAzFf5LMuFgFcESUKUM6lSaJxrVPlfCzUNwtZBMK12j46LJ6eMAVs3/i+atV DYkaYt3lAX8kW+t8gMBlXq1i3TxxDoD1P/CO7spqVd+aOoOQsCTPhSTzcnPMhUPgUSZ0 zkZ3ga174C+g+FGaQ/yxWX9AiUkVWhN7LIW9JptLtFu4lJLgfgdgICWGYWzG7XB14EUM tTm3zRNBKsSQh9RsNaC7CpW00pCThzH+s/HTBj8K9weM1aL7Y+P6dKl+aDeLWS/BcaUc ct9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Tr2zH7micHSXUeeeM7DiKZNyewx1wfbg0fdC6H8DW7M=; b=hZYmIvyFd9rrxXPeViIfjzG2ZZRGbu3PoQjI5umY9WNaP1XBgGKEg8C4thvZZlbnZf h9Th9jhJd3yClAyGckB+BdDrUmZRhTZwIkk60YdG3rneicmSps2CiswNEZVjah40LyHu hjJ87YNdhK79m540K09lEMutsm6uzL9XtfNyk0hbSpubkEb9luR9BUWqOQyY0wySKcIu Copo+Fg/dnIxeGHaM3w1EYcnIrCc4jsZyg4UrszoFAXItvNw0wag30rmDw0bGAMats2i d51nvjONxnPcfwiN8O441sSUigPwOlBS3F6dz2MWxODmjXHdgdUAS5PIcn6Yt4iIbR0d kqsA== X-Gm-Message-State: APjAAAXFEsspfP7SDsu3OATCpOKWdwNvpCe3WgCAE0GRbHXsmasgjx++ 4rWjA7vayKazffRvE904CZN1maZbSCgF X-Google-Smtp-Source: APXvYqxQ6oNep5VHFsKmVgQiaWDI0Ef3/kXMOzS/oR8u9ScuSWOi9AA4aagTlWHFEvSVokg66W0f1Q== X-Received: by 2002:a17:90a:aa95:: with SMTP id l21mr1941500pjq.69.1567984813165; Sun, 08 Sep 2019 16:20:13 -0700 (PDT) Received: from bobrowski ([110.232.114.101]) by smtp.gmail.com with ESMTPSA id g24sm12828119pfo.178.2019.09.08.16.20.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Sep 2019 16:20:12 -0700 (PDT) Date: Mon, 9 Sep 2019 09:20:07 +1000 From: Matthew Bobrowski To: tytso@mit.edu, jack@suse.cz, adilger.kernel@dilger.ca Cc: linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org, david@fromorbit.com, hch@infradead.org, darrick.wong@oracle.com Subject: [PATCH v2 6/6] ext4: cleanup legacy buffer_head direct IO code Message-ID: <8c22105d832117969690af05d13782e69fb7619a.1567978633.git.mbobrowski@mbobrowski.org> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Remove buffer_head direct IO code that is now redundant as a result of porting across the read/write paths to use iomap infrastructure. Signed-off-by: Matthew Bobrowski Reviewed-by: Ritesh Harjani --- fs/ext4/ext4.h | 3 - fs/ext4/extents.c | 11 +- fs/ext4/file.c | 7 - fs/ext4/inode.c | 398 +--------------------------------------------- 4 files changed, 5 insertions(+), 414 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index bf660aa7a9e0..2ab91815f52d 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1555,7 +1555,6 @@ enum { EXT4_STATE_NO_EXPAND, /* No space for expansion */ EXT4_STATE_DA_ALLOC_CLOSE, /* Alloc DA blks on close */ EXT4_STATE_EXT_MIGRATE, /* Inode is migrating */ - EXT4_STATE_DIO_UNWRITTEN, /* need convert on dio done*/ EXT4_STATE_NEWENTRY, /* File just added to dir */ EXT4_STATE_MAY_INLINE_DATA, /* may have in-inode data */ EXT4_STATE_EXT_PRECACHED, /* extents have been precached */ @@ -2522,8 +2521,6 @@ int ext4_get_block_unwritten(struct inode *inode, sector_t iblock, struct buffer_head *bh_result, int create); int ext4_get_block(struct inode *inode, sector_t iblock, struct buffer_head *bh_result, int create); -int ext4_dio_get_block(struct inode *inode, sector_t iblock, - struct buffer_head *bh_result, int create); int ext4_da_get_block_prep(struct inode *inode, sector_t iblock, struct buffer_head *bh, int create); int ext4_walk_page_buffers(handle_t *handle, diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c index 92266a2da7d6..a869e206bd81 100644 --- a/fs/ext4/extents.c +++ b/fs/ext4/extents.c @@ -1753,16 +1753,9 @@ ext4_can_extents_be_merged(struct inode *inode, struct ext4_extent *ex1, */ if (ext1_ee_len + ext2_ee_len > EXT_INIT_MAX_LEN) return 0; - /* - * The check for IO to unwritten extent is somewhat racy as we - * increment i_unwritten / set EXT4_STATE_DIO_UNWRITTEN only after - * dropping i_data_sem. But reserved blocks should save us in that - * case. - */ + if (ext4_ext_is_unwritten(ex1) && - (ext4_test_inode_state(inode, EXT4_STATE_DIO_UNWRITTEN) || - atomic_read(&EXT4_I(inode)->i_unwritten) || - (ext1_ee_len + ext2_ee_len > EXT_UNWRITTEN_MAX_LEN))) + (ext1_ee_len + ext2_ee_len > EXT_UNWRITTEN_MAX_LEN)) return 0; #ifdef AGGRESSIVE_TEST if (ext1_ee_len >= 4) diff --git a/fs/ext4/file.c b/fs/ext4/file.c index bf22425a6a6f..d4b9a82aed6c 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -155,13 +155,6 @@ static int ext4_release_file(struct inode *inode, struct file *filp) return 0; } -static void ext4_unwritten_wait(struct inode *inode) -{ - wait_queue_head_t *wq = ext4_ioend_wq(inode); - - wait_event(*wq, (atomic_read(&EXT4_I(inode)->i_unwritten) == 0)); -} - /* * This tests whether the IO in question is block-aligned or not. * Ext4 utilizes unwritten extents when hole-filling during direct IO, and they diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index f52ad3065236..a4f0749527c7 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -826,136 +826,6 @@ int ext4_get_block_unwritten(struct inode *inode, sector_t iblock, /* Maximum number of blocks we map for direct IO at once. */ #define DIO_MAX_BLOCKS 4096 -/* - * Get blocks function for the cases that need to start a transaction - - * generally difference cases of direct IO and DAX IO. It also handles retries - * in case of ENOSPC. - */ -static int ext4_get_block_trans(struct inode *inode, sector_t iblock, - struct buffer_head *bh_result, int flags) -{ - int dio_credits; - handle_t *handle; - int retries = 0; - int ret; - - /* Trim mapping request to maximum we can map at once for DIO */ - if (bh_result->b_size >> inode->i_blkbits > DIO_MAX_BLOCKS) - bh_result->b_size = DIO_MAX_BLOCKS << inode->i_blkbits; - dio_credits = ext4_chunk_trans_blocks(inode, - bh_result->b_size >> inode->i_blkbits); -retry: - handle = ext4_journal_start(inode, EXT4_HT_MAP_BLOCKS, dio_credits); - if (IS_ERR(handle)) - return PTR_ERR(handle); - - ret = _ext4_get_block(inode, iblock, bh_result, flags); - ext4_journal_stop(handle); - - if (ret == -ENOSPC && ext4_should_retry_alloc(inode->i_sb, &retries)) - goto retry; - return ret; -} - -/* Get block function for DIO reads and writes to inodes without extents */ -int ext4_dio_get_block(struct inode *inode, sector_t iblock, - struct buffer_head *bh, int create) -{ - /* We don't expect handle for direct IO */ - WARN_ON_ONCE(ext4_journal_current_handle()); - - if (!create) - return _ext4_get_block(inode, iblock, bh, 0); - return ext4_get_block_trans(inode, iblock, bh, EXT4_GET_BLOCKS_CREATE); -} - -/* - * Get block function for AIO DIO writes when we create unwritten extent if - * blocks are not allocated yet. The extent will be converted to written - * after IO is complete. - */ -static int ext4_dio_get_block_unwritten_async(struct inode *inode, - sector_t iblock, struct buffer_head *bh_result, int create) -{ - int ret; - - /* We don't expect handle for direct IO */ - WARN_ON_ONCE(ext4_journal_current_handle()); - - ret = ext4_get_block_trans(inode, iblock, bh_result, - EXT4_GET_BLOCKS_IO_CREATE_EXT); - - /* - * When doing DIO using unwritten extents, we need io_end to convert - * unwritten extents to written on IO completion. We allocate io_end - * once we spot unwritten extent and store it in b_private. Generic - * DIO code keeps b_private set and furthermore passes the value to - * our completion callback in 'private' argument. - */ - if (!ret && buffer_unwritten(bh_result)) { - if (!bh_result->b_private) { - ext4_io_end_t *io_end; - - io_end = ext4_init_io_end(inode, GFP_KERNEL); - if (!io_end) - return -ENOMEM; - bh_result->b_private = io_end; - ext4_set_io_unwritten_flag(inode, io_end); - } - set_buffer_defer_completion(bh_result); - } - - return ret; -} - -/* - * Get block function for non-AIO DIO writes when we create unwritten extent if - * blocks are not allocated yet. The extent will be converted to written - * after IO is complete by ext4_direct_IO_write(). - */ -static int ext4_dio_get_block_unwritten_sync(struct inode *inode, - sector_t iblock, struct buffer_head *bh_result, int create) -{ - int ret; - - /* We don't expect handle for direct IO */ - WARN_ON_ONCE(ext4_journal_current_handle()); - - ret = ext4_get_block_trans(inode, iblock, bh_result, - EXT4_GET_BLOCKS_IO_CREATE_EXT); - - /* - * Mark inode as having pending DIO writes to unwritten extents. - * ext4_direct_IO_write() checks this flag and converts extents to - * written. - */ - if (!ret && buffer_unwritten(bh_result)) - ext4_set_inode_state(inode, EXT4_STATE_DIO_UNWRITTEN); - - return ret; -} - -static int ext4_dio_get_block_overwrite(struct inode *inode, sector_t iblock, - struct buffer_head *bh_result, int create) -{ - int ret; - - ext4_debug("ext4_dio_get_block_overwrite: inode %lu, create flag %d\n", - inode->i_ino, create); - /* We don't expect handle for direct IO */ - WARN_ON_ONCE(ext4_journal_current_handle()); - - ret = _ext4_get_block(inode, iblock, bh_result, 0); - /* - * Blocks should have been preallocated! ext4_file_write_iter() checks - * that. - */ - WARN_ON_ONCE(!buffer_mapped(bh_result) || buffer_unwritten(bh_result)); - - return ret; -} - - /* * `handle' can be NULL if create is zero */ @@ -3653,268 +3523,6 @@ const struct iomap_ops ext4_iomap_ops = { .iomap_end = ext4_iomap_end, }; -static int ext4_end_io_dio(struct kiocb *iocb, loff_t offset, - ssize_t size, void *private) -{ - ext4_io_end_t *io_end = private; - - /* if not async direct IO just return */ - if (!io_end) - return 0; - - ext_debug("ext4_end_io_dio(): io_end 0x%p " - "for inode %lu, iocb 0x%p, offset %llu, size %zd\n", - io_end, io_end->inode->i_ino, iocb, offset, size); - - /* - * Error during AIO DIO. We cannot convert unwritten extents as the - * data was not written. Just clear the unwritten flag and drop io_end. - */ - if (size <= 0) { - ext4_clear_io_unwritten_flag(io_end); - size = 0; - } - io_end->offset = offset; - io_end->size = size; - ext4_put_io_end(io_end); - - return 0; -} - -/* - * Handling of direct IO writes. - * - * For ext4 extent files, ext4 will do direct-io write even to holes, - * preallocated extents, and those write extend the file, no need to - * fall back to buffered IO. - * - * For holes, we fallocate those blocks, mark them as unwritten - * If those blocks were preallocated, we mark sure they are split, but - * still keep the range to write as unwritten. - * - * The unwritten extents will be converted to written when DIO is completed. - * For async direct IO, since the IO may still pending when return, we - * set up an end_io call back function, which will do the conversion - * when async direct IO completed. - * - * If the O_DIRECT write will extend the file then add this inode to the - * orphan list. So recovery will truncate it back to the original size - * if the machine crashes during the write. - * - */ -static ssize_t ext4_direct_IO_write(struct kiocb *iocb, struct iov_iter *iter) -{ - struct file *file = iocb->ki_filp; - struct inode *inode = file->f_mapping->host; - struct ext4_inode_info *ei = EXT4_I(inode); - ssize_t ret; - loff_t offset = iocb->ki_pos; - size_t count = iov_iter_count(iter); - int overwrite = 0; - get_block_t *get_block_func = NULL; - int dio_flags = 0; - loff_t final_size = offset + count; - int orphan = 0; - handle_t *handle; - - if (final_size > inode->i_size || final_size > ei->i_disksize) { - /* Credits for sb + inode write */ - handle = ext4_journal_start(inode, EXT4_HT_INODE, 2); - if (IS_ERR(handle)) { - ret = PTR_ERR(handle); - goto out; - } - ret = ext4_orphan_add(handle, inode); - if (ret) { - ext4_journal_stop(handle); - goto out; - } - orphan = 1; - ext4_update_i_disksize(inode, inode->i_size); - ext4_journal_stop(handle); - } - - BUG_ON(iocb->private == NULL); - - /* - * Make all waiters for direct IO properly wait also for extent - * conversion. This also disallows race between truncate() and - * overwrite DIO as i_dio_count needs to be incremented under i_mutex. - */ - inode_dio_begin(inode); - - /* If we do a overwrite dio, i_mutex locking can be released */ - overwrite = *((int *)iocb->private); - - if (overwrite) - inode_unlock(inode); - - /* - * For extent mapped files we could direct write to holes and fallocate. - * - * Allocated blocks to fill the hole are marked as unwritten to prevent - * parallel buffered read to expose the stale data before DIO complete - * the data IO. - * - * As to previously fallocated extents, ext4 get_block will just simply - * mark the buffer mapped but still keep the extents unwritten. - * - * For non AIO case, we will convert those unwritten extents to written - * after return back from blockdev_direct_IO. That way we save us from - * allocating io_end structure and also the overhead of offloading - * the extent convertion to a workqueue. - * - * For async DIO, the conversion needs to be deferred when the - * IO is completed. The ext4 end_io callback function will be - * called to take care of the conversion work. Here for async - * case, we allocate an io_end structure to hook to the iocb. - */ - iocb->private = NULL; - if (overwrite) - get_block_func = ext4_dio_get_block_overwrite; - else if (!ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS) || - round_down(offset, i_blocksize(inode)) >= inode->i_size) { - get_block_func = ext4_dio_get_block; - dio_flags = DIO_LOCKING | DIO_SKIP_HOLES; - } else if (is_sync_kiocb(iocb)) { - get_block_func = ext4_dio_get_block_unwritten_sync; - dio_flags = DIO_LOCKING; - } else { - get_block_func = ext4_dio_get_block_unwritten_async; - dio_flags = DIO_LOCKING; - } - ret = __blockdev_direct_IO(iocb, inode, inode->i_sb->s_bdev, iter, - get_block_func, ext4_end_io_dio, NULL, - dio_flags); - - if (ret > 0 && !overwrite && ext4_test_inode_state(inode, - EXT4_STATE_DIO_UNWRITTEN)) { - int err; - /* - * for non AIO case, since the IO is already - * completed, we could do the conversion right here - */ - err = ext4_convert_unwritten_extents(NULL, inode, - offset, ret); - if (err < 0) - ret = err; - ext4_clear_inode_state(inode, EXT4_STATE_DIO_UNWRITTEN); - } - - inode_dio_end(inode); - /* take i_mutex locking again if we do a ovewrite dio */ - if (overwrite) - inode_lock(inode); - - if (ret < 0 && final_size > inode->i_size) - ext4_truncate_failed_write(inode); - - /* Handle extending of i_size after direct IO write */ - if (orphan) { - int err; - - /* Credits for sb + inode write */ - handle = ext4_journal_start(inode, EXT4_HT_INODE, 2); - if (IS_ERR(handle)) { - /* - * We wrote the data but cannot extend - * i_size. Bail out. In async io case, we do - * not return error here because we have - * already submmitted the corresponding - * bio. Returning error here makes the caller - * think that this IO is done and failed - * resulting in race with bio's completion - * handler. - */ - if (!ret) - ret = PTR_ERR(handle); - if (inode->i_nlink) - ext4_orphan_del(NULL, inode); - - goto out; - } - if (inode->i_nlink) - ext4_orphan_del(handle, inode); - if (ret > 0) { - loff_t end = offset + ret; - if (end > inode->i_size || end > ei->i_disksize) { - ext4_update_i_disksize(inode, end); - if (end > inode->i_size) - i_size_write(inode, end); - /* - * We're going to return a positive `ret' - * here due to non-zero-length I/O, so there's - * no way of reporting error returns from - * ext4_mark_inode_dirty() to userspace. So - * ignore it. - */ - ext4_mark_inode_dirty(handle, inode); - } - } - err = ext4_journal_stop(handle); - if (ret == 0) - ret = err; - } -out: - return ret; -} - -static ssize_t ext4_direct_IO_read(struct kiocb *iocb, struct iov_iter *iter) -{ - struct address_space *mapping = iocb->ki_filp->f_mapping; - struct inode *inode = mapping->host; - size_t count = iov_iter_count(iter); - ssize_t ret; - - /* - * Shared inode_lock is enough for us - it protects against concurrent - * writes & truncates and since we take care of writing back page cache, - * we are protected against page writeback as well. - */ - inode_lock_shared(inode); - ret = filemap_write_and_wait_range(mapping, iocb->ki_pos, - iocb->ki_pos + count - 1); - if (ret) - goto out_unlock; - ret = __blockdev_direct_IO(iocb, inode, inode->i_sb->s_bdev, - iter, ext4_dio_get_block, NULL, NULL, 0); -out_unlock: - inode_unlock_shared(inode); - return ret; -} - -static ssize_t ext4_direct_IO(struct kiocb *iocb, struct iov_iter *iter) -{ - struct file *file = iocb->ki_filp; - struct inode *inode = file->f_mapping->host; - size_t count = iov_iter_count(iter); - loff_t offset = iocb->ki_pos; - ssize_t ret; - -#ifdef CONFIG_FS_ENCRYPTION - if (IS_ENCRYPTED(inode) && S_ISREG(inode->i_mode)) - return 0; -#endif - - /* - * If we are doing data journalling we don't support O_DIRECT - */ - if (ext4_should_journal_data(inode)) - return 0; - - /* Let buffer I/O handle the inline data case. */ - if (ext4_has_inline_data(inode)) - return 0; - - trace_ext4_direct_IO_enter(inode, offset, count, iov_iter_rw(iter)); - if (iov_iter_rw(iter) == READ) - ret = ext4_direct_IO_read(iocb, iter); - else - ret = ext4_direct_IO_write(iocb, iter); - trace_ext4_direct_IO_exit(inode, offset, count, iov_iter_rw(iter), ret); - return ret; -} - /* * Pages can be marked dirty completely asynchronously from ext4's journalling * activity. By filemap_sync_pte(), try_to_unmap_one(), etc. We cannot do @@ -3952,7 +3560,7 @@ static const struct address_space_operations ext4_aops = { .bmap = ext4_bmap, .invalidatepage = ext4_invalidatepage, .releasepage = ext4_releasepage, - .direct_IO = ext4_direct_IO, + .direct_IO = noop_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, .error_remove_page = generic_error_remove_page, @@ -3969,7 +3577,7 @@ static const struct address_space_operations ext4_journalled_aops = { .bmap = ext4_bmap, .invalidatepage = ext4_journalled_invalidatepage, .releasepage = ext4_releasepage, - .direct_IO = ext4_direct_IO, + .direct_IO = noop_direct_IO, .is_partially_uptodate = block_is_partially_uptodate, .error_remove_page = generic_error_remove_page, }; @@ -3985,7 +3593,7 @@ static const struct address_space_operations ext4_da_aops = { .bmap = ext4_bmap, .invalidatepage = ext4_da_invalidatepage, .releasepage = ext4_releasepage, - .direct_IO = ext4_direct_IO, + .direct_IO = noop_direct_IO, .migratepage = buffer_migrate_page, .is_partially_uptodate = block_is_partially_uptodate, .error_remove_page = generic_error_remove_page,