From patchwork Sat Feb 3 07:11:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Hellwig X-Patchwork-Id: 13543907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1718C4828F for ; Sat, 3 Feb 2024 07:12:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 515B16B0088; Sat, 3 Feb 2024 02:12:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 49E356B0089; Sat, 3 Feb 2024 02:12:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 316D06B008A; Sat, 3 Feb 2024 02:12:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 209AB6B0088 for ; Sat, 3 Feb 2024 02:12:57 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 00D2A140536 for ; Sat, 3 Feb 2024 07:12:56 +0000 (UTC) X-FDA: 81749625594.25.5448A4D Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf24.hostedemail.com (Postfix) with ESMTP id 4A14F180005 for ; Sat, 3 Feb 2024 07:12:55 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=0fuITZBJ; spf=none (imf24.hostedemail.com: domain of BATV+a0ed29cff9639d748e7e+7468+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+a0ed29cff9639d748e7e+7468+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706944375; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GezxIWW+q+u2zC46RjSY4N2DxF7mdi1d7/13SlWu4e0=; b=M66CgJbzVMOHfcY+M7Iz1+P3rBmIT1jhjyn8X4iHvB/Xks3NuViuou+VziOOf1rdiwfPWV 2NuoHZo0wywV6DaoOMTHPrGEKKy/csYgV4BMaANBoWQXF5ekLQqrcP776xWiVIixvQpOEJ ds+gxH63gDAUyMWFuzMoHHnr/vlvSsc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706944375; a=rsa-sha256; cv=none; b=4SYCZDMljCR/X9u49DYzZCU+Tk3iiEX3OoNBK4IaBjBemr+M1U7hXgRTn4HhY6rI06tYrZ fkfwawpa4+yhI3RoXRo4GrJTfbWybytmbn9blcz0TJzJBCbSLi95B+UX6dk0Nev/x+dfgN a0lghirIJWCJApjx1bibp/Kw7Qb8+gc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=0fuITZBJ; spf=none (imf24.hostedemail.com: domain of BATV+a0ed29cff9639d748e7e+7468+infradead.org+hch@bombadil.srs.infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=BATV+a0ed29cff9639d748e7e+7468+infradead.org+hch@bombadil.srs.infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=GezxIWW+q+u2zC46RjSY4N2DxF7mdi1d7/13SlWu4e0=; b=0fuITZBJrufOpvkZIZVTWLsXDG Tb1yRAHTRSZM10wbA9mMQmPVY7Dc/QPRmF7qOsXh9cUUf9u8BZ4kCuh0PsgskRbJvPn1EXAWIHPo/ 2lUsRZffOvsZzX/H8/b9ym45gzAp3Xs0w0hhnCArnihPgDVLNKiYo/O+eRsH5bw/DH4TgV9psc3II Ri+1RmASwfnAIZaowi8VPt9u+QXKPsK7mH9qI+9Bnz/jojbY9mOU/Z1FWHmh4z/tvtaZZmu7dVmrK mTYhghZYwghLpblIlUFbA1bxNC9ycEhtE3cNU/inSmeuTiqmjjbnkEOFTaPl0XNmjuDmFeuE1ihUx NqEc+Dew==; Received: from [89.144.222.32] (helo=localhost) by bombadil.infradead.org with esmtpsa (Exim 4.97.1 #2 (Red Hat Linux)) id 1rWACr-0000000FkKP-28zg; Sat, 03 Feb 2024 07:12:54 +0000 From: Christoph Hellwig To: linux-mm@kvack.org Cc: Matthew Wilcox , Jan Kara , David Howells , Brian Foster , Christian Brauner , "Darrick J. Wong" , linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 12/13] writeback: add a writeback iterator Date: Sat, 3 Feb 2024 08:11:46 +0100 Message-Id: <20240203071147.862076-13-hch@lst.de> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240203071147.862076-1-hch@lst.de> References: <20240203071147.862076-1-hch@lst.de> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by bombadil.infradead.org. See http://www.infradead.org/rpr.html X-Rspamd-Queue-Id: 4A14F180005 X-Rspam-User: X-Stat-Signature: eg1jwa6upppmj8huiadm68rhkk5a6c4e X-Rspamd-Server: rspam03 X-HE-Tag: 1706944375-783204 X-HE-Meta: U2FsdGVkX1/S4kt3baM9e7eCdCPKk0cd6M7eiBizrU4qOI4Mkb+f5LOEXUV4IZ4cSLSBcHdpQKeRTq8v/grrKN6OqeOriVGSV3u26EY2N5vS1BMfM7LGCxElTqwipcDiKcobgkM1T1x01aYApQDcXepdhMqDVTtAH0Z5BGJgF4C/rnkFQGNp4UsCdRN2RFUGm+gUrcVoFT0bucC93lleesf9Y4KS46fUvr8z7QlUTwqD7Favhzmvwt/k5c05Hq2rUJO96Uvfavn/O8h3RfrIsYm2yosMBOpNuB8M69AIk0ySbqZvHDT9x8BtXf5L6VNE0CcuL2jJWAsP5ssZG15rs/a8LufkPhbfkwrcW08HXUQgsIgBGGYwKs02q+fYc+OCmnqGA1fCVNo7/6g2iE1mytUPWJbaRtdTDHesF6epBzFDd6ofM4NCK61pTAWnvp7J84X/HGm8wZ00RqjmYem9HWHMbX4o1Gi9NI9zOQiZAUn5EIuMje5yAutFELVur3bOKr54O7HDXf4IksXeCzUKMqINzHzjdL1hGHe0jSjRyiE3iGGERBw92l5O1+jFt972+7YGVCv4rQ7RwcypgZieBrYe5mUMt1U8eD3Ygv3oiWCHHxiWu7HCiHlo3lTHiTA9wu+jPiIprn41NREcm3E0Zsvol0wualZgviA9hfMAbPvV20Dayac6LOvb5VHRSAkN2VGNXhhji5nyyaGiqY4JKMl7V+WuI2c3plv7Ak2uhD2d7Ik0YiqdIahHCr7QR3Yqvq4cV4dJb5FyyJ5wz4PSaNaaMll/A7reYcyKg5xE0KGU0AyQ6MbVVkf4AFUjIfKeWhkGTyIBkgz886BzsBQvzQd8eUG9ycLrYJ5On6+F236SRNnYrFln9NLVUNQOPIWiMxmfZS9nb7bZFhA/W6XJDuK6lHTRwfT9FDAF36d+cZbhtH0voMp8EPvv39K/lICMM0i4qT6BLUSQ4z1ZXha eghLz6dX +5JGmi1ZOPONiwPdHyctQvhsgPldKMX22keCrmk5/NqNvAb7jZ8ivIPs3Z/A5pxCC75qpmtO/fYuzuNRvqsNSmonoiAdyS53vIjZpAEFa3h9kzFoH55lSAJBz5tophEYJUTg3GNV0w8c6B2pBm9Ux0oQ1ikeHXsnJsJhMwbZjcTq3aKHt7TXrx/4aqev/0ifn+mFbxClvsdZxWKtAkowwecizek7Aikfdd9MKvNTls8vJiJ5TMkEl2GPGTMc+IAt2tX9B21AUXcSuRDb8fnkxBJb8GVM/s910ue6re+TZeTXe5UIW9FtIYd1+n6Bsjn8/uA1I80eis52UYnDacS6vw62pkXPEnwi3iafKJkouspVBf82/G4bB0X8TwNGyAvb0xAStE/VtR4RdUW8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor the code left in write_cache_pages into an iterator that the file system can call to get the next folio for a writeback operation: struct folio *folio = NULL; while ((folio = writeback_iter(mapping, wbc, folio, &error))) { error = ; } The twist here is that the error value is passed by reference, so that the iterator can restore it when breaking out of the loop. Handling of the magic AOP_WRITEPAGE_ACTIVATE value stays outside the iterator and needs is just kept in the write_cache_pages legacy wrapper. in preparation for eventually killing it off. Heavily based on a for_each* based iterator from Matthew Wilcox. Signed-off-by: Christoph Hellwig Reviewed-by: Jan Kara Reviewed-by: Brian Foster --- include/linux/writeback.h | 4 + mm/page-writeback.c | 192 ++++++++++++++++++++++---------------- 2 files changed, 118 insertions(+), 78 deletions(-) diff --git a/include/linux/writeback.h b/include/linux/writeback.h index f67b3ea866a0fb..9845cb62e40b2d 100644 --- a/include/linux/writeback.h +++ b/include/linux/writeback.h @@ -82,6 +82,7 @@ struct writeback_control { /* internal fields used by the ->writepages implementation: */ struct folio_batch fbatch; pgoff_t index; + int saved_err; #ifdef CONFIG_CGROUP_WRITEBACK struct bdi_writeback *wb; /* wb this writeback is issued under */ @@ -366,6 +367,9 @@ int balance_dirty_pages_ratelimited_flags(struct address_space *mapping, bool wb_over_bg_thresh(struct bdi_writeback *wb); +struct folio *writeback_iter(struct address_space *mapping, + struct writeback_control *wbc, struct folio *folio, int *error); + typedef int (*writepage_t)(struct folio *folio, struct writeback_control *wbc, void *data); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 3abb053e70580e..5fe4cdb7dbd61a 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2325,18 +2325,18 @@ void __init page_writeback_init(void) } /** - * tag_pages_for_writeback - tag pages to be written by write_cache_pages + * tag_pages_for_writeback - tag pages to be written by writeback * @mapping: address space structure to write * @start: starting page index * @end: ending page index (inclusive) * * This function scans the page range from @start to @end (inclusive) and tags - * all pages that have DIRTY tag set with a special TOWRITE tag. The idea is - * that write_cache_pages (or whoever calls this function) will then use - * TOWRITE tag to identify pages eligible for writeback. This mechanism is - * used to avoid livelocking of writeback by a process steadily creating new - * dirty pages in the file (thus it is important for this function to be quick - * so that it can tag pages faster than a dirtying process can create them). + * all pages that have DIRTY tag set with a special TOWRITE tag. The caller + * can then use the TOWRITE tag to identify pages eligible for writeback. + * This mechanism is used to avoid livelocking of writeback by a process + * steadily creating new dirty pages in the file (thus it is important for this + * function to be quick so that it can tag pages faster than a dirtying process + * can create them). */ void tag_pages_for_writeback(struct address_space *mapping, pgoff_t start, pgoff_t end) @@ -2434,69 +2434,68 @@ static struct folio *writeback_get_folio(struct address_space *mapping, } /** - * write_cache_pages - walk the list of dirty pages of the given address space and write all of them. + * writeback_iter - iterate folio of a mapping for writeback * @mapping: address space structure to write - * @wbc: subtract the number of written pages from *@wbc->nr_to_write - * @writepage: function called for each page - * @data: data passed to writepage function + * @wbc: writeback context + * @folio: previously iterated folio (%NULL to start) + * @error: in-out pointer for writeback errors (see below) * - * If a page is already under I/O, write_cache_pages() skips it, even - * if it's dirty. This is desirable behaviour for memory-cleaning writeback, - * but it is INCORRECT for data-integrity system calls such as fsync(). fsync() - * and msync() need to guarantee that all the data which was dirty at the time - * the call was made get new I/O started against them. If wbc->sync_mode is - * WB_SYNC_ALL then we were called for data integrity and we must wait for - * existing IO to complete. - * - * To avoid livelocks (when other process dirties new pages), we first tag - * pages which should be written back with TOWRITE tag and only then start - * writing them. For data-integrity sync we have to be careful so that we do - * not miss some pages (e.g., because some other process has cleared TOWRITE - * tag we set). The rule we follow is that TOWRITE tag can be cleared only - * by the process clearing the DIRTY tag (and submitting the page for IO). - * - * To avoid deadlocks between range_cyclic writeback and callers that hold - * pages in PageWriteback to aggregate IO until write_cache_pages() returns, - * we do not loop back to the start of the file. Doing so causes a page - * lock/page writeback access order inversion - we should only ever lock - * multiple pages in ascending page->index order, and looping back to the start - * of the file violates that rule and causes deadlocks. + * This function returns the next folio for the writeback operation described by + * @wbc on @mapping and should be called in a while loop in the ->writepages + * implementation. * - * Return: %0 on success, negative error code otherwise + * To start the writeback operation, %NULL is passed in the @folio argument, and + * for every subsequent iteration the folio returned previously should be passed + * back in. + * + * If there was an error in the per-folio writeback inside the writeback_iter() + * loop, @error should be set to the error value. + * + * Once the writeback described in @wbc has finished, this function will return + * %NULL and if there was an error in any iteration restore it to @error. + * + * Note: callers should not manually break out of the loop using break or goto + * but must keep calling writeback_iter() until it returns %NULL. + * + * Return: the folio to write or %NULL if the loop is done. */ -int write_cache_pages(struct address_space *mapping, - struct writeback_control *wbc, writepage_t writepage, - void *data) +struct folio *writeback_iter(struct address_space *mapping, + struct writeback_control *wbc, struct folio *folio, int *error) { - int ret = 0; - int error; - struct folio *folio; - pgoff_t end; /* Inclusive */ - - if (wbc->range_cyclic) { - wbc->index = mapping->writeback_index; /* prev offset */ - end = -1; - } else { - wbc->index = wbc->range_start >> PAGE_SHIFT; - end = wbc->range_end >> PAGE_SHIFT; - } - if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages) - tag_pages_for_writeback(mapping, wbc->index, end); - - folio_batch_init(&wbc->fbatch); + if (!folio) { + folio_batch_init(&wbc->fbatch); + wbc->saved_err = *error = 0; - for (;;) { - folio = writeback_get_folio(mapping, wbc); - if (!folio) - break; + /* + * For range cyclic writeback we remember where we stopped so + * that we can continue where we stopped. + * + * For non-cyclic writeback we always start at the beginning of + * the passed in range. + */ + if (wbc->range_cyclic) + wbc->index = mapping->writeback_index; + else + wbc->index = wbc->range_start >> PAGE_SHIFT; - error = writepage(folio, wbc, data); + /* + * To avoid livelocks when other processes dirty new pages, we + * first tag pages which should be written back and only then + * start writing them. + * + * For data-integrity writeback we have to be careful so that we + * do not miss some pages (e.g., because some other process has + * cleared the TOWRITE tag we set). The rule we follow is that + * TOWRITE tag can be cleared only by the process clearing the + * DIRTY tag (and submitting the page for I/O). + */ + if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages) + tag_pages_for_writeback(mapping, wbc->index, + wbc_end(wbc)); + } else { wbc->nr_to_write -= folio_nr_pages(folio); - if (error == AOP_WRITEPAGE_ACTIVATE) { - folio_unlock(folio); - error = 0; - } + WARN_ON_ONCE(*error > 0); /* * For integrity writeback we have to keep going until we have @@ -2510,33 +2509,70 @@ int write_cache_pages(struct address_space *mapping, * wbc->nr_to_write or encounter the first error. */ if (wbc->sync_mode == WB_SYNC_ALL) { - if (error && !ret) - ret = error; + if (*error && !wbc->saved_err) + wbc->saved_err = *error; } else { - if (error || wbc->nr_to_write <= 0) + if (*error || wbc->nr_to_write <= 0) goto done; } } - /* - * For range cyclic writeback we need to remember where we stopped so - * that we can continue there next time we are called. If we hit the - * last page and there is more work to be done, wrap back to the start - * of the file. - * - * For non-cyclic writeback we always start looking up at the beginning - * of the file if we are called again, which can only happen due to - * -ENOMEM from the file system. - */ - folio_batch_release(&wbc->fbatch); - if (wbc->range_cyclic) - mapping->writeback_index = 0; - return ret; + folio = writeback_get_folio(mapping, wbc); + if (!folio) { + /* + * To avoid deadlocks between range_cyclic writeback and callers + * that hold pages in PageWriteback to aggregate I/O until + * the writeback iteration finishes, we do not loop back to the + * start of the file. Doing so causes a page lock/page + * writeback access order inversion - we should only ever lock + * multiple pages in ascending page->index order, and looping + * back to the start of the file violates that rule and causes + * deadlocks. + */ + if (wbc->range_cyclic) + mapping->writeback_index = 0; + + /* + * Return the first error we encountered (if there was any) to + * the caller. + */ + *error = wbc->saved_err; + } + return folio; done: folio_batch_release(&wbc->fbatch); if (wbc->range_cyclic) mapping->writeback_index = folio->index + folio_nr_pages(folio); + return NULL; +} + +/** + * write_cache_pages - walk the list of dirty pages of the given address space and write all of them. + * @mapping: address space structure to write + * @wbc: subtract the number of written pages from *@wbc->nr_to_write + * @writepage: function called for each page + * @data: data passed to writepage function + * + * Return: %0 on success, negative error code otherwise + * + * Note: please use writeback_iter() instead. + */ +int write_cache_pages(struct address_space *mapping, + struct writeback_control *wbc, writepage_t writepage, + void *data) +{ + struct folio *folio = NULL; + int error; + + while ((folio = writeback_iter(mapping, wbc, folio, &error))) { + error = writepage(folio, wbc, data); + if (error == AOP_WRITEPAGE_ACTIVATE) { + folio_unlock(folio); + error = 0; + } + } + return error; } EXPORT_SYMBOL(write_cache_pages);