From patchwork Fri Nov 8 17:43:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13868803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66766D64072 for ; Fri, 8 Nov 2024 17:45:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D6C06B00E7; Fri, 8 Nov 2024 12:45:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 084EA6B00E9; Fri, 8 Nov 2024 12:45:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF3B86B00E8; Fri, 8 Nov 2024 12:45:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BAA026B00E5 for ; Fri, 8 Nov 2024 12:45:31 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D2EAA1A0EB5 for ; Fri, 8 Nov 2024 17:45:30 +0000 (UTC) X-FDA: 82763652804.09.263473E Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) by imf24.hostedemail.com (Postfix) with ESMTP id CF5E7180020 for ; Fri, 8 Nov 2024 17:45:23 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=17eBT1gp; spf=pass (imf24.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.178 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731087701; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UUX+L5JwxoRfm7kNG+SFpzZJbOBH//rHSLdNBno0i1Q=; b=hP/Qn5AAnvj0gfE0kaaBZSZv+GUZqz0lwfhutge40Z4zm1J+FB1Iww8pRF9R4WDhn6dEq9 vuN06BPlQ0azJQlE1Yv/Knnd2Pti8CkfwOMdZRNYt4QYxdrsReAeI7uzS4etyfl6yWIh1/ 1Lp7rwonAUexHf51zIEnt7RQX14Pt/Y= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=17eBT1gp; spf=pass (imf24.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.178 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731087701; a=rsa-sha256; cv=none; b=Ui1/sntzfocx537Z3QyKi5WfvCh3KBNzoM8SENdFLV/tV3Rsyxsqr08d7bnXb91t1Hv0HG mHyAc55gHzU2ZSBDfrprss5QCm/Jn+znIlaVxfZ6qQ843CNIgHxP6a75B7EK+u07U/8H8t ZOFEYm6NeW69ATXz0lBbhXJAuxa6WcM= Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3e60966297fso1316671b6e.1 for ; Fri, 08 Nov 2024 09:45:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731087927; x=1731692727; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UUX+L5JwxoRfm7kNG+SFpzZJbOBH//rHSLdNBno0i1Q=; b=17eBT1gp1GuQErem6BJ5e1aRfkrMZ+N6rFD2MAFy5B0SQHcqN3aIWsyl0pIGGp+p1O jXsbWdPLOhyTny2ffdM0HwqwKXIfaWf1l0Y7ImeoCE3TY7KkAmfY/Mz/z8zyeZrqhUEP gPQdifcThJe8ZyWzJ0A7Eh8RWQZX5+03JvcMuxeG90sdF1K6occ2pqdWDV23nMlaICOy bmReoghjRRO8xRuvHZGl0Sp1pm+xwxBK0tTewnsJ8g2WFtn4snWqkvKwTyKnuiIlOHfo mipCNojMLXcUl4vwDUbht4AWWZusCbHt2N8E3D7MAyE8RvXBgID0//HmGch+dxYjAhYn Gy2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731087927; x=1731692727; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UUX+L5JwxoRfm7kNG+SFpzZJbOBH//rHSLdNBno0i1Q=; b=Wj6Qfl/PmkZWnUs2L1pQdyneUZ0iC0b2Pa/UoMGmRoiINrgUFBTr0rFrbLz+sa68Wb kVVPiUNDWMAVch3cYmkDeJFfQB0fObugVmpmkCUoZ6m7DuGO0IZpAp+rOs20YmJcd+7x hGGWhF0BqtdWklH495tUglCfzEHzQ03SIZYmE8AI26VnNa1qggnYAQPnUUpEBOILdk1w G3gZYWRddys8OmywIizm/jxGPXZBeoAzsCW+Dluc2u6VgIkC/hfx7KSKYw78ryFb5J2J HgJ+grP5ySm5FyHfKjQ6Q+4wX5e7zxgmy5aOdL6YBtiVr/YPdXS0HYw96X7CyfKIcd6i Tr7w== X-Gm-Message-State: AOJu0Yz0PX8iq1hDkpfuH5eAq4Pji1026/f73r/6M1pZkus8gWQj1jgF ET2iveAzYUaO2VUMvVIMnNge+YXk/zKBq+50Q7+pHEsqpxCUX86IGw8uNWyUGAlrrESUZyaJdKY fyqQ= X-Google-Smtp-Source: AGHT+IEeKABK38xxsKod0vZKixk/J7Qylyl5MmSX7mHW/jyfME5Ez4w2eu08IqUOJbkoLHt8qw1oZw== X-Received: by 2002:a05:6808:16a1:b0:3e7:63ce:1f26 with SMTP id 5614622812f47-3e7946a678fmr4543740b6e.24.1731087927472; Fri, 08 Nov 2024 09:45:27 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3e78cd28f80sm780969b6e.39.2024.11.08.09.45.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Nov 2024 09:45:26 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, Jens Axboe Subject: [PATCH 10/13] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Fri, 8 Nov 2024 10:43:33 -0700 Message-ID: <20241108174505.1214230-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241108174505.1214230-1-axboe@kernel.dk> References: <20241108174505.1214230-1-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CF5E7180020 X-Stat-Signature: p9q95sstwhxyre6gkw6tfeaip1mw6gpt X-Rspam-User: X-HE-Tag: 1731087923-560208 X-HE-Meta: U2FsdGVkX19tDmtdgRyDm8EF3JCfIcG4vPZCM8iz5Nxj3W9WWftPvVYLPhMSv3D7ixvLlPcS7NgYArygpWiW85JTDR2Apn6yMjFqn1/T79aKmOshP66opB9mDlnCMMeyAnx7w60mf6IWLCGaXlZ25kzi3d087PaymSKTQ2N6bP9fB39u6hX9NbhLS9BKBqLjyLeHBQGFDPfCJEInl7EBC4n39EvN3vYo60JQtZscyk/0Wu0XqTVV5z+m6mluTskmcarWioFKaJbjIYidEREtylFX2bLNi246oKgHfZKJV4LO6mBIx13BtzYftIm6qMr6P/GKl/iUIivYm3t0Ez8R6W3n0vqVL7RmUBLsvE3s5NkNbvn25xACZaCamLfeC3wDlJk2zPX34Nr0FhrRpAqdmhrQIeWnSorNmut+nIx+tK81CWY9zWWesQBV+lz0cu3gdGtZJgJCaYGPtY+H1THZXGRRRe10J8OFbkHUNa5JiwsV6Ohz4N7SATzsrju89+6rkRmw9hDYUZK3PNgdftvYw6KEmCIwpJpSWPYnN+X65FFZbnwpUaX/QuIAW/3OOTsT31rkpprXssJLnw9o9EUkWN0iUGy8mkeGqriwn8V/A52haxxfe7yryvXVGMrcKzIN5jOfPchf488pudVKsZyk0fO7zzYlnuRRBNHNvZ1ZLYoVcGFJr1m9x8wCapqib39G+ex3wG63680Q4pTlyX7iScwCuto5/x2FIlNf7vJuyUwo6RP2TarvPgy551wemeEe8C+7Y+fOM6zGNoYUv8jUzFt+jZ/vY+REu721rJ2HdJFA3zzS8p8HTaAgqKfyNxrZjqLWkGmc1L3gYftIYeUaB4XZQZt3rHXsXKUaMRDYZGALxQW7bReB/nYUY29av/lrRmaFKtKPxuRGmeyVYP0Y3VMW27dm+rwC4iRpmkTa+Wt1lRhkDRVNYp0VErLEhfpdtuKDaUD0/4LvTwctzHU nrgjswj4 vkx/iIFZfITb39f5uST22cT6qciSY42PZnfVOTZUFs4nYd49MYazOaMWIfd8Pwcc0yErm0knUjRHrF8QdIef9I/KvMfZbkbowqq3i7mzKH0h+s59D+W/X09uOCZ4XYAouYSYihAeRpp6Au188IKOM4Lwg8qPzsxEl0gVlurCBCkAVMRNWsw1AVM55zPzASqn77zJKR/1053VgreF3EbyOs5u2H3soX8gHbsAlrwo5bG/+KVVqweZkhiSSufd5GpUNjrgA5gulhqbjHmWZnZCmPnCdd9ICwwHTHB7iJfUIDlHxaZ+Ny8Mc/JqiFPp7OgERcJcUVSbNxBFDgl3Fn7DcstXpbFUCagrt4iKq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then writeback completion will drop the pages. The write_iter handler simply kicks off writeback for the pages, and writeback completion will take care of the rest. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec, MB=196035 2s: 132308MB/sec, MB=328147 3s: 132438MB/sec, MB=460586 4s: 116528MB/sec, MB=577115 5s: 103898MB/sec, MB=681014 6s: 108893MB/sec, MB=789907 7s: 99678MB/sec, MB=889586 8s: 106545MB/sec, MB=996132 9s: 106826MB/sec, MB=1102958 10s: 101544MB/sec, MB=1204503 11s: 111044MB/sec, MB=1315548 12s: 124257MB/sec, MB=1441121 13s: 116031MB/sec, MB=1557153 14s: 114540MB/sec, MB=1671694 15s: 115011MB/sec, MB=1786705 16s: 115260MB/sec, MB=1901966 17s: 116068MB/sec, MB=2018034 18s: 116096MB/sec, MB=2134131 where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe --- mm/filemap.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 1e455ca872b5..d4c5928c5e2a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1610,6 +1610,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); */ void folio_end_writeback(struct folio *folio) { + bool folio_uncached; + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); /* @@ -1631,6 +1633,7 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake_bit(). */ folio_get(folio); + folio_uncached = folio_test_clear_uncached(folio); if (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); @@ -1639,12 +1642,10 @@ void folio_end_writeback(struct folio *folio) * If folio is marked as uncached, then pages should be dropped when * writeback completes. Do that now. */ - if (folio_test_uncached(folio)) { - folio_lock(folio); - if (invalidate_complete_folio2(folio->mapping, folio, 0)) - folio_clear_uncached(folio); + if (folio_uncached && folio_trylock(folio)) { + if (folio->mapping) + invalidate_complete_folio2(folio->mapping, folio, 0); folio_unlock(folio); - } folio_put(folio); } @@ -4082,6 +4083,9 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) if (unlikely(status < 0)) break; + if (iocb->ki_flags & IOCB_UNCACHED) + folio_set_uncached(folio); + offset = offset_in_folio(folio, pos); if (bytes > folio_size(folio) - offset) bytes = folio_size(folio) - offset; @@ -4122,6 +4126,12 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) if (!written) return status; + if (iocb->ki_flags & IOCB_UNCACHED) { + /* kick off uncached writeback, completion will drop it */ + __filemap_fdatawrite_range(mapping, iocb->ki_pos, + iocb->ki_pos + written, + WB_SYNC_NONE); + } iocb->ki_pos += written; return written; }