From patchwork Thu Nov 14 15:25:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D0A6D68B31 for ; Thu, 14 Nov 2024 15:28:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C065F6B00A0; Thu, 14 Nov 2024 10:28:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B83976B00A1; Thu, 14 Nov 2024 10:28:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A01176B00A2; Thu, 14 Nov 2024 10:28:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7A16A6B00A0 for ; Thu, 14 Nov 2024 10:28:19 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F12FE121314 for ; Thu, 14 Nov 2024 15:28:18 +0000 (UTC) X-FDA: 82785080532.23.9654A3B Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) by imf16.hostedemail.com (Postfix) with ESMTP id 1FA0F180015 for ; Thu, 14 Nov 2024 15:27:32 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=SzevJp7D; spf=pass (imf16.hostedemail.com: domain of axboe@kernel.dk designates 209.85.161.46 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731597952; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xBThbwgpFnDYguERwRJ3Nd3OfeHrc6kNH6KxzsoIqNs=; b=zNoNHX/O25Hw+WPrFzjTZUyWth87hc8sN6zQqRpBeN7C9Sf+bqcHCrPFmyCP734bDUZjja iaKN4NaCISfet7RnyLuM1VlqAIy93tKYcihJWHUUrdyBp2en3Q8RkWzqH4dGy+XS5AuSG9 wZmrVvZlg/AlepBCOKrhOGngL5xZ/eo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731597952; a=rsa-sha256; cv=none; b=1hzen1qGQnNJzylEvQrJfb7uRrGWWZ9DtHqY0zHbKI20jY7PgEL+Qi6OEMHskNbMjw+hh+ 9wgQDWd39kAWbeA9N1RDXr5ag0nYaQ81CxvQr9NpBmb6q/A9yk1r1eHTZ7I1TEacJYqlqX lsGTfgHsib+uaPcDkaKMQ9YBU6h3jak= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=SzevJp7D; spf=pass (imf16.hostedemail.com: domain of axboe@kernel.dk designates 209.85.161.46 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-5eb70a779baso359155eaf.1 for ; Thu, 14 Nov 2024 07:28:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598096; x=1732202896; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xBThbwgpFnDYguERwRJ3Nd3OfeHrc6kNH6KxzsoIqNs=; b=SzevJp7DGBDUdWfKncudqYOWXlOL9pjl8BUKgF0C8YE1LhijIiX6JywxCKsBiIuFam buE//0deKgrl2Ph5vUq8lbX96RQHpU4AigHsvb3rT78yYLxMaioLPTj8t5tUmW72dUAn khi/VS94lHilu8zRUv3g+qpSTrpMPgdX+8ysqE+tpZ58EU6MfBftS9QUjxABMTApNxHo 69smrTTMmeh0ZxbS5T36gT91yN+RwYu7vULbUiIxUsXvW/nGn9HIvnXZFKHDfHkXKbJl YIn2IQWmtKi1Sq24v+aKvWKChxcZiJ2fnrxYGPIyy1djOaHfxw0oz1qH9hD+AtGh35nV Q3qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598096; x=1732202896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xBThbwgpFnDYguERwRJ3Nd3OfeHrc6kNH6KxzsoIqNs=; b=LL1Tzyx9p/VDqUKFZtZyTyE7ATLiPX6s1VqlkBSShY6s0BzDZiFdAhnyxvTgbZIIis TwMSjeTQLxzOC/GQkHO3hq/wH7of4nyHuvj7M+fm0X8m25NOECx0K3OnZGxMqZXPXYsW 4RwdRdf735nSm3QyK56NPdhwOlNS26VVtpKUUtuQ/aMTW9uM4caXns9DBqmjNV341JDX DjRW04KlEwN9AwzNKnYpBO/IPMG0e6DZFOcqYCMXdyAaeuKIn6RciipgDKvw0H3E5cy3 QdLTLglkcg8edPn1Zbys68jBuwYu6hnWxm4iRhfr3bJgXGr1Esy0Hu0h8+XQyRTZLFIw 5lug== X-Gm-Message-State: AOJu0YzHaiUeEfbRs1/Y41RW/uBztgkCIDzDsGJxD737KQ3FI6YYbClV 7iceGTn6u5/S8733LetC/ke2ff7BRAv9Ok0CD1iJNuUCxHgnpZDti191/xXoAUVFBIMX/eYbH6k bQoE= X-Google-Smtp-Source: AGHT+IHjZdeOSBooCNtlJS5s8uHCzSyESjo+TDYveTg1n/VGl8pzguc4+c1Edpma/7ckXWFt8H0gGQ== X-Received: by 2002:a4a:ec4c:0:b0:5e1:cd24:c19c with SMTP id 006d021491bc7-5ee868b0bf0mr9239441eaf.0.1731598095749; Thu, 14 Nov 2024 07:28:15 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:15 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 10/17] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Thu, 14 Nov 2024 08:25:14 -0700 Message-ID: <20241114152743.2381672-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1FA0F180015 X-Stat-Signature: c56iriicug7sbh6nr6zt3e738oy5hmsh X-HE-Tag: 1731598052-874731 X-HE-Meta: U2FsdGVkX1/CCEITTiJRlbo60Xo6e25K/j9GeIkYEyh2BoIlDVVpuIlBhd52v7ZqViYJyngmKbWLlR7pZ8WKRMXgsR6DNi3RcL62jlqxsMQgtRV9vZOZC9g7LjPtb457YBFRRovD5TkWTvOhV6yElo7FeE0tRenbtgYK5MTeOGeKgcpkMQbI3epDEoRW0X2QK0CF2yCpKa5Rh3P1m0kgIJuSwSbaHwJKRjiXNKUFyvOfUIEzF8dKzQHALDNibnk8emkVx0ERgukBSd3sMylQG+x+PqtshhD1J9iPtSanByYGl5ri8YhD2NPYznCPhKvegdy+j3bZ37Al1Es3u8sZk7xqwrVA77v9TaS+8xPEPyfmcY+JQN1v9zor/WUZgWo06nhWdbhjgOFN53lTtQ7lt3u7aSj22+DwETUopJiRt7gPHpB8CRTeevAxVI5yVSTDfmQYd/bsZEMz3GwI2vSwjK005q5mHFyiwJDH85J+olPIYoej3ttjYufv+jDEGoQMAkaCo9HaEcp7WgLUSN7FM/u7jrzyb6SvAuAi00d3Mfai7VWQylrqgJ7y5nEiehIIlyVfSGGqOoOT1+5zhGUAgNeQyRA7oNT9isQ3b4uApRrctM4cgU+WP6jZ2Ug52I8/2XH17UdMIKL5sUbgq36pdVp4aB95NKI66Z99E6jBVV63V1rpTJ+b6N0Xwnegm2nspKAQYYSL0PfvphAbNKLK369gFRBjRaYBMfQmpg08EVLlAOwbFND6r4kxv4FRAXuQUr+/OBwjtTJuYX9+JDAGmL7SJhYNhKVOBcZt1ZhLjDtnru1MDFS4gRZaY9BDddd2sedJoJn9z8yTTrW0YAYNvzCuW6TfbWa9NZPnknvFP3ROU5c1QL5TS4q3ct7ftaiAExGijMT9WxrHfRggKr32eh0LYSgWiNJXu6KDePITyGOtlXbc6TxnYagvnGOp4DqQON+sWpWdLlI1Ja87uo3 YS2o1zkh tb6PH0sG+ZDe4PwTZZpa20sl5u48F5mZCjP6dwzVw/v4hIxEQg8VL9XkW9DFRo4ciPTqDeM8QSKH5I72A+Qi9SF5ScCwZnC7/P4zJk4rIXhmcu12XRvWlslrlN5zPNx2GJlELMqQP+w62ZuyxzbNUQqSmyDlFa9edFobA1PjlnWO0vuGrUo1ZiV71Bf1LVxmdSAI0+XI8ZeIVi9qrFVGQGGTVWVhIp2xFacpk2VjMvIv4TrLpCJox/QvoHxs2cZ2ppMggTD+37TIfoEU410D1M7cHSY9DjDaDLWu6COkrWrRA8z2TV6yT8IVz/ZDM1UV7zRj+r5vlxfZ+kwawF1M4phWCS9PfOiO70bsv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary. Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec 2s: 132308MB/sec 3s: 132438MB/sec 4s: 116528MB/sec 5s: 103898MB/sec 6s: 108893MB/sec 7s: 99678MB/sec 8s: 106545MB/sec 9s: 106826MB/sec 10s: 101544MB/sec 11s: 111044MB/sec 12s: 124257MB/sec 13s: 116031MB/sec 14s: 114540MB/sec 15s: 115011MB/sec 16s: 115260MB/sec 17s: 116068MB/sec 18s: 116096MB/sec where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe --- include/linux/fs.h | 5 +++++ include/linux/pagemap.h | 9 +++++++++ mm/filemap.c | 12 +++++++++++- 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 45510d0b8de0..122ae821989f 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2877,6 +2877,11 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count) (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); if (ret) return ret; + } else if (iocb->ki_flags & IOCB_UNCACHED) { + struct address_space *mapping = iocb->ki_filp->f_mapping; + + filemap_fdatawrite_range_kick(mapping, iocb->ki_pos, + iocb->ki_pos + count); } return count; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d55bf995bd9e..cc02518d338d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,6 +14,7 @@ #include #include #include /* for in_interrupt() */ +#include #include struct folio_batch; @@ -70,6 +71,14 @@ static inline int filemap_write_and_wait(struct address_space *mapping) return filemap_write_and_wait_range(mapping, 0, LLONG_MAX); } +/* + * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write, + * and the ->write_begin() handler on a file system supporting FOP_UNCACHED + * must check for this and pass FGP_UNCACHED for folio creation. + */ +#define foliop_uncached ((struct folio *) 0xfee1c001) +#define foliop_is_uncached(foliop) (*(foliop) == foliop_uncached) + /** * filemap_set_wb_err - set a writeback error on an address_space * @mapping: mapping in which to set writeback error diff --git a/mm/filemap.c b/mm/filemap.c index 13815194ed8a..297cb53332ff 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -4076,7 +4076,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) ssize_t written = 0; do { - struct folio *folio; + struct folio *folio = NULL; size_t offset; /* Offset into folio */ size_t bytes; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ @@ -4104,6 +4104,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) break; } + /* + * If IOCB_UNCACHED is set here, we now the file system + * supports it. And hence it'll know to check folip for being + * set to this magic value. If so, it's an uncached write. + * Whenever ->write_begin() changes prototypes again, this + * can go away and just pass iocb or iocb flags. + */ + if (iocb->ki_flags & IOCB_UNCACHED) + folio = foliop_uncached; + status = a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0))