From patchwork Sun Nov 10 15:27:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869929 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ECE814437C for ; Sun, 10 Nov 2024 15:29:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252557; cv=none; b=aO2ZLeTjz27u4vGcNdEIcwTt8It58WAOT+GoH5GDl++juPrymOUDI/GHc7oHI7I02zemiHbSk+QZzhSxC8+JwfPjQvawDx/EqAAjs0z/mIl5F4GwjmSNvX3m4Hmk5I9YXCTykaCY4GqotVu40+Y3QCTKgTb/ZXMIcl88CeHEGnk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252557; c=relaxed/simple; bh=ZCGFBh3RRNjXY6L8R9jv8JwBpMjmEe8zEwEeTVk4ewI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UQFm1UPs0OlRUdQKiPaBsQuIPPhBK8RDXmcty62rqS3yNFEV1UJMa/XBrcM3Z2QIoVWAW23E8gpui60jTXUvHuSjhAIcx9XoJ5pPqy+82CBzxTEWd580mkqQiOTtb5p7iKZoW5OiXQVN5BD2iZ860zA3/fj2BFC6kS8GKmDnL5Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=KTl4ld6s; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="KTl4ld6s" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-720e94d36c8so4106188b3a.1 for ; Sun, 10 Nov 2024 07:29:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252554; x=1731857354; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ATQlrsiKbF4ovWTP3SQYfEdawQE8siSIL4Gn+HYptZ0=; b=KTl4ld6skRxSUjxGhT1V8ehwnsGKMPL+zhhjRER3/6dCrjKJ/md8tyGXKDg0irkY5Y XHzCen6Vqy7NTNVGL3M0RnPhdxZ9sGlA8kB3KXX8Pz+pRXdTpkfZv10ALipRUNipGHo3 rBX5fqdQLoZz6Q9KEuT9ZtOp3bY+oWqfZrKZeg1x3VGfOm+orvUFnKWuqgkft6XoNS0H R3j0lfKERtziifFyeRBvNlzL28fX6o/PwpvzJaQUCwlYsYdxtjohDLWQ7mOPxJRhVplC /BZn/wGFrBxnLh3JwzGV+87H5QVpCPphfebHRsoH16ArCTAhcWZtdGE5yfU2PDUEyiHb YH4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252554; x=1731857354; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ATQlrsiKbF4ovWTP3SQYfEdawQE8siSIL4Gn+HYptZ0=; b=VhXtU3Xb9S366wdYfwPYO+QD50t+c+JtYdMtWMHQLATGXufY7bH80zhbq59HylayY5 En15AVw7E8XL+yjT2bqK0HxRZdSqfjzfTOoME4TUzCCCVTMjXSM9V1GrF5ZFWPnHz671 ErFrBnd7kODXlG4zbLbnS4HMEjddT5d+fUexYyLKVzhzRutT4gdNCeGXOyPsnPot+GtV /+g+biav5sg+RB8KOLBEz3IsBax8fIyfA1dDfmqF+LAZVHhD2hWa1LlnV36rv3pUbZy0 hdJMvn65aR+N4N3V8AQq9wneffsoym/wycHyOfgrQUSxMQuYQW63x9myHTfvwPjiBzYP v0QA== X-Forwarded-Encrypted: i=1; AJvYcCVYcUfHKLAcQFa7DKgLuffw2sBb4Vmuyq+B5Rs41rJRMoV+uvE4mVaQ68ZHuaCAvkLsdJAy7o4Ft7iWLAyo@vger.kernel.org X-Gm-Message-State: AOJu0Yy3g6Z8Kg7o2AhNCptAgpTyBOPxvEmv8RfKfq8Uk5VypOQnZwDx Ltwz8oxCYtZh/xpKaenu+03X00b8E3qoSH7y3I8oVch/Lmf7MR2K9IMo7+E8VQw= X-Google-Smtp-Source: AGHT+IHudq5kHes2nSMTtH3m+uxQGkBDk1+1Ad9rVBdouFER/SNEbqW3PARIVnuSTIfSKRPsHGRgYA== X-Received: by 2002:a17:90b:388d:b0:2e2:b94c:d6a2 with SMTP id 98e67ed59e1d1-2e9b1e13166mr14624611a91.0.1731252554158; Sun, 10 Nov 2024 07:29:14 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:13 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 01/15] mm/filemap: change filemap_create_folio() to take a struct kiocb Date: Sun, 10 Nov 2024 08:27:53 -0700 Message-ID: <20241110152906.1747545-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than pass in both the file and position directly from the kiocb, just take a struct kiocb instead. While doing so, move the ki_flags checking into filemap_create_folio() as well. In preparation for actually needing the kiocb in the function. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 36d22968be9a..0b187938b999 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2460,15 +2460,17 @@ static int filemap_update_page(struct kiocb *iocb, return error; } -static int filemap_create_folio(struct file *file, - struct address_space *mapping, loff_t pos, - struct folio_batch *fbatch) +static int filemap_create_folio(struct kiocb *iocb, + struct address_space *mapping, struct folio_batch *fbatch) { struct folio *folio; int error; unsigned int min_order = mapping_min_folio_order(mapping); pgoff_t index; + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) + return -EAGAIN; + folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; @@ -2487,7 +2489,7 @@ static int filemap_create_folio(struct file *file, * well to keep locking rules simple. */ filemap_invalidate_lock_shared(mapping); - index = (pos >> (PAGE_SHIFT + min_order)) << min_order; + index = (iocb->ki_pos >> (PAGE_SHIFT + min_order)) << min_order; error = filemap_add_folio(mapping, folio, index, mapping_gfp_constraint(mapping, GFP_KERNEL)); if (error == -EEXIST) @@ -2495,7 +2497,8 @@ static int filemap_create_folio(struct file *file, if (error) goto error; - error = filemap_read_folio(file, mapping->a_ops->read_folio, folio); + error = filemap_read_folio(iocb->ki_filp, mapping->a_ops->read_folio, + folio); if (error) goto error; @@ -2551,9 +2554,7 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); } if (!folio_batch_count(fbatch)) { - if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) - return -EAGAIN; - err = filemap_create_folio(filp, mapping, iocb->ki_pos, fbatch); + err = filemap_create_folio(iocb, mapping, fbatch); if (err == AOP_TRUNCATED_PAGE) goto retry; return err; From patchwork Sun Nov 10 15:27:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869930 Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7421C4F21D for ; Sun, 10 Nov 2024 15:29:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252557; cv=none; b=RDUUvHFi2RWTCXHvuiCwr+5oghXrTYZAxjfGXdRt6cOP0xQnFwZDJUDm1mD4EqmefhARn2eL4hJk04hmqhc8wINMaCgndEsMlBOD7tp8c86DAJwCNfOmEs80QEzAe3Tm3Rn8j/Wbe4OBVSSJ9bTteI4T3G5+1qnfKNQJ/gyhrP0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252557; c=relaxed/simple; bh=wdE7j3SXsvf8mZ3WLeO9bDbqT1xBl9uoI5Ta1MifPnA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rXTeVAiR/lhdCDJ4+wqfI+Wu2t1bq6PkMcB9cIpluZx8upb5uZNXgb6LbSQbkJEvdf5vyy+z86xOJkuT9g8Dgnp1LNKcI9ThwHRDUyNFDI0OgY9Etb6KrnhgrkUrOSQksHjzVDIRx20OVCex0KvMHBXVO97LnJnui8wz+nBRHjE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=uCQqbBCW; arc=none smtp.client-ip=209.85.215.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="uCQqbBCW" Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-7ea76a12c32so2769145a12.1 for ; Sun, 10 Nov 2024 07:29:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252556; x=1731857356; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G7YIma1TSdS4GVKncfhre22V8DGIHalzmKUoqzgwI0E=; b=uCQqbBCWKXwXYqwtTldcjEciA9O/jswHvB8sLHez+k1MKrdrexbKmmLlB/2xSJNehQ zuenZR0qWPmvPing0Yi5k+ZkBLHdMbPMqOqZJQnQiVwt9tVjJhPoykkq92T+mJD7mQoa lni6bK8H6oHmdCTtAeO2TEYmshXmLFggiMg0gaFKq+ZH17jiPFxRK8ssBeviSxnhO72r lI7jc0oBbhqdTX8v8xLwtecgCnTQw//ddS5raCd0WoJiLtKDMxyfa5Lb5SlI3KVhG+b5 l3W4+EASOfHXCGRQqQM4MFMb6MuaBaR/gwe8C6CDdjetfO2m8GEvGkYZKTcWmA5kYsiw 4Bgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252556; x=1731857356; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G7YIma1TSdS4GVKncfhre22V8DGIHalzmKUoqzgwI0E=; b=oo2N7M1jhuuSD+Z7/hXRglTRb/J8TEJGooJqhEsJ9zOUZYCdeXdLKxSB897/03jTxQ KXWNAt2xAGEzM6vlcT120jXl+p6Qz0MkFpudShCIJ6PcG2Ev62MHgY5UUvtT56M5eFpa gk47X9sn8H+ZhGLrvtBG6OKI0ykx78ykQnld4lfvUmlTc2oH2GLSVSywEXqxV2IcMJVB D904kC5ZYG2+pK03Wq5/65ZNq7P4vk+m+6x6leNb2Ip9SetN5q3+9ie9CXz8eVuLBgA/ kfIfEZKAwA8vS4E2iht/XaU49C0CZ8M2DfiG2hbKLLEwFT0WNV8TYpa+tIMmVhaZaa2P 3WCw== X-Forwarded-Encrypted: i=1; AJvYcCUWXwFqMMZWh3s3/S2d+JuNt1yFlcUTp9S1AysRyRmaTjZ8XFCxIxu0tUZ8pOv1YwbvroM9W7nEN2ui1B2j@vger.kernel.org X-Gm-Message-State: AOJu0YyN0UUWSHazAhUj19CuCPOuH1EIFtcSW54ZlKIE2L9MtWbYQNUn A1pG3alsOr4rBZLBlCU8B4aEveTlLctEmG+wpIMMFhW/5sF8EHvsdmMyuE+VUZA= X-Google-Smtp-Source: AGHT+IGMsIWblwQtQWO8JcRHCLRTN0pIGof64EqqksTz4TFFvZ2UD/wQcBuuSf4CK9eKe+atZiR/kw== X-Received: by 2002:a17:90b:4acb:b0:2e2:be64:488f with SMTP id 98e67ed59e1d1-2e9b1655945mr13617134a91.6.1731252555726; Sun, 10 Nov 2024 07:29:15 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:14 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 02/15] mm/readahead: add folio allocation helper Date: Sun, 10 Nov 2024 08:27:54 -0700 Message-ID: <20241110152906.1747545-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just a wrapper around filemap_alloc_folio() for now, but add it in preparation for modifying the folio based on the 'ractl' being passed in. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/readahead.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index 3dc6c7a128dd..003cfe79880d 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -188,6 +188,12 @@ static void read_pages(struct readahead_control *rac) BUG_ON(readahead_count(rac)); } +static struct folio *ractl_alloc_folio(struct readahead_control *ractl, + gfp_t gfp_mask, unsigned int order) +{ + return filemap_alloc_folio(gfp_mask, order); +} + /** * page_cache_ra_unbounded - Start unchecked readahead. * @ractl: Readahead control. @@ -260,8 +266,8 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, continue; } - folio = filemap_alloc_folio(gfp_mask, - mapping_min_folio_order(mapping)); + folio = ractl_alloc_folio(ractl, gfp_mask, + mapping_min_folio_order(mapping)); if (!folio) break; @@ -431,7 +437,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, pgoff_t mark, unsigned int order, gfp_t gfp) { int err; - struct folio *folio = filemap_alloc_folio(gfp, order); + struct folio *folio = ractl_alloc_folio(ractl, gfp, order); if (!folio) return -ENOMEM; @@ -753,7 +759,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; @@ -782,7 +788,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; From patchwork Sun Nov 10 15:27:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869931 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF360132132 for ; Sun, 10 Nov 2024 15:29:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252561; cv=none; b=XCa44vFwtibJXMWVlle6rO625klce8kIgf9XLeMDo9xREK+kIUdQtg08c62wlS+F1vDu0jTZiICaV5Yjax8TDm0WvHyArOp3g2QCvhDGopCKHElkhtRCWsPLR7I5wlnEsXVyjDy5U/Ie9XG09+p8Wujodi4rVDQcV91jDstuzJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252561; c=relaxed/simple; bh=klEANlDo0QcivFe1NIYDNiOFa1lRCa2wf9JjLv/2My8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cinqsgihByUTsl0r+slb4YjFXMuZZWRZY+TEL7dLvL5ImK8ZrwFwr0x4eVvDDMjw0xTl/y89/zyv/U1BRPX59yHuE+MlmlFXq7GuhFzpy6Z+SMASFx0MMRhdInfMwNPSazsWTHu/Ral790r+jYrf8Dis4LhVf4M40mWeJBX3S28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Ouq80bgN; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Ouq80bgN" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2e91403950dso2822704a91.3 for ; Sun, 10 Nov 2024 07:29:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252558; x=1731857358; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9QDQ8u+XOjRX6rdX1KxWEpFxXqQ9HPCp5iTXnKjwIis=; b=Ouq80bgN+n62Hh9S9w/3uvP7bBk73EYoisR8aw+jBueVfm+r4+aWtgK9tEB+wPWIJ0 KTgwvudsL7/hDAnzDN948WPH195zh88m4Pv1TNRzon2pLz9Ea+JNHHKBC196Gu1FtZR1 /F1rRfYGoAHkKOjjx9KaWXSu6SINBA4LeRoPcUNL+7VzvudwSQNrvBCjxb4QxNTgDpqU WgQyQCEfKf05RFRBvSjs9tDDP6fcXnbrWYTxSkuKKlkKL10oAjxG4TK1Bk5mzgYxM2w1 0etMBm3oPlVdz9oNcJUtYzrxfMLHhmEHEE7pjqMr3w5mZ+5lJH0KAUKOtQ7VN5A3n+pZ 6XIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252558; x=1731857358; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9QDQ8u+XOjRX6rdX1KxWEpFxXqQ9HPCp5iTXnKjwIis=; b=ifQvzZR/edEhAuBZXWGwTvnXlX95cMi+uMQMq6eBMhPZBrbD7ZzlX4241R+K6ZfHOs ZFdnsETZ3qKfVuQliXtcm7UpFPdPOA7/DXW8De360+tNkoMdp0f8kkDcmuAN5l0YuF2J WW8MkTKCz2+j1YMXqz8O7a37Ey0vuARYEHeL4xm7susyF0ORw7xVBNuWCs5z4KlUCPuV zrJzeEzY7tH6cLoLzGZMa8sh9qP2GAll7e/e/ChnQXSAohynZB/7cQdB7eELhq8edjmN UeO0KeJH/nvyw7r7qa2gvhMJVLImcIsUB4cJOCrnp/FhAnRLv2dibfvltGN/Rt4d3uZs UN5g== X-Forwarded-Encrypted: i=1; AJvYcCXfgQQIU0RpoNjpAeob03TZAz4d9wplq39gSDBalDdnKdQq4VE/GWj/+wCmUgfVza86FpG1+vZxKpL3dYsO@vger.kernel.org X-Gm-Message-State: AOJu0Yydj4SRWIttJEPbEtiOHFartSEVOzKRG6/3sC8AXh8hv3VLlr/z 7P9q2l7OZe9ZN4tF8uafkcPR6uQ4y+YxMRfCIxiK5AVUFOJEMDjYCem1XRO6CnU= X-Google-Smtp-Source: AGHT+IEK+RdpBHKlpp5v8yHzxXuRsIFP9YbjrZgEGVdARDnq+2H6kYv6T3B7kfQSRNLPJFM3Uvu8sw== X-Received: by 2002:a17:90b:35cf:b0:2e9:48d0:3b59 with SMTP id 98e67ed59e1d1-2e9b16eb007mr12818842a91.8.1731252557781; Sun, 10 Nov 2024 07:29:17 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:16 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 03/15] mm: add PG_uncached page flag Date: Sun, 10 Nov 2024 08:27:55 -0700 Message-ID: <20241110152906.1747545-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed. Signed-off-by: Jens Axboe --- include/linux/page-flags.h | 5 +++++ include/trace/events/mmflags.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index cc839e4365c1..3c4003495929 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -110,6 +110,7 @@ enum pageflags { PG_reclaim, /* To be reclaimed asap */ PG_swapbacked, /* Page is backed by RAM/swap */ PG_unevictable, /* Page is "unevictable" */ + PG_uncached, /* uncached read/write IO */ #ifdef CONFIG_MMU PG_mlocked, /* Page is vma mlocked */ #endif @@ -562,6 +563,10 @@ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) FOLIO_FLAG(readahead, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(readahead, FOLIO_HEAD_PAGE) +FOLIO_FLAG(uncached, FOLIO_HEAD_PAGE) + FOLIO_TEST_CLEAR_FLAG(uncached, FOLIO_HEAD_PAGE) + __FOLIO_SET_FLAG(uncached, FOLIO_HEAD_PAGE) + #ifdef CONFIG_HIGHMEM /* * Must use a macro here due to header dependency issues. page_zone() is not diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..b60057284102 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -116,7 +116,8 @@ DEF_PAGEFLAG_NAME(head), \ DEF_PAGEFLAG_NAME(reclaim), \ DEF_PAGEFLAG_NAME(swapbacked), \ - DEF_PAGEFLAG_NAME(unevictable) \ + DEF_PAGEFLAG_NAME(unevictable), \ + DEF_PAGEFLAG_NAME(uncached) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ From patchwork Sun Nov 10 15:27:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869932 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF81C132139 for ; Sun, 10 Nov 2024 15:29:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252562; cv=none; b=hGVuGXjipt5ge/X2Ze55EqU3ufbUfWqw59s246HtgxYOVqvcgXoog4ztP998B0DIGGlDhjQuIl94kMPVaklU1K9rxXTJ6Zzc6fcQjGB3xhcVfHL53Ijuh7WXqR+8enpH60qLQ48xJeAQsnPCJ/H7G63Vzg0VFkUALxgT4z5f9uA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252562; c=relaxed/simple; bh=AFLrZl3fH+/hMhwZTjgZ0OWM0ndgzDi220n1zUDQffQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HBAh30OH2l+fiWXiTk83FMK+TKGQgv0/xz46TVKUZfMJny5lCPodkz/KWVZZaYz2BQs0tvYNBM9Bp0tct5OxJvAzk9WVxi8RO/dUOHedGX3D7WMCKrcv0iFx+Da8EkjA/0SyZZf4de3TsKVgT/Pz21A2ayDtcNBsOPlmnXzC9aw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ARRTGlBZ; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ARRTGlBZ" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2e2eba31d3aso2824179a91.2 for ; Sun, 10 Nov 2024 07:29:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252559; x=1731857359; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jvfAlDie45/dO/q2Oa4nMUlSYRbUQddnQckYeJvFoiA=; b=ARRTGlBZxa5PHUsy9Q2tvu9pwq5hBq6iX1QXmL76/dMi8cd4bCAVhakos3YMhFgzxM 8L0F5BfA/KXHRhfZ+5fXXW3wO+YH7oZiJbghZtvvdOF1Zg0dWlXnAQOckFK6uHIOrq3z 3NMgg2Uk3HgKDmBEeCB/W4yVCBuF5a4ovwOgYDiEaoqon8TL8jt9CPezX4OQe6P6/tzM ikqmm5eCnG3bBlPI7w9H0f6sQLZGCByWLUs7u9Oyw2wdxEPovJEXROlrYoUZnbU5Fdqa nQXK9H8GE4ofJDQVp3nVL/DVlViWOayXNDNydyF2FEB4EBCB1nAUJ48JW2/o7aE3jkDA OFSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252559; x=1731857359; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jvfAlDie45/dO/q2Oa4nMUlSYRbUQddnQckYeJvFoiA=; b=NY6WbXU8SQuEblIpVqs2DU73IC4huhtGGWSq4Nr47kYIUcF1bMuaH1d5Eq1Of2Z99T uBSaePQTwzyEpCec5ESciD9czIE+Hvquiy97Q/ZEqwwFV8e8Vx397K2Kmoi9rsUshJiX pdiJSU3n+nwlIvDlrsWDGh++jhAZdU6+PTfgQSJXSZbw/aV6y6b4Zd2eN8s/CUcl0M9b BHj/u3Hvs9lnmvZGfmpWoVMhWRvIQH8OdLlaXAW2hgYVjQN5tYPMA23XoCz42Fkf9L0z i1s1NoPkrRTXtutX7wLWaMO9+jTamErmVJo3vjeeGsLqEe+YXB5hD5S+eYgaPNbukrgf xJtg== X-Forwarded-Encrypted: i=1; AJvYcCVuTA3R0viYhYP3b0Xfo9P8vnbHec7aw22q91YPjH6a2XCtzlI/4LyoaDY7cZ2GDx0RlfmuqVzyOWF+pkHK@vger.kernel.org X-Gm-Message-State: AOJu0YxuwosVx7o3RE2rs+zjY7klUpQb0RpWhWm158KSq/D2xI4pC3tj i2jiySLsLFKJjdkUi8Utr6xMu4l5VVWz/cs1XxhQWgI63MUzrwijKNG/sB1kZAM= X-Google-Smtp-Source: AGHT+IGu/u+FhGsz3wBr1Zsic2L7vJ+0uwzRrgnEJRvL3npe1QZdpxBowle6ldDc43UF6F8F8hYjFQ== X-Received: by 2002:a17:90b:2ecb:b0:2e5:5ab5:ba52 with SMTP id 98e67ed59e1d1-2e9b173f1ddmr13558707a91.20.1731252559257; Sun, 10 Nov 2024 07:29:19 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:18 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 04/15] mm/readahead: add readahead_control->uncached member Date: Sun, 10 Nov 2024 08:27:56 -0700 Message-ID: <20241110152906.1747545-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If ractl->uncached is set to true, then folios created are marked as uncached as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 1 + mm/readahead.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 68a5f1ff3301..8afacb7520d4 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1350,6 +1350,7 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool uncached; bool _workingset; unsigned long _pflags; }; diff --git a/mm/readahead.c b/mm/readahead.c index 003cfe79880d..8dbeab9bc1f0 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -191,7 +191,13 @@ static void read_pages(struct readahead_control *rac) static struct folio *ractl_alloc_folio(struct readahead_control *ractl, gfp_t gfp_mask, unsigned int order) { - return filemap_alloc_folio(gfp_mask, order); + struct folio *folio; + + folio = filemap_alloc_folio(gfp_mask, order); + if (folio && ractl->uncached) + __folio_set_uncached(folio); + + return folio; } /** From patchwork Sun Nov 10 15:27:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869933 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 731E0139579 for ; Sun, 10 Nov 2024 15:29:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252562; cv=none; b=sfhne75USlmyfsGSEArQn5eHGiXEFcDt3pc+im2MJwdyo3V4/lJui/IEDoFwp0q1FJeopRkPCqUp/Fta9cFZVHieOvjP+vn3Y5wbHOn/Ym6ppz9xbUS4UOJI1tR4cFHL8e0NzK9XrzcNw1TiR2PAkNwXMw+/mD82DkHhrLF9Km8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252562; c=relaxed/simple; bh=FAuL9fA2byaQyHKz3osLBcHrpXlsfUGCQq+TdYUSPV4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gxQSxlbbo44Ile806WCvZDyghMJfPfJualYo1VsIXgIYcOAmistEikXJZHDagopgGFHgqtn2NHdUvU1Kc8+W9fEghn43Frdgfccz8yCavdscSpUvH9maDMFso0hnlCnmspiw+ZIU7uGmK7wM6DpHgzuBgGexG5pC//hcrFmYmLI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=qmYWEuXR; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="qmYWEuXR" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-20c803787abso30229965ad.0 for ; Sun, 10 Nov 2024 07:29:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252560; x=1731857360; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uN7tjL2s/tmxM6CJqz4ta5HuqLPCPNJgi5Uvj68apZw=; b=qmYWEuXR0dF4ao1qm23FHBkC1KFFPpXHCNxeCrVBSTu51G7sJ5NGRSHFiR/HEauAil 1bKE3AArI9plvFqdKod2vE1CxWBDCIPEvDdX8U3IE6Gmpu8W726qKIzA2pAqf75aHTw0 jNvaw6j86lAd8mX/79Ko2mKpI/5phTJOF7cGf+OA/QgV8fCts5OTxY7Ij3SYr3YQocl7 yPhrMab/vSExHReUDUim/U3rWGDLxsxkQe2y6fpSqhCE+A4Kd2ksgIayFprt4TuRAMJn XZsVkotq+HbOlVYdMp721vkbPTcIAWNiF1Bh5o8yvOOcoriMu5eu4xtCMNtFkWxUhsie d6NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252560; x=1731857360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uN7tjL2s/tmxM6CJqz4ta5HuqLPCPNJgi5Uvj68apZw=; b=m9LNQpIWIyEnAv5p8uhr1h+nJpFkVyD0Gn7Hf9BnIoUmBZrVjaHKDtBhgi1ETiOVPR dMCx2rxDLkXEYZQ2aB8evIVZHCMSw3pawYmbXzpLLljctFwO1BLvW/bTrOT68ZGCNTmQ yzo/c4V9UkKI0O9zCtZAcXh6qQbwM184WwCoOKA+rMTwDZFFehaiJSqSlDgjAQNWBqQL 3HAizXbKHx/8nSd830l4fnhk1ta77oP2MczGaW/Ibd7ygut4H8dt8BUnQYOB25kJwyTz NbEb6QmRxvq5OHqY07oOTXT0gI5PAt23wGvBHI6fKE2Tcyb1X+qGRLCjPasQQoqJ2jME ZHyg== X-Forwarded-Encrypted: i=1; AJvYcCWPCAp2KdStbQ8FQqQwVtMSL5d5zluPqg0Hp0BHwoNtV8GtZ2+Y3vXCFNvXbvqUiGRMNiNH/6qxRXscFc9z@vger.kernel.org X-Gm-Message-State: AOJu0Yw8Pyd0UbxQttSWJ4teyMQCJ27iIzgFX3m9zHO0AXIz7KmdPuGW vGI33uGGuBszGiR4CsLsLcWbuzuiEzDcBpGG/Cnrhs81JKykrFuV+9AKnS7wkR4= X-Google-Smtp-Source: AGHT+IEPVPrIsvzVKM9BRnKywqUBY+49ysTGl3h3gnVcSq9byO4UW1dwGcp/l7t7WWnOfvV7vMqjdA== X-Received: by 2002:a17:902:e812:b0:20c:f6c5:7f6c with SMTP id d9443c01a7336-211821c4546mr156324045ad.16.1731252560665; Sun, 10 Nov 2024 07:29:20 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:19 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 05/15] mm/filemap: use page_cache_sync_ra() to kick off read-ahead Date: Sun, 10 Nov 2024 08:27:57 -0700 Message-ID: <20241110152906.1747545-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than use the page_cache_sync_readahead() helper, define our own ractl and use page_cache_sync_ra() directly. In preparation for needing to modify ractl inside filemap_get_pages(). No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0b187938b999..38dc94b761b7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2528,7 +2528,6 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; - struct file_ra_state *ra = &filp->f_ra; pgoff_t index = iocb->ki_pos >> PAGE_SHIFT; pgoff_t last_index; struct folio *folio; @@ -2543,12 +2542,13 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); if (!folio_batch_count(fbatch)) { + DEFINE_READAHEAD(ractl, filp, &filp->f_ra, mapping, index); + if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); - page_cache_sync_readahead(mapping, ra, filp, index, - last_index - index); + page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); filemap_get_read_batch(mapping, index, last_index - 1, fbatch); From patchwork Sun Nov 10 15:27:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869934 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A7BB13C8F3 for ; Sun, 10 Nov 2024 15:29:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252564; cv=none; b=l4LIOYVew3cXPqhVyqeESBbRj0aB4V7Gbvz4J1i6973Y8jwCDqwZH/+2wBpWuUNCgUTwlnW4rh4ns34NgM6hbEMny9zvn6hMzwj3pC5bnOKhL3+FidVEA15TrTuPP4LJt0avtgB1Lva6tNoLbWAbJQ0Fzx0UBiTcqGRQ26aY6SE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252564; c=relaxed/simple; bh=aEnrAJGEshe44zlb6XbbKSp/FXpeH0RZvF9K0y5SLj4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dzjN/hL0ozaLM/hUPLD4mRgNNVvOQ49CMSmyjg8VpribhbMJp45ZACH+sSA+KxiGn/pqeKp4UJEaTcytqwpt83M6j2yjXxUTGDr1stypszp19ZCHrmDa7HlxN3mjfbHp97gknslIBo8KWG54FNCaKSi4/YQd7pRZDAJqzDeHV2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ZfYZoDf+; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ZfYZoDf+" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-7f3da2c2cb5so2733162a12.2 for ; Sun, 10 Nov 2024 07:29:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252562; x=1731857362; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SJBisbj/rGvidSFxNEf05dxDvPbQrKvqbgVXmZ2uTrI=; b=ZfYZoDf+c7eHi5RDHnl2c3+aGOxYMh/suv0VAX/EdlySskaQd/kZLeqqhvnLCvEofX c0YWuNqg/JguAUt09caneDLVxGTZujlvsqfOu/vK+01pgqFvp4YrHOvooGMDCfSXp0nQ tUuhFDrkr5BRwMPIYZwVD2GtMn/g3yTZCOCK/N6zHI+7rBaps2s7kqqm+HjbsWW4WJI+ c+eedGZy5PrI+Kk3WzBak8rf300HSTYVdPfuZvNPmRvM/pklP8IqE/CIWJGEjjYb7qBU yrXnlnuNudylAeDbLPKeI3pn3R47UgLsQc5WTyy7srFK+oQCrEOlL7Tos+6gWGX4vlw/ woQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252562; x=1731857362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SJBisbj/rGvidSFxNEf05dxDvPbQrKvqbgVXmZ2uTrI=; b=V35WD4Jwm/4FXvalIlbgRQ1PGjrcRXdZntjhY960EtoGavcbnIt0QR+QgYY3BbcvIi LvPMHxr1sMs9gurjyRYh2BmyeJRgA/i4JoCln8/Fzs1kng3+BX7yLZIWtJDcoaxGItIt anewVgdU2sXAtKi/0/KPbum83Ocaj0QdbK6fI4F9JJzuXLdMPgbw8fUhckP1327THwG4 Vn5vXPYeGY00h7xBOuqGcUqSY53iXLHPO7wzFuJYy4uZOyrLQH/5zrLNyDD+BA+YLEsg 72kBsYK6TytlZSUpChVsmVvdf4f2upqM87G1ROHCspJyq1R7HLCs287XAI/K+H6lplIx 4Ucg== X-Forwarded-Encrypted: i=1; AJvYcCXK1TSP1uThIBXv9sCvGQZbe77Dvfa17x7B+LJEQmva4pXvWpPAoAkgBWHqNDJhiVR6d3cWyPf5MsN57Mdp@vger.kernel.org X-Gm-Message-State: AOJu0Yzq2OeGuO9XI4UYqGyiZZ9NVTiOIuhF/llw5fgrNjISx0yT9MFT vWHRKsfPRsyd8txajFF70/R3Cxa0EWW+Q7zBPu4KUfMqSZHDZDWQoSDjDL2g1s56m3S8m0rqUyw tSyc= X-Google-Smtp-Source: AGHT+IEztLXy7pfG6RYnicaB5W8MU9mXcvU3VePtiJO2A6vGFaDJWmS6lrttgNl/g5B6W7Nsuo57hw== X-Received: by 2002:a17:90b:224f:b0:2e2:da6e:8807 with SMTP id 98e67ed59e1d1-2e9b177fc52mr13814904a91.26.1731252562169; Sun, 10 Nov 2024 07:29:22 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:21 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 06/15] mm/truncate: make invalidate_complete_folio2() public Date: Sun, 10 Nov 2024 08:27:58 -0700 Message-ID: <20241110152906.1747545-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Make invalidate_complete_folio2() be publicly available, and have it take a gfp_t mask as well rather than hardcode GFP_KERNEL. The only caller just passes in GFP_KERNEL, no functional changes in this patch. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/truncate.c | 9 +++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8afacb7520d4..0122b3fbe2ac 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -34,6 +34,8 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); int filemap_invalidate_pages(struct address_space *mapping, loff_t pos, loff_t end, bool nowait); +int invalidate_complete_folio2(struct address_space *mapping, + struct folio *folio, gfp_t gfp_mask); int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/truncate.c b/mm/truncate.c index 0668cd340a46..e084f7aa9370 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -546,13 +546,13 @@ EXPORT_SYMBOL(invalidate_mapping_pages); * shrink_folio_list() has a temp ref on them, or because they're transiently * sitting in the folio_add_lru() caches. */ -static int invalidate_complete_folio2(struct address_space *mapping, - struct folio *folio) +int invalidate_complete_folio2(struct address_space *mapping, + struct folio *folio, gfp_t gfp_mask) { if (folio->mapping != mapping) return 0; - if (!filemap_release_folio(folio, GFP_KERNEL)) + if (!filemap_release_folio(folio, gfp_mask)) return 0; spin_lock(&mapping->host->i_lock); @@ -650,7 +650,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, ret2 = folio_launder(mapping, folio); if (ret2 == 0) { - if (!invalidate_complete_folio2(mapping, folio)) + if (!invalidate_complete_folio2(mapping, folio, + GFP_KERNEL)) ret2 = -EBUSY; } if (ret2 < 0) From patchwork Sun Nov 10 15:27:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869935 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CAEC12F375 for ; Sun, 10 Nov 2024 15:29:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252565; cv=none; b=ru6/groeQ1xrm141Pa/79ON+/CiN56PftJ8Xtl4jtaZfw6GRFFiItYGjuOXsAVWxp1epcQB4DeZb/b5FRs8NWmYPIi0hk8apyrA63jzzVH6UZpio/3SlY0Vy+GAjVbU8a53ZNQdzZriKPbZ+PbAOAIk3yAESUT/1KHCLVFWcE1Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252565; c=relaxed/simple; bh=QqsnT+PFB0stp1XOLxW6GR03t+Q+Rw9k/OtfDS+hMd8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gjc57iwA7sOyQDU4AugqjIBp8NxLKms9JE6P4Vh4VslYPorfY9TKa6lZHVjMu8IFmzfM4F4vTxyOk43N6u/hoBGeDFzAsy0Y5S6mJHivX5aIdaSRs718mGEH9tIPKt5gFgGpQsNxFIsSUGhrGxpjXCLVG2mT9yb/OJiAjW77ikA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ITqFiO9j; arc=none smtp.client-ip=209.85.215.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ITqFiO9j" Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-7f3da2c2cb5so2733165a12.2 for ; Sun, 10 Nov 2024 07:29:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252564; x=1731857364; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BpaeGG8sCxZf4JRh/p7NGjZl2gbHvBRPjf0TUvSvRrE=; b=ITqFiO9jdn5hfePmhO3CDfJ4/Dmi47iutuGrtDiahBPQhv0vuNt8DvJBOLO3Vxbm3j csJv0r53dA3Iw5r4Oyz+o8HNT+AnndUKoaQQv8Plso7XHK9X5ztaaObQL/PX9x5+v3AK svwxzNw08U3xQl9ftcmwmXj8N0SH1rhpCyajbwU9uOHps8jGHylvxgWT6cEhoORiWMSc DfWarKen3KehFQL5V0oQ4lXV4hOKfHYjbrxiU1wH/GwCNC9ZtZ/ylBhYS2sSgy4LlVIA 7iYkVAJkiGiVf6foE4lxi3B77XWykVOy6Ww0w2X40CY/7+KkhxAXQ/vRX8B39Zjr/M4y by3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252564; x=1731857364; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BpaeGG8sCxZf4JRh/p7NGjZl2gbHvBRPjf0TUvSvRrE=; b=N04fRwybkVOTDUAOl42XDoayXNEG51sjKXvZvV4a5Imk3/xn+8usDTLsulEFx22YS8 v/xVDw2D0AFDE8O1gvBgrKtpPCMSRBse9A5jJ9IPEMhncPb/gQEKVhKQkeyjebe0wzUV zTNPxqbyRjy8TJraSR9YhmEsQ/FQ3KW2Gk4+g+0xAu4QfAwJFtxDgLu3HHPBrFewKhls /TSnfCAbFyM86uFxbItizmCvyrq7oiq5FYsJByE38oZ+IZ2bu3nLhi/ML7U8cHGUtde1 66pbpjuGfBII3aBEGv19sv2efd+ei5p0O16W/jcnygMx6bJccyTTGhQANGCyXqvOBlea PVlw== X-Forwarded-Encrypted: i=1; AJvYcCUUYwAVuwOq/Ud+Xfj7vXNGDTZxmas2FIZgnsfFKs+PTwUpQEATBrUzZiqrE/2h48KHRbnOyIqEF5KGcc+w@vger.kernel.org X-Gm-Message-State: AOJu0YyBAnA4nQtPVeSEgbioBwEVjWw5fmhjslhiGimwhL+XmAISuiZU 0Z9YvkkfRKSqZ4/mc8+UdXNSxLo6VLdhr0FJN2tjnUVVF+njMLyuKx7QA4FdDLY= X-Google-Smtp-Source: AGHT+IGs3z/1IuRBhhgcQrQmIVWwPr43n/jctzILt+apLswhddJAGBym568yRqijFSP667Ww7gbDZA== X-Received: by 2002:a17:90b:1f8e:b0:2e0:d957:1b9d with SMTP id 98e67ed59e1d1-2e9b17163cbmr13877494a91.13.1731252563753; Sun, 10 Nov 2024 07:29:23 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:23 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 07/15] fs: add RWF_UNCACHED iocb and FOP_UNCACHED file_operations flag Date: Sun, 10 Nov 2024 08:27:59 -0700 Message-ID: <20241110152906.1747545-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If a file system supports uncached buffered IO, it may set FOP_UNCACHED and enable RWF_UNCACHED. If RWF_UNCACHED is attempted without the file system supporting it, it'll get errored with -EOPNOTSUPP. Signed-off-by: Jens Axboe --- include/linux/fs.h | 10 +++++++++- include/uapi/linux/fs.h | 6 +++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 3559446279c1..5abc53991cd0 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -320,6 +320,7 @@ struct readahead_control; #define IOCB_NOWAIT (__force int) RWF_NOWAIT #define IOCB_APPEND (__force int) RWF_APPEND #define IOCB_ATOMIC (__force int) RWF_ATOMIC +#define IOCB_UNCACHED (__force int) RWF_UNCACHED /* non-RWF related bits - start at 16 */ #define IOCB_EVENTFD (1 << 16) @@ -354,7 +355,8 @@ struct readahead_control; { IOCB_SYNC, "SYNC" }, \ { IOCB_NOWAIT, "NOWAIT" }, \ { IOCB_APPEND, "APPEND" }, \ - { IOCB_ATOMIC, "ATOMIC"}, \ + { IOCB_ATOMIC, "ATOMIC" }, \ + { IOCB_UNCACHED, "UNCACHED" }, \ { IOCB_EVENTFD, "EVENTFD"}, \ { IOCB_DIRECT, "DIRECT" }, \ { IOCB_WRITE, "WRITE" }, \ @@ -2116,6 +2118,8 @@ struct file_operations { #define FOP_HUGE_PAGES ((__force fop_flags_t)(1 << 4)) /* Treat loff_t as unsigned (e.g., /dev/mem) */ #define FOP_UNSIGNED_OFFSET ((__force fop_flags_t)(1 << 5)) +/* File system supports uncached read/write buffered IO */ +#define FOP_UNCACHED ((__force fop_flags_t)(1 << 6)) /* Wrap a directory iterator that needs exclusive inode access */ int wrap_directory_iterator(struct file *, struct dir_context *, @@ -3532,6 +3536,10 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, if (!(ki->ki_filp->f_mode & FMODE_CAN_ATOMIC_WRITE)) return -EOPNOTSUPP; } + if (flags & RWF_UNCACHED) { + if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED)) + return -EOPNOTSUPP; + } kiocb_flags |= (__force int) (flags & RWF_SUPPORTED); if (flags & RWF_SYNC) kiocb_flags |= IOCB_DSYNC; diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..dc77cd8ae1a3 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -332,9 +332,13 @@ typedef int __bitwise __kernel_rwf_t; /* Atomic Write */ #define RWF_ATOMIC ((__force __kernel_rwf_t)0x00000040) +/* buffered IO that drops the cache after reading or writing data */ +#define RWF_UNCACHED ((__force __kernel_rwf_t)0x00000080) + /* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC) + RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\ + RWF_UNCACHED) #define PROCFS_IOCTL_MAGIC 'f' From patchwork Sun Nov 10 15:28:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869936 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3034156F4A for ; Sun, 10 Nov 2024 15:29:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252568; cv=none; b=gSzDvJJLJW5H3Fsx6DooVCcXJUb6At9GrAR9xpXPCVYeSv/9/9C/RhZLhv4lFcoPWB3pRVO58g46ydt/fm7pIsFLn0mSngUf11ss1dnftyThHs3yKFCColxbCZ0yOKo2YsCQiUkVuCJ2ph2Vxm8My23n15gwgGnVnM87Qi2VH8w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252568; c=relaxed/simple; bh=GtEZ1vh2U+C3BSovAlp6aWdv1IA1Yxh/R7JebXsMPYk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gM5MTfU2yRUqR/IoVrNWsNH8yyxyfv0pfgN7eqmJ+9n6GzT7A4Nc2hzBTXVWidqHwiye/Z5ML5MUgwdnNM00lAl2q3ysjejzC9QeZV0eqAP3LloXbJ9YvR6IA1IXmUXRz3XTY2M3v9oLE5SK2uiBg969tKlqsSlOT7mJYh7BYbk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=HHaWkcpa; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="HHaWkcpa" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2e2bd0e2c4fso2982971a91.3 for ; Sun, 10 Nov 2024 07:29:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252565; x=1731857365; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f2GrDSrtQqt8cMSioXW5eWMYaGSDX9LD8WbU7yXVjqU=; b=HHaWkcpaZkDLnKu53ZAGR9OB/MAQgZlLs61GpZhp9XOUDwp0vjTxEsTwscDQ9z9Gnp SOgjDyWlUXTFMieY/hjSZM2vRS51GGsBObOTwjUP4AWPbHxmJA1Tx1Q7EIuPfXwXcRTs /s9HKiYJMd5pH8ng5ngieLvwhWmkMCwguQy2yyPDTNMTXvy+9Rhqr8IOOwLHNr0aB73x KwWJHy7Owdp77qRB5HtDHb+MlhHt6GBFjfk/DtpKHW2oxSAOhmFQetclgajE1hrUR1j3 ryEa+/KaaacpqB3fo9+k3R+ZlkuIHuxJOkT8dXl/rsz6rAwHE7FO6ITIQqSv+1P+6TxN n8pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252565; x=1731857365; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f2GrDSrtQqt8cMSioXW5eWMYaGSDX9LD8WbU7yXVjqU=; b=h3cbpK9WmqGLl0SZudXpdR6mDMHQz0klw6B0VOHSkoSQB+H581KHosu5EauDf+/2jC uYcHC9uInehibOuzbnyGbSKufnxc6uzATbEl2dMjJubdfXja04SVxw2moOkcpys7eupX pEllIQe1qLIjg/3o1iodj4rdxcjieQVd/DGI5I67kQGXQ3rSXR82VVveNA6CBbyvDIZg 8v0OvibAaJ0hbmtLu9VfCM62Zs44wbzbMK8mF0oS0MPyXJfxm5n6iy7JEE8sr0RGToII L3ADBaXc/rhC3CE37ONad3DgLFQ4GGjL0M/bafGp8O7j8/Mh/IiBidScFtTRtd68Raz1 7hBw== X-Forwarded-Encrypted: i=1; AJvYcCXDQ/lBAvt04ukqOFAtZKP1pFwxIRSi4itAKQaIbUr/RnV9eBjhvA8rcVHVa6gFREsnZXEGQDLODEw7oH2Y@vger.kernel.org X-Gm-Message-State: AOJu0YyG1dI4ILueIj5obVNEo5mODUF3zda+agLjPiNNPtOkYaSYSuTm S0CZ5URZCy/PajK93Zuw2DI19W705vdr3yr3VdDRH02yp3mLqBcq4tXpWhapvpY= X-Google-Smtp-Source: AGHT+IHD5ZOaM1/ydWmKI6E2td5s5ih4bl0qHFridFSAcX9UJ78BlPPQ+BWJS6tide+7RYqVyEL5yA== X-Received: by 2002:a17:90b:2748:b0:2d8:840b:9654 with SMTP id 98e67ed59e1d1-2e9b1754affmr13371709a91.34.1731252565293; Sun, 10 Nov 2024 07:29:25 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:24 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 08/15] mm/filemap: add read support for RWF_UNCACHED Date: Sun, 10 Nov 2024 08:28:00 -0700 Message-ID: <20241110152906.1747545-9-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add RWF_UNCACHED as a read operation flag, which means that any data read wil be removed from the page cache upon completion. Uses the page cache to synchronize, and simply prunes folios that were instantiated when the operation completes. While it would be possible to use private pages for this, using the page cache as synchronization is handy for a variety of reasons: 1) No special truncate magic is needed 2) Async buffered reads need some place to serialize, using the page cache is a lot easier than writing extra code for this 3) The pruning cost is pretty reasonable and the code to support this is much simpler as a result. You can think of uncached buffered IO as being the much more attractive cousing of O_DIRECT - it has none of the restrictions of O_DIRECT. Yes, it will copy the data, but unlike regular buffered IO, it doesn't run into the unpredictability of the page cache in terms of reclaim. As an example, on a test box with 32 drives, reading them with buffered IO looks as follows: Reading bs 65536, uncached 0 1s: 145945MB/sec 2s: 158067MB/sec 3s: 157007MB/sec 4s: 148622MB/sec 5s: 118824MB/sec 6s: 70494MB/sec 7s: 41754MB/sec 8s: 90811MB/sec 9s: 92204MB/sec 10s: 95178MB/sec 11s: 95488MB/sec 12s: 95552MB/sec 13s: 96275MB/sec where it's quite easy to see where the page cache filled up, and performance went from good to erratic, and finally settles at a much lower rate. Looking at top while this is ongoing, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7535 root 20 0 267004 0 0 S 3199 0.0 8:40.65 uncached 3326 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd4 3327 root 20 0 0 0 0 R 100.0 0.0 0:17.22 kswapd5 3328 root 20 0 0 0 0 R 100.0 0.0 0:13.29 kswapd6 3332 root 20 0 0 0 0 R 100.0 0.0 0:11.11 kswapd10 3339 root 20 0 0 0 0 R 100.0 0.0 0:16.25 kswapd17 3348 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd26 3343 root 20 0 0 0 0 R 100.0 0.0 0:16.30 kswapd21 3344 root 20 0 0 0 0 R 100.0 0.0 0:11.92 kswapd22 3349 root 20 0 0 0 0 R 100.0 0.0 0:16.28 kswapd27 3352 root 20 0 0 0 0 R 99.7 0.0 0:11.89 kswapd30 3353 root 20 0 0 0 0 R 96.7 0.0 0:16.04 kswapd31 3329 root 20 0 0 0 0 R 96.4 0.0 0:11.41 kswapd7 3345 root 20 0 0 0 0 R 96.4 0.0 0:13.40 kswapd23 3330 root 20 0 0 0 0 S 91.1 0.0 0:08.28 kswapd8 3350 root 20 0 0 0 0 S 86.8 0.0 0:11.13 kswapd28 3325 root 20 0 0 0 0 S 76.3 0.0 0:07.43 kswapd3 3341 root 20 0 0 0 0 S 74.7 0.0 0:08.85 kswapd19 3334 root 20 0 0 0 0 S 71.7 0.0 0:10.04 kswapd12 3351 root 20 0 0 0 0 R 60.5 0.0 0:09.59 kswapd29 3323 root 20 0 0 0 0 R 57.6 0.0 0:11.50 kswapd1 [...] which is just showing a partial list of the 32 kswapd threads that are running mostly full tilt, burning ~28 full CPU cores. If the same test case is run with RWF_UNCACHED set for the buffered read, the output looks as follows: Reading bs 65536, uncached 0 1s: 153144MB/sec 2s: 156760MB/sec 3s: 158110MB/sec 4s: 158009MB/sec 5s: 158043MB/sec 6s: 157638MB/sec 7s: 157999MB/sec 8s: 158024MB/sec 9s: 157764MB/sec 10s: 157477MB/sec 11s: 157417MB/sec 12s: 157455MB/sec 13s: 157233MB/sec 14s: 156692MB/sec which is just chugging along at ~155GB/sec of read performance. Looking at top, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7961 root 20 0 267004 0 0 S 3180 0.0 5:37.95 uncached 8024 axboe 20 0 14292 4096 0 R 1.0 0.0 0:00.13 top where just the test app is using CPU, no reclaim is taking place outside of the main thread. Not only is performance 65% better, it's also using half the CPU to do it. Signed-off-by: Jens Axboe --- mm/filemap.c | 18 ++++++++++++++++-- mm/swap.c | 2 ++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 38dc94b761b7..bd698340ef24 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2474,6 +2474,8 @@ static int filemap_create_folio(struct kiocb *iocb, folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; + if (iocb->ki_flags & IOCB_UNCACHED) + __folio_set_uncached(folio); /* * Protect against truncate / hole punch. Grabbing invalidate_lock @@ -2519,6 +2521,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_async_ra(&ractl, folio, last_index - folio->index); return 0; } @@ -2548,6 +2552,8 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); @@ -2706,8 +2712,16 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, } } put_folios: - for (i = 0; i < folio_batch_count(&fbatch); i++) - folio_put(fbatch.folios[i]); + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + + if (folio_test_uncached(folio)) { + folio_lock(folio); + invalidate_complete_folio2(mapping, folio, 0); + folio_unlock(folio); + } + folio_put(folio); + } folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); diff --git a/mm/swap.c b/mm/swap.c index 835bdf324b76..f2457acae383 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -472,6 +472,8 @@ static void folio_inc_refs(struct folio *folio) */ void folio_mark_accessed(struct folio *folio) { + if (folio_test_uncached(folio)) + return; if (lru_gen_enabled()) { folio_inc_refs(folio); return; From patchwork Sun Nov 10 15:28:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869937 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79D9B156F5F for ; Sun, 10 Nov 2024 15:29:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252568; cv=none; b=qoA1FHb0cqmqUbM1qjxx4VyhaToDXAWH20rkWy2YslXMlwh0pdprt1lRP3FUQdyP80cmSlOu5stuGz9PSPpOnam01y/LRckJnbH5/z4YD/M99RJgh8AqURu2NYbF9mkEJRmI8XisjuHc9l+r1S10B7j1cUrOpJK6sGEEWctQJp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252568; c=relaxed/simple; bh=AKt+S745U93BEmbA6t/wcRBIgQj/soY+rT3nJS7qXJU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JL2gF/I6U8OUidraJa+cDu9epFGKVU1mWKonfGVnR8xB1oOIjoZiB55Z/LTAAaXJSqcwu5c53CBnpYdrM/RQKUEOGwzeqDeyJ+I0gqHfAkEpA6xHpZzJy+jV3lKwUIxdszDRWPS46nh8mzIi0DknIm42aDyOVpyx0cu0FF9xxb8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=jpxkdGSA; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="jpxkdGSA" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21145812538so34653245ad.0 for ; Sun, 10 Nov 2024 07:29:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252566; x=1731857366; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xtqsp3FpRWE9h3nra5m9D1cpUkXI8hkSCKuFCrWSblw=; b=jpxkdGSACE2RIfxZR30I6RLxlbnIVcV8CQP+caD3TJTf9MGEGC2rENCIMrdqFSyPRW sximQLDX7ngw6JwKo6AoBMqN06YroHpC69obYTg9zcR5TfthsZZKuRbYrE4DxXnDpU+W yNp+1ynCzu3wN3TExVe/C0PsKAYsSkZusARdnCjiszGVukDMpprVY8wcpxSJn8tJNM/A DojcecxZxsAZafCdfcjlenpPY0f1UhUBl2p0KFkYWWdxG3LXKTHoTUbrm22m5NvlpL9D tfYK5FwXHhCl7N+++ecZbUuTVnkhOQeBJHYZ0QrL8yM7tLJY1HQ1KkcNb8WwDQTc8uV+ pAjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252566; x=1731857366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xtqsp3FpRWE9h3nra5m9D1cpUkXI8hkSCKuFCrWSblw=; b=iCKx4PEsbMYYWDn93ESPNWAgbS6Z591xOHRlhGm8OeZfOBRhoGe5QlqyVWn2qyaPka x0NtV9QpuqxQ2bsPEKOFYzji8LmgK7Nz4MBPt5w0g4y2J4ctvZFRy7BGL8iAtkDWwT4q XnUEnaeCCa2V3ofJlOJEe9Rp12SmhVZSfBFSxfJ5UH4c7zYY0XlvDSsbPrPGI15ARXMB 8vQsA58+FdZ/uigKlLbADolLHouQ/KYwGGJIUMsggo8yCYZ/lPJIfHnHjEimCdINzlER eUykJj01KajZJKdWV+EWACR4/fJWH+w6KrujFjjTEmuOaHVdEwg4joGwUpNy0Xwnywo0 rGKw== X-Forwarded-Encrypted: i=1; AJvYcCVpI5LsmN7GSp+cHaANI6LLVdCx7Hd2gd0/YjUS3aN7eNmc/FxdGwrDnLYGyCViCL9iNLaVsthSGsOay1//@vger.kernel.org X-Gm-Message-State: AOJu0YySOSDGb4UuR86hTvnSLnt/NUo9GdFMz6JbAYGSy8P9QNcVJoif HOIEHGYS/fmBrHJW1jQ9f7p1cRqaj6TTP3Og9R3GpK46NEqd7sJnY38UG2/s7rI= X-Google-Smtp-Source: AGHT+IF5qTNuztLawL22nFdoWM3hBa8g+iEMn3AucUKZX2m0zot7CflQJgWfjjOvLTZBT0mpwh+/bA== X-Received: by 2002:a17:902:cf02:b0:20c:af07:a816 with SMTP id d9443c01a7336-21183d087c7mr129753345ad.31.1731252566678; Sun, 10 Nov 2024 07:29:26 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:25 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 09/15] mm/filemap: drop uncached pages when writeback completes Date: Sun, 10 Nov 2024 08:28:01 -0700 Message-ID: <20241110152906.1747545-10-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO. Signed-off-by: Jens Axboe --- mm/filemap.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index bd698340ef24..efd02b047541 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1600,6 +1600,23 @@ int folio_wait_private_2_killable(struct folio *folio) } EXPORT_SYMBOL(folio_wait_private_2_killable); +/* + * If folio was marked as uncached, then pages should be dropped when writeback + * completes. Do that now. If we fail, it's likely because of a big folio - + * just reset uncached for that case and latter completions should invalidate. + */ +static void folio_end_uncached(struct folio *folio) +{ + bool reset = true; + + if (folio_trylock(folio)) { + reset = !invalidate_complete_folio2(folio->mapping, folio, 0); + folio_unlock(folio); + } + if (reset) + folio_set_uncached(folio); +} + /** * folio_end_writeback - End writeback against a folio. * @folio: The folio. @@ -1610,6 +1627,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); */ void folio_end_writeback(struct folio *folio) { + bool folio_uncached; + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); /* @@ -1631,9 +1650,13 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake_bit(). */ folio_get(folio); + folio_uncached = folio_test_clear_uncached(folio); if (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); + + if (folio_uncached) + folio_end_uncached(folio); folio_put(folio); } EXPORT_SYMBOL(folio_end_writeback); From patchwork Sun Nov 10 15:28:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869938 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D543C1581F9 for ; Sun, 10 Nov 2024 15:29:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252570; cv=none; b=jIekJ/mSYDc/Q6sIbdIgGP7ENEoX62TjkbwxwAQ9cubdCPMn7iDXuFCpfxcSGhYetUGet640p5rJzg4fPGyWOjR9mhjBo6gKFmnogLPeKv7xCnCwDhhS3f1Po+j3nqkbFav97/hObnRYD/ADf/lySqRHETGi4Ndp8GZHEKfTRAg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252570; c=relaxed/simple; bh=DvOUyhYLGoxN2Hc3acnkKQpks6bE7rAJDoBJQe3HSFQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hXJsJS5RS5TYJlg3YCaF4Srw2UGLFNr6MNo+L4FIJuZrSp2hTc7Y4ty+DvWM5fj95WqtddqzuIGmbsJr1vd08vnEPkINtV6rdlr8qUjXHskFvKX0zUA3DEAWrqMTlhVmJwFm4CTNtm1IepnzAX0oywgc3nwrLatOnyLeQXs+Ikg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=jDTXabpf; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="jDTXabpf" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-20cdb889222so38293565ad.3 for ; Sun, 10 Nov 2024 07:29:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252568; x=1731857368; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c6ru6+H4oQlmjhy4ZS59TJ0rgrpyxDTaEbRj/Vr+gBQ=; b=jDTXabpfNmkfTfN2pKKft5LnVS8ofQE3fRSSULoPO1MAPz4TIQ+PKVaf7z0nIi0kSO KvragGR9eX+cguqb+WAeeso1/5o2cWWY6UyU6Hx6wZA3vNGcq5XOUR/iWOTYfYhABKvp KZRFEfWMokybmbHgkrKIRLqw4ChQvepExYoYND9RJ9OnCER2tIexck99O/ZvAZTa3Buc CPoGYkOnUEoEx4kbQ1ir7xeL0g3xVNL0S/AmGa9T8ktMxatRNVoDfc1jW0R2MtFnEErs rOxpD/plCD7khf76VgDabfo57/lzrl+xXlCt+c8ofJpdhN2AswS/fI/cAQJjgbAuqfv2 gsbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252568; x=1731857368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c6ru6+H4oQlmjhy4ZS59TJ0rgrpyxDTaEbRj/Vr+gBQ=; b=EN7yJgW3rL31dMTZMvJVye4MgM7MlShBFCmO7UnKDvaPzixIkMCEjH59ZEDrLVaCjr AwGPL9oFhRvB9WnEXK/Am5nmeCIH29rR8gTgtcc9UFUrrzZPqLf6TjdymSbko/x+banL TiqZ1/A0MV8ZDSftw/sIX2P7k3O3oBFTV+0WY6hmnVWiQaY0MXVdthWBve8dMrPBwQ82 fjG2GhkYCN79gRxwvuaXcGGTFzX5Ow4EJgyyXRQUKlcVGWMKdLhPkBK4fL3v8TOjxPlS BID72yhB2LYsAUKooowb3hUD4HaQhBJb0M8WA/s17KIlC5BsHt3wgdMkzfzpqZixGe85 D16g== X-Forwarded-Encrypted: i=1; AJvYcCXPKcDra+icaH07OPTTQQNwFG0CTmUOVSdaLk01qNi2lVbxZFVkRh5dn1YykoHETKG07rQu7EPTGpH7IP1H@vger.kernel.org X-Gm-Message-State: AOJu0YzPC9Cc/7FsHkrzMXL1Uif3pW7cXk9HVjOrAzl/3yN2xkN7X139 jNHn+Vmr0AP45tNsliZYeNj/6zfOsLnuW7u2BULS3E1NmsWdPF5ptksPpQiCekg= X-Google-Smtp-Source: AGHT+IEOVFhFei02kQq1ltQWyF+j1C3XIjJZGZV1LAM2AT1TQ/js4+EAJqEYuNTXGh31ae8isAOkHA== X-Received: by 2002:a17:903:2391:b0:20c:b274:34d0 with SMTP id d9443c01a7336-211835d19e6mr136506735ad.46.1731252568109; Sun, 10 Nov 2024 07:29:28 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:27 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 10/15] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Sun, 10 Nov 2024 08:28:02 -0700 Message-ID: <20241110152906.1747545-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary. Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached. Add a helper, generic_uncached_write(), that generic_file_write_iter() calls upon successful completion of an uncached write. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec, MB=196035 2s: 132308MB/sec, MB=328147 3s: 132438MB/sec, MB=460586 4s: 116528MB/sec, MB=577115 5s: 103898MB/sec, MB=681014 6s: 108893MB/sec, MB=789907 7s: 99678MB/sec, MB=889586 8s: 106545MB/sec, MB=996132 9s: 106826MB/sec, MB=1102958 10s: 101544MB/sec, MB=1204503 11s: 111044MB/sec, MB=1315548 12s: 124257MB/sec, MB=1441121 13s: 116031MB/sec, MB=1557153 14s: 114540MB/sec, MB=1671694 15s: 115011MB/sec, MB=1786705 16s: 115260MB/sec, MB=1901966 17s: 116068MB/sec, MB=2018034 18s: 116096MB/sec, MB=2134131 where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 29 +++++++++++++++++++++++++++++ mm/filemap.c | 26 +++++++++++++++++++++++--- 2 files changed, 52 insertions(+), 3 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0122b3fbe2ac..5469664f66c3 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,6 +14,7 @@ #include #include #include /* for in_interrupt() */ +#include #include struct folio_batch; @@ -70,6 +71,34 @@ static inline int filemap_write_and_wait(struct address_space *mapping) return filemap_write_and_wait_range(mapping, 0, LLONG_MAX); } +/* + * generic_uncached_write - start uncached writeback + * @iocb: the iocb that was written + * @written: the amount of bytes written + * + * When writeback has been handled by write_iter, this helper should be called + * if the file system supports uncached writes. If %IOCB_UNCACHED is set, it + * will kick off writeback for the specified range. + */ +static inline void generic_uncached_write(struct kiocb *iocb, ssize_t written) +{ + if (iocb->ki_flags & IOCB_UNCACHED) { + struct address_space *mapping = iocb->ki_filp->f_mapping; + + /* kick off uncached writeback */ + __filemap_fdatawrite_range(mapping, iocb->ki_pos, + iocb->ki_pos + written, WB_SYNC_NONE); + } +} + +/* + * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write, + * and the ->write_begin() handler on a file system supporting FOP_UNCACHED + * must check for this and pass FGP_UNCACHED for folio creation. + */ +#define foliop_uncached ((struct folio *) 0xfee1c001) +#define foliop_is_uncached(foliop) (*(foliop) == foliop_uncached) + /** * filemap_set_wb_err - set a writeback error on an address_space * @mapping: mapping in which to set writeback error diff --git a/mm/filemap.c b/mm/filemap.c index efd02b047541..cfbfc8b14b1f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -430,6 +430,7 @@ int __filemap_fdatawrite_range(struct address_space *mapping, loff_t start, return filemap_fdatawrite_wbc(mapping, &wbc); } +EXPORT_SYMBOL_GPL(__filemap_fdatawrite_range); static inline int __filemap_fdatawrite(struct address_space *mapping, int sync_mode) @@ -1609,7 +1610,14 @@ static void folio_end_uncached(struct folio *folio) { bool reset = true; - if (folio_trylock(folio)) { + /* + * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but + * can happen if normal writeback just happens to find dirty folios + * that were created as part of uncached writeback, and that writeback + * would otherwise not need non-IRQ handling. Just skip the + * invalidation in that case. + */ + if (in_task() && folio_trylock(folio)) { reset = !invalidate_complete_folio2(folio->mapping, folio, 0); folio_unlock(folio); } @@ -4061,7 +4069,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) ssize_t written = 0; do { - struct folio *folio; + struct folio *folio = NULL; size_t offset; /* Offset into folio */ size_t bytes; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ @@ -4089,6 +4097,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) break; } + /* + * If IOCB_UNCACHED is set here, we now the file system + * supports it. And hence it'll know to check folip for being + * set to this magic value. If so, it's an uncached write. + * Whenever ->write_begin() changes prototypes again, this + * can go away and just pass iocb or iocb flags. + */ + if (iocb->ki_flags & IOCB_UNCACHED) + folio = foliop_uncached; + status = a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0)) @@ -4219,8 +4237,10 @@ ssize_t generic_file_write_iter(struct kiocb *iocb, struct iov_iter *from) ret = __generic_file_write_iter(iocb, from); inode_unlock(inode); - if (ret > 0) + if (ret > 0) { + generic_uncached_write(iocb, ret); ret = generic_write_sync(iocb, ret); + } return ret; } EXPORT_SYMBOL(generic_file_write_iter); From patchwork Sun Nov 10 15:28:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869939 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FD981586F2 for ; Sun, 10 Nov 2024 15:29:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252571; cv=none; b=DER+ZwzIYc3p9faWrFU8QHSn5S1XhW9FTKrjNQ1sFjxU8jhqjO0YcmtcgmIHA6oIGY48MJ14ZXQnn2pwnkHXVtYuBYCJ7WlieuZc/pn0SKTokBX9U+Km9ChwBqFAvJUJlATRzspUUwzw4IEo37b1lwE98acyXmVX4ZfVUAUZvvQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252571; c=relaxed/simple; bh=7n4q+dbKFb0eQ3Qygk7Z+kektGylKUoxfQQqH74ekkQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C51dOXtY+xI0+lkhWzv4na5anAsdf7Z5HPbfS+mwZIEaiZDjE08Dck4dDbyk8epBQ01iKmcFAJJais+4zDKSZFURon12HS1UW4q1xWdnh8FQZAv2cpGWZddlzz0cX6LJuLtUrsvXOnkXaKQ/tQHBacp+wbI6HTEZ3Yla9rpty6M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Wy+4sM/C; arc=none smtp.client-ip=209.85.216.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Wy+4sM/C" Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-2e2bb1efe78so2713895a91.1 for ; Sun, 10 Nov 2024 07:29:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252569; x=1731857369; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2Ft1nD1O8TW6ajHAbmpyRJ7eqllXRgDsj9KVxLudGNI=; b=Wy+4sM/Cvbe99m987p/3qS0IogzcZGYbEAMzNhAT4Eaj0llnpLqnLvEMmp5lJJHWJA zyaq38zy5wPRH0tnH7L+Y5YJKpTO1ZFJYv3tnOvys6EvpbVMao+xkiACAmkS0MoKYbuP WuTlPbhxGXVtjRWii2iAcJeS0eI+eKiTgHhuJ/3Wqhd7ad3KJRHuOhFVueF6SDVDU7Vj LDHQvPS3vVIymeGFiMUdlJjHVyqDzvlWd6znEyUeWK285+L77j+Bv+aNGUahC6fy1WTG gKZ8c22qIckzpA1GNJAhs5lHilnHB3AdkzruNGxnEzUCX/P0Kr6hmZ9cd45FoCWBeSNF 7+fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252569; x=1731857369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2Ft1nD1O8TW6ajHAbmpyRJ7eqllXRgDsj9KVxLudGNI=; b=AM1X/l6k/ji317uMm1ZIbdUbbuwUL5/jwAixmqUddJl0Zar17Z/moCdJC6OKuJfPv7 Whjb6VMcuEBMO5inGcJeatJdRZFV0BEa3iNBe4DVGQ8JJN9h27WV1B3bKQWzEmopLDUz pYL8NXihefrqKoEfGvYOkev1X5341VUEF1mOvlPFbOo/N+6pqj5YVUkqUwyOg6Ngv9Ek vYD6Q6vzK9QWrQxuJ08iB5zwiz1qcTCFh2pKpPZ7v06lrf4zbOVR02BYaepfxjcz1BOy 5XrZAcyy+lr/WEuNCFDL83X4rU/xznhooSq2ExrmkeGdR24FFuJUvNzfg93AZnBD+pOp 8YEw== X-Forwarded-Encrypted: i=1; AJvYcCWInWRpXRvqBBSQEZFgGlWs1wCyPu72OBwvWj07k+IZJMRbeJTfaU1bYpNudU9RVN+F3cm+lSe/sBqztIGh@vger.kernel.org X-Gm-Message-State: AOJu0YyFegzKwJgs3RH7J+lggecm3qgXpB4/3k4C8syb6khQyvPMF7mU 9nRAEpu8S9RTNXXIAAU3fCrjlqaSgzhY8OMtlKlHjayoD5HrfKDo0GGaHLtthbI= X-Google-Smtp-Source: AGHT+IGnzg2hDaZ2lVFGocJncR7soOdtnLL/Yc809zXi6ImqKmn+BDcRSWipzylZxh/jlIYcFDHDiw== X-Received: by 2002:a17:90b:1bc3:b0:2e0:7b03:1908 with SMTP id 98e67ed59e1d1-2e9b0a57d33mr14506762a91.10.1731252569648; Sun, 10 Nov 2024 07:29:29 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:28 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 11/15] mm: add FGP_UNCACHED folio creation flag Date: Sun, 10 Nov 2024 08:28:03 -0700 Message-ID: <20241110152906.1747545-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Callers can pass this in for uncached folio creation, in which case if a folio is newly created it gets marked as uncached. If a folio exists for this index and lookup succeeds, then it will not get marked as uncached. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 5469664f66c3..de0ed906cd2d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -741,6 +741,7 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, * * %FGP_NOFS - __GFP_FS will get cleared in gfp. * * %FGP_NOWAIT - Don't block on the folio lock. * * %FGP_STABLE - Wait for the folio to be stable (finished writeback) + * * %FGP_UNCACHED - Uncached buffered IO * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */ @@ -754,6 +755,7 @@ typedef unsigned int __bitwise fgf_t; #define FGP_NOWAIT ((__force fgf_t)0x00000020) #define FGP_FOR_MMAP ((__force fgf_t)0x00000040) #define FGP_STABLE ((__force fgf_t)0x00000080) +#define FGP_UNCACHED ((__force fgf_t)0x00000100) #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) diff --git a/mm/filemap.c b/mm/filemap.c index cfbfc8b14b1f..4fdf3c4ae00f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1987,6 +1987,8 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, /* Init accessed so avoid atomic mark_page_accessed later */ if (fgp_flags & FGP_ACCESSED) __folio_set_referenced(folio); + if (fgp_flags & FGP_UNCACHED) + __folio_set_uncached(folio); err = filemap_add_folio(mapping, folio, index, gfp); if (!err) From patchwork Sun Nov 10 15:28:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869940 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E39A480027 for ; Sun, 10 Nov 2024 15:29:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252573; cv=none; b=JYkMPsTx3VQgOU7VfxYQVkIVmXSAFKOdzhbx/SPQljCp075HkUiltoq2HHkxsPPSW577m3DmXkTMuAS2iAimy8wuHk1rpSqyyGT3jRRamQB9YvBtMuO0kllLStyDYHQaG8dme4V4+W6DBraA0Z7nuMEpX9DhnQWyZl9Z2RueC1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252573; c=relaxed/simple; bh=6ZSmJO+bG/O0VvG/m0Wnv2BJ9smDd+VmRMf5KfhKfrc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CN9Dm/1OxUjNdZ7pnSFZi1jtsRoC8faZxUn/fpvZ27A71jHfNqqntYkUh1FpkTw5taaezoVpEtNSajsIu1P0uOQyQnWVh8I1s50+0ql+02IYufbwsXFWiVu4r5OetsWSHuFELe4gpMlqf94I6GvbR2zOsL+54Y031STGiThiuN4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=fpRWAWEC; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="fpRWAWEC" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2e2b549799eso2991507a91.3 for ; Sun, 10 Nov 2024 07:29:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252571; x=1731857371; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TkS4Adcluut9E9TBSf0Q5Dr6BdVnPYdslAPkTGEhwiY=; b=fpRWAWECI5OtTQlcLOn/rgoWffS5D1nG5AfXfjwzuPyRLJ8+q4/d1VRom1qGPd9paI 3wZm6TypwMh8/eeTb7hrykWDheH5TKWVjEdZP9S6jtkO3On+oPWhGGRz6L6nMgnRLyHg kRrTTdnBQoh9VNuPxK/nMU6aDLydSJNxkO1knvpLPPuGyqUuKkMg/HkYwFzpGgZnLcGa pJBW+xlqZuQb32kfOp190V4ODE7ol7jvjmZxNL0lPnzUsyqfe3xxdAvW1p5m1Trf+VCI Keiya1aYrm4Yqd4XZqCOH6zE7ywioLUtihFSRxR4uB5xQ/wKeN9IvJuATU6+sODzi3sL IOtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252571; x=1731857371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TkS4Adcluut9E9TBSf0Q5Dr6BdVnPYdslAPkTGEhwiY=; b=OjhrX+9aRuMsGRDXXBfJ7SQZvsqTYDBTsL8/mM2AtT5uzVSRN5Hu84qoTd+aMN7mp4 x1iLEFq5g2Y7RGGnMIiWWkIiE2PcGVFfKQfXJ6h8PTeY8h8da5srGyzuIj+B3qQ4mHNC hK7Eq4sdLmCUZw5Lgmbf9BJQXr4UCODbkiNyCcavTEfOlB+v7MbXcRRQa/g9dDC949Vo Ivn0ku7ZT7otvSNfLGeViMqvPr4GKKNUP2CGTbK7DVezHoBkq0fmyBgFz1C6AN/X0RUW 4/KZ+VGUyySBB0OnPjWXUxbW7xJPxs0i6xstS85o/qS163CekzJCVRWzCgCHEc1BsgW6 +BtQ== X-Forwarded-Encrypted: i=1; AJvYcCVY6FHwXxqefgYRdzif3+GbIa8raQOnGDlclKiQmGN0RcphzymaAdZcjfuZ0eJYfGFLDJVTo03YHrdqUrY/@vger.kernel.org X-Gm-Message-State: AOJu0YxltErYGqFLPrSw1oIuwEQ0T7VFs2AzMXG99zxVMET6STo18jvi RL9ArVApIdzopJuiO3HdCLj8yIwWKoaysb74OKDxp1XjxFBujt+QdGsJJFv0srtruPtwzXqP+Q8 kamE= X-Google-Smtp-Source: AGHT+IHRHbLBUb09CwGoIXN66uipNoc+nR4rS0OGfJWtMF1GUwRkiOluIK+uk59p6smvN8SFsh8EQg== X-Received: by 2002:a17:90b:2b8e:b0:2e2:b69c:2a9 with SMTP id 98e67ed59e1d1-2e9b1770362mr13736497a91.26.1731252571252; Sun, 10 Nov 2024 07:29:31 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:30 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 12/15] ext4: add RWF_UNCACHED write support Date: Sun, 10 Nov 2024 08:28:04 -0700 Message-ID: <20241110152906.1747545-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that. If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache. Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it. Signed-off-by: Jens Axboe --- fs/ext4/ext4.h | 1 + fs/ext4/file.c | 2 +- fs/ext4/inline.c | 7 ++++++- fs/ext4/inode.c | 18 ++++++++++++++++-- fs/ext4/page-io.c | 28 ++++++++++++++++------------ 5 files changed, 40 insertions(+), 16 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 44b0d418143c..60dc9ffae076 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -279,6 +279,7 @@ struct ext4_system_blocks { * Flags for ext4_io_end->flags */ #define EXT4_IO_END_UNWRITTEN 0x0001 +#define EXT4_IO_UNCACHED 0x0002 struct ext4_io_end_vec { struct list_head list; /* list of io_end_vec */ diff --git a/fs/ext4/file.c b/fs/ext4/file.c index f14aed14b9cf..0ef39d738598 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -944,7 +944,7 @@ const struct file_operations ext4_file_operations = { .splice_write = iter_file_splice_write, .fallocate = ext4_fallocate, .fop_flags = FOP_MMAP_SYNC | FOP_BUFFER_RASYNC | - FOP_DIO_PARALLEL_WRITE, + FOP_DIO_PARALLEL_WRITE | FOP_UNCACHED, }; const struct inode_operations ext4_file_inode_operations = { diff --git a/fs/ext4/inline.c b/fs/ext4/inline.c index 3536ca7e4fcc..4089d0744164 100644 --- a/fs/ext4/inline.c +++ b/fs/ext4/inline.c @@ -667,6 +667,7 @@ int ext4_try_to_write_inline_data(struct address_space *mapping, handle_t *handle; struct folio *folio; struct ext4_iloc iloc; + fgf_t fgp_flags; if (pos + len > ext4_get_max_inline_size(inode)) goto convert; @@ -702,7 +703,11 @@ int ext4_try_to_write_inline_data(struct address_space *mapping, if (ret) goto out; - folio = __filemap_get_folio(mapping, 0, FGP_WRITEBEGIN | FGP_NOFS, + fgp_flags = FGP_WRITEBEGIN | FGP_NOFS; + if (*foliop == foliop_uncached) + fgp_flags |= FGP_UNCACHED; + + folio = __filemap_get_folio(mapping, 0, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) { ret = PTR_ERR(folio); diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 54bdd4884fe6..afae3ab64c9e 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, int ret, needed_blocks; handle_t *handle; int retries = 0; + fgf_t fgp_flags; struct folio *folio; pgoff_t index; unsigned from, to; @@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, return 0; } + /* + * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop contains + * foliop_uncached. That's how generic_perform_write() informs us + * that this is an uncached write. + */ + fgp_flags = FGP_WRITEBEGIN; + if (*foliop == foliop_uncached) + fgp_flags |= FGP_UNCACHED; + /* * __filemap_get_folio() can take a long time if the * system is thrashing due to memory pressure, or if the folio @@ -1172,7 +1182,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, * the folio (if needed) without using GFP_NOFS. */ retry_grab: - folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, + folio = __filemap_get_folio(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) return PTR_ERR(folio); @@ -2903,6 +2913,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping, struct folio *folio; pgoff_t index; struct inode *inode = mapping->host; + fgf_t fgp_flags; if (unlikely(ext4_forced_shutdown(inode->i_sb))) return -EIO; @@ -2926,8 +2937,11 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping, return 0; } + fgp_flags = FGP_WRITEBEGIN; + if (*foliop == foliop_uncached) + fgp_flags |= FGP_UNCACHED; retry: - folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, + folio = __filemap_get_folio(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) return PTR_ERR(folio); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index ad5543866d21..10447c3c4ff1 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -226,8 +226,6 @@ static void ext4_add_complete_io(ext4_io_end_t *io_end) unsigned long flags; /* Only reserved conversions from writeback should enter here */ - WARN_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN)); - WARN_ON(!io_end->handle && sbi->s_journal); spin_lock_irqsave(&ei->i_completed_io_lock, flags); wq = sbi->rsv_conversion_wq; if (list_empty(&ei->i_rsv_conversion_list)) @@ -252,7 +250,7 @@ static int ext4_do_flush_completed_IO(struct inode *inode, while (!list_empty(&unwritten)) { io_end = list_entry(unwritten.next, ext4_io_end_t, list); - BUG_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN)); + BUG_ON(!(io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED))); list_del_init(&io_end->list); err = ext4_end_io_end(io_end); @@ -287,14 +285,15 @@ ext4_io_end_t *ext4_init_io_end(struct inode *inode, gfp_t flags) void ext4_put_io_end_defer(ext4_io_end_t *io_end) { - if (refcount_dec_and_test(&io_end->count)) { - if (!(io_end->flag & EXT4_IO_END_UNWRITTEN) || - list_empty(&io_end->list_vec)) { - ext4_release_io_end(io_end); - return; - } - ext4_add_complete_io(io_end); + if (!refcount_dec_and_test(&io_end->count)) + return; + if ((!(io_end->flag & EXT4_IO_END_UNWRITTEN) || + list_empty(&io_end->list_vec)) && + !(io_end->flag & EXT4_IO_UNCACHED)) { + ext4_release_io_end(io_end); + return; } + ext4_add_complete_io(io_end); } int ext4_put_io_end(ext4_io_end_t *io_end) @@ -348,7 +347,7 @@ static void ext4_end_bio(struct bio *bio) blk_status_to_errno(bio->bi_status)); } - if (io_end->flag & EXT4_IO_END_UNWRITTEN) { + if (io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED)) { /* * Link bio into list hanging from io_end. We have to do it * atomically as bio completions can be racing against each @@ -417,8 +416,13 @@ static void io_submit_add_bh(struct ext4_io_submit *io, submit_and_retry: ext4_io_submit(io); } - if (io->io_bio == NULL) + if (io->io_bio == NULL) { io_submit_init_bio(io, bh); + if (folio_test_uncached(folio)) { + ext4_io_end_t *io_end = io->io_bio->bi_private; + io_end->flag |= EXT4_IO_UNCACHED; + } + } if (!bio_add_folio(io->io_bio, io_folio, bh->b_size, bh_offset(bh))) goto submit_and_retry; wbc_account_cgroup_owner(io->io_wbc, &folio->page, bh->b_size); From patchwork Sun Nov 10 15:28:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869941 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57E9915E5C2 for ; Sun, 10 Nov 2024 15:29:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252574; cv=none; b=s4AwynYE2fWgUjkSv/W7C+7XmotnQd7Dj/E3h0W1oaB2x2vLlQljmqyOT2xzj3ysPXHzsJBOLGdrVuvHM9kNmdapAi22yS1vuau86Ktyht20xcNX8vTS5pRE6wSO8D6WDBbXXdf5wxDLP/MUxme17ijukZjOB69FTKi0rPaGcYw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252574; c=relaxed/simple; bh=3H9STUASulXCHklYsN+md87lfFFsAKUuDScGACrWthk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sCFnR60y2WEKAUUdrPqqHhb3jo0biFDZfZgQe70AqdFjs6KM4u29fcAIlavGCbN27gR1B4i6KsSIBvFg250mWGLgaMDCYYRPxQi12bbp+CRxkUZYRFJTT/6+da+hqrvVvJXZMTAmFl3pcFjio7N2UcGw6uL1OfNRx6WDnhiVqz0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=OI/Ky1or; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="OI/Ky1or" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-71ec997ad06so3029464b3a.3 for ; Sun, 10 Nov 2024 07:29:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252572; x=1731857372; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QHHnA1NsQjr1van9cnJ+PQRWRXhY+a6ETIPA3kXFfjk=; b=OI/Ky1orJg/GBr9s+FgVuIuEalVhEHGaXMQuHn25NW3gIZ/Ttc5S3RVPzHrSyU1a45 8/H0FAdjTx8OKHd5zHDluKqXO/cxBAALgmb3ngFG1S8q0graYRHyAmyWEt3Lx54pNApI L+XinvPTxzVdMyVeTKk6LIjENaoxTWyyzkf8hIPn/zIM+UH83uvkWvkgZ0AXzGwy6DYf XTGRA1vNNwlqd9zamy/7e5Fmyl39evdcvNE817XPskBp+vQPfxQo7uI71i5KOhOWuTkp a8y6r8F3Y2HpuICeeC2zA9uWMN9Pr4vEMBvhy4KV/8mVrR5mvaPWkvQUcNKb2OvPwg8E QmNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252572; x=1731857372; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QHHnA1NsQjr1van9cnJ+PQRWRXhY+a6ETIPA3kXFfjk=; b=Dlt7loG8kvZhO7eanzGcFv681AEiB3ja80iMFXNbVhnTS3k9JNYTfFff67dSgzt2qJ EwZ8Sbttfizd8YA1mIlwOPzYCKJ/Cq1StOzAAICixBPw0oM/QPKaexVBlxJorwzEu7Xb 3MSoXu8kGNnoN8EmALaDIgYPr+71UQYtAFLecP6jxdTBOH8Zo6iAgC9zf/5pj2MrYri+ 4zziwvFMzHC+OwWujsvjdQF4dMBiHCbYjHweE4mmPquug4mVYEwr6CyueACm8OdAC3ss 6vf/r0L67OV0v9ev4BYZmgK6ihWqXsTRkIGekbJ0hlkH7GYWYa3gTewRp3ubnTPucsEN 37QQ== X-Forwarded-Encrypted: i=1; AJvYcCWRoxj+Kig5V9Laa0yk0KWTxyPR9OKBwBbWY6EDBo9hQciw3rqICT5mazTAKiEnYdtS7LQrG+PCUQegNUy2@vger.kernel.org X-Gm-Message-State: AOJu0YxhZZD5MtXEYQX2FXMDnj5ebDwzd5FN83LqPk0syf20VggRF1Tm 9/r05lNhSpy0YGujCHAWmNXQul+B7/m+iNRRDvmLmn6OVtnoddNPChsJo39QGx4= X-Google-Smtp-Source: AGHT+IHV0wAiD3bQchKcabbTvXh2pU9AbMQUOshna304t+XB0bKs6GgV7ufQgRJwzKJwVoPNWeBA0A== X-Received: by 2002:a17:90a:e7c1:b0:2da:9115:15ce with SMTP id 98e67ed59e1d1-2e9b1682714mr12810753a91.15.1731252572603; Sun, 10 Nov 2024 07:29:32 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:31 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 13/15] iomap: make buffered writes work with RWF_UNCACHED Date: Sun, 10 Nov 2024 08:28:05 -0700 Message-ID: <20241110152906.1747545-14-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written with drop_writeback. Then writeback completion will drop the pages. The write_iter handler simply kicks off writeback for the pages, and writeback completion will take care of the rest. This still needs the user of the iomap buffered write helpers to call iocb_uncached_write() upon successful issue of the writes. Signed-off-by: Jens Axboe --- fs/iomap/buffered-io.c | 15 +++++++++++++-- include/linux/iomap.h | 4 +++- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index ef0b68bccbb6..2f2a5db04a68 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -603,6 +603,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len) if (iter->flags & IOMAP_NOWAIT) fgp |= FGP_NOWAIT; + if (iter->flags & IOMAP_UNCACHED) + fgp |= FGP_UNCACHED; fgp |= fgf_set_order(len); return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT, @@ -1023,8 +1025,9 @@ ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i, const struct iomap_ops *ops, void *private) { + struct address_space *mapping = iocb->ki_filp->f_mapping; struct iomap_iter iter = { - .inode = iocb->ki_filp->f_mapping->host, + .inode = mapping->host, .pos = iocb->ki_pos, .len = iov_iter_count(i), .flags = IOMAP_WRITE, @@ -1034,9 +1037,14 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i, if (iocb->ki_flags & IOCB_NOWAIT) iter.flags |= IOMAP_NOWAIT; + if (iocb->ki_flags & IOCB_UNCACHED) + iter.flags |= IOMAP_UNCACHED; - while ((ret = iomap_iter(&iter, ops)) > 0) + while ((ret = iomap_iter(&iter, ops)) > 0) { + if (iocb->ki_flags & IOCB_UNCACHED) + iter.iomap.flags |= IOMAP_F_UNCACHED; iter.processed = iomap_write_iter(&iter, i); + } if (unlikely(iter.pos == iocb->ki_pos)) return ret; @@ -1770,6 +1778,9 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, size_t poff = offset_in_folio(folio, pos); int error; + if (folio_test_uncached(folio)) + wpc->iomap.flags |= IOMAP_F_UNCACHED; + if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, pos)) { new_ioend: error = iomap_submit_ioend(wpc, 0); diff --git a/include/linux/iomap.h b/include/linux/iomap.h index f61407e3b121..2efc72df19a2 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -64,6 +64,7 @@ struct vm_fault; #define IOMAP_F_BUFFER_HEAD 0 #endif /* CONFIG_BUFFER_HEAD */ #define IOMAP_F_XATTR (1U << 5) +#define IOMAP_F_UNCACHED (1U << 6) /* * Flags set by the core iomap code during operations: @@ -173,8 +174,9 @@ struct iomap_folio_ops { #define IOMAP_NOWAIT (1 << 5) /* do not block */ #define IOMAP_OVERWRITE_ONLY (1 << 6) /* only pure overwrites allowed */ #define IOMAP_UNSHARE (1 << 7) /* unshare_file_range */ +#define IOMAP_UNCACHED (1 << 8) /* uncached IO */ #ifdef CONFIG_FS_DAX -#define IOMAP_DAX (1 << 8) /* DAX mapping */ +#define IOMAP_DAX (1 << 9) /* DAX mapping */ #else #define IOMAP_DAX 0 #endif /* CONFIG_FS_DAX */ From patchwork Sun Nov 10 15:28:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869942 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF67F16DECB for ; Sun, 10 Nov 2024 15:29:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252576; cv=none; b=qwzOhGI5I2XX6R4kP2xj1vlt2DKk7nHv7CZ/oma5rTqEgZQT1seEEm6zPEtidT3gHI5pk5wpmWJKzsiAEEqVw553ddGs9Xm+EDC2EtdCxH4joxlyy6uJoNwldOj8jtNLDb/mrocZ3eiDLLR8BRSxgwI2ZwnD8znZnCk04OVchMc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252576; c=relaxed/simple; bh=JN6ziuaz3h3w93syhui0sSuROUWYBUO3x3xlg3f4CsA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pxVuwtKOAXvIKPgc06hrmAEDXSmVAyAaz7t5IB4p1XlBDCXye3iZ85+17t3Dt/4sjHb1utiTRHk6EVeiHmhmXv8dH16XoTs8d2ogwVB3G3WWHeuYzbm1DOkinPA716+acpyLmYeMNyJbm0KWgODH6oisQVKrNMxqC7w+TgUKbho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=u8x6MYx/; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="u8x6MYx/" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-71e8235f0b6so3113389b3a.3 for ; Sun, 10 Nov 2024 07:29:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252574; x=1731857374; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LL88Qd2tMlsmBvcuEyGfQPoq7EBSjG4mmD1oB3CTmL4=; b=u8x6MYx/UtKOMks7NJ+TLYm6DBTL4EOTB7sb6PmubdEYVNejDlMm2Eq76FR3nXWymp r3zVlFa6uMYain7P9bUHme/pWSDRtjdvIRZ5SMtKrWvTifsLiEgWVOJ9FOTpnL2CiGRy 7+j71tC8+NhT3Mx1+LDHq1Eh0iNEvGcTkJ1Kw1FQCPWqDGZhpknvWngSsX3eu91RDTeJ Ssh4TMi06dZ4R0X0frUPK79mIeN1IbzJElZIyguXJHhFqDbPyARWZ4mjBWIDG7/s6Tig aLBvrPBXUgKqT3se0RKHpC/2YT+XO9kdLOabVfuirvLl8LLFRMAAY5gqNp/VvQn7xS3o Gzeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252574; x=1731857374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LL88Qd2tMlsmBvcuEyGfQPoq7EBSjG4mmD1oB3CTmL4=; b=HBeNpfMVTRtAnZurgGnVFEcqIRhKnt/CrbtMFViOQ1lOlturYTQXabz51vX4KWej0D Z8GNDPoc49MwUV1/q6MrdkXQc7l7jxlC/5pUlky1LlUNaEzsiXeB+A9VTsiDOHCtjPJD o9xxYa2F64rGV9K+Jg1rFqJcWi4w2RwG99sHGjcOt25T+Zv4cIiOoi42nDB2vt34tjxs eTa/4SlSTbQ0+voSGIEwg0xssZdTBuzzQYkcXu2F9ky2VDh9993D6WEEIo0VAv0NCKyF uVWziiAiLoSxRzNGOR9vdVWJcIvV6CViwD54lw0JyWBpzzJIHYMWAtLc7kEmrTsnNF0i YdhQ== X-Forwarded-Encrypted: i=1; AJvYcCXKwF4wv8sjm1zBMh2EOKotMeJ8+Zlo55SgrIWywkk6MBhdcCYzwmR1glXkZB4giQcY9lBsvONhJL5Mc1ba@vger.kernel.org X-Gm-Message-State: AOJu0Ywl/bWolE6nUAj/lV9geHICgtQXu00FbZSSK77QKfAkMr3R9h7q iKHja/a43jIhF3SIiqhRdFXiOuQ0bU2ReRyAuoHmJkYfmn9kKc/L673c3dzCoAc= X-Google-Smtp-Source: AGHT+IGDHhrWZvwz3+S2z66PgLKGWEEUzZeIBjMPJzYATyn0eBE8Y9PpfOEANie1fvuwvTvVxNqbng== X-Received: by 2002:a17:90b:38ce:b0:2e2:d15c:1a24 with SMTP id 98e67ed59e1d1-2e9b174113cmr12568464a91.23.1731252574323; Sun, 10 Nov 2024 07:29:34 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:33 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 14/15] xfs: punt uncached write completions to the completion wq Date: Sun, 10 Nov 2024 08:28:06 -0700 Message-ID: <20241110152906.1747545-15-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 They need non-irq context guaranteed, to be able to prune ranges from the page cache. Treat them like unwritten extents and punt them to the completion workqueue. Signed-off-by: Jens Axboe --- fs/xfs/xfs_aops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 559a3a577097..c86fc2b8f344 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -416,9 +416,12 @@ xfs_prepare_ioend( memalloc_nofs_restore(nofs_flag); - /* send ioends that might require a transaction to the completion wq */ + /* + * Send ioends that might require a transaction or need blocking + * context to the completion wq + */ if (xfs_ioend_is_append(ioend) || ioend->io_type == IOMAP_UNWRITTEN || - (ioend->io_flags & IOMAP_F_SHARED)) + (ioend->io_flags & (IOMAP_F_SHARED|IOMAP_F_UNCACHED))) ioend->io_bio.bi_end_io = xfs_end_bio; return status; } From patchwork Sun Nov 10 15:28:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13869943 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E6AB175D26 for ; Sun, 10 Nov 2024 15:29:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252577; cv=none; b=NuhvAeuYRSQV4zkOi3FjqzKQ6NkRC4X1g9vV+hpOr9C7fShkFqo/qzKAOri5RSUmUiQdHZJp0hXoWu48aESgDdPral67fJCt+1E/hI7oNnylc7NiFVdp5jVHXpe4XpNJDOzgipgcTdj5tKnLoyun0ZG0Q0rcxKw2vmeU6XrtR84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731252577; c=relaxed/simple; bh=cP6gr8wADb4EASrEnj9VSH3qpQrBlalS1qET7KiSVew=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WB+wbSOqe6t+g1/Z3U39sj4N8X/nZczC2Nte4BZo4xwup+wePXvzHoWQWSYMYgA8HA8P4r9QGyOnszbETwSCRrz1Wf4QROS5+Rd8LPIbWXETzclIL9aKrc2gIAttJQ//7SReXL9YP1DX1lngbmRpbKuM9m+J1QIMB+Gv1UMhr2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=nNgGyGlD; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="nNgGyGlD" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2e9b55b83d2so1735736a91.3 for ; Sun, 10 Nov 2024 07:29:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731252576; x=1731857376; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ieWCO7ZvnQOESu0rOvKqnHS1oC/GZ0fSczrbrfzlBys=; b=nNgGyGlDhossD3IYkb7cZKYY6OTPoNH8VrLUz3yxK10gKKdjyEc06AeqtI+xzgtn9y asAtORaqqju5OGinAGYQT6QTzMzhZsfzGfhPtkcdO8LPVWc2y8I9eg3YfZCfb5rQbGlR mkCp1zVc1X3KE2JNODRI9ebbUz2QijrYD3qOBu37zA0KFAMiQalNd2sOI1sT7bgWMtAv k536RBeeAvvx9NGQeBeWob19O+jR9mNz0ckCjgjou+XfHhFTddN4vj+/DbuTobM+Tqtu /0HFT+Zkj3nV5Fw7lqZHUFqv6/cdZ3XLJav+YMeCGKWKbdj/If+WJ4KsPVqYrXVPveCb mu0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731252576; x=1731857376; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ieWCO7ZvnQOESu0rOvKqnHS1oC/GZ0fSczrbrfzlBys=; b=mRCg47Y8/qJouT6AaVofrg63ztYZ57j05KKMFfygVKMknyxvPa0eHoGxOk+3c7IiTn i4Mma1/Xptd/CKRpM82p1LRCzxFd/P79Uk92hG0cHl22Qp4JpF40rdgQECoq2CD56bBx IRqqGCDWBtwbHMyEGjihxFfMQ6STEQC/icaZfhooRmq6jbLnsDqSWh0p2omI9oSLLQ+Z a8JvGoQQEuIlJAqA3Q1fp3zyYR3JgJ+kzHWLUqGzSC5gRAHwJk3VkA4Ffp1RL51KCDkS aqVXiQgMiZSgwQU3LWEBhH5QFkQ3Vqj5NU4yDJlAqym/rZZjLiSWt92GdbFpTy8VSn4x 24bA== X-Forwarded-Encrypted: i=1; AJvYcCXVPgxsaOqXWlLNqrybIOAM3QI2cmtmsSmRV9GTliNR142ILyGlHXHh+uOWWgcUl9sZC+iaqIshrtdYqfiG@vger.kernel.org X-Gm-Message-State: AOJu0Yy+WRZ2MoHEjf4GuL0rVIjTCB88YizVx6j6X8j73X5FFQBI3o0z vKezqd6+XZymyw3tf6L9KuLNeg3ZlCRxCuNjhcEAQJtsw+i/fvzD0hFySN54sLc= X-Google-Smtp-Source: AGHT+IHHwm5ckcwW6tzDCyvD/cFUv8RX7EzbVytEaqdSin9tImlfNCqPLYYgjj9DyXwqjBBAmsxOog== X-Received: by 2002:a17:90b:180a:b0:2e2:d434:854c with SMTP id 98e67ed59e1d1-2e9b1655951mr13339967a91.2.1731252575687; Sun, 10 Nov 2024 07:29:35 -0800 (PST) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e99a5f935dsm9940973a91.35.2024.11.10.07.29.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Nov 2024 07:29:35 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, Jens Axboe Subject: [PATCH 15/15] xfs: flag as supporting FOP_UNCACHED Date: Sun, 10 Nov 2024 08:28:07 -0700 Message-ID: <20241110152906.1747545-16-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241110152906.1747545-1-axboe@kernel.dk> References: <20241110152906.1747545-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Read side was already fully supported, for the write side all that's needed now is calling generic_uncached_write() when uncached writes have been submitted. With that, enable the use of RWF_UNCACHED with XFS by flagging support with FOP_UNCACHED. Signed-off-by: Jens Axboe --- fs/xfs/xfs_file.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index b19916b11fd5..1a7f46e13464 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -825,6 +825,7 @@ xfs_file_buffered_write( if (ret > 0) { XFS_STATS_ADD(ip->i_mount, xs_write_bytes, ret); + generic_uncached_write(iocb, ret); /* Handle various SYNC-type writes */ ret = generic_write_sync(iocb, ret); } @@ -1595,7 +1596,8 @@ const struct file_operations xfs_file_operations = { .fadvise = xfs_file_fadvise, .remap_file_range = xfs_file_remap_range, .fop_flags = FOP_MMAP_SYNC | FOP_BUFFER_RASYNC | - FOP_BUFFER_WASYNC | FOP_DIO_PARALLEL_WRITE, + FOP_BUFFER_WASYNC | FOP_DIO_PARALLEL_WRITE | + FOP_UNCACHED, }; const struct file_operations xfs_dir_file_operations = {