From patchwork Tue Dec 3 15:31:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892605 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B37E01F76C7 for ; Tue, 3 Dec 2024 15:32:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239970; cv=none; b=VDk41p6rpIpDKsOyTRtunlkIPxA7le1I82kq7u8QSewZ0WwNQxJCQZJUYD7IHExd66yzEM2PGibFgLbe01qxcSTP03ajcMdDL0cWzLkqfeo2+XuwfyaLYzfPknrofHZIHH/6I+FZ8PGd29bj7OKBL/BSTywm93FH9rASrXUYkPE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239970; c=relaxed/simple; bh=89Ccy3GI5ZUKxFiNR9lyKoxkgOUx6zEEiBrhRigykbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rddwmrrk79piGPTMSiMvxdgg1rRn7PBSdZCZ1LLrC2zJPHbzAiREPcNzDc02uxxbvDGFejdDRpvHpTWcNoZww2nujh0ZNzkUNfpQHx4l2JcV0M26f/os8GD5j2w23lsEphVtKkVKeK4cMD7u7g12HbHo+G7Mu+1+NZiK7m3YI7o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ssdLS1LP; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ssdLS1LP" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3ea55d16d73so2002513b6e.2 for ; Tue, 03 Dec 2024 07:32:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239967; x=1733844767; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=ssdLS1LPBUHL1S31Q32Y/smkGXM0559dx9pYbkyDBWJ7eY6YZD+xY2uqJAfV12RuVW xwtblMjAy2gaeg7xJ3mkFST5v0G+JzIFu+gF7FWes+8Q7MPyarpDkxEWlN68cx39P79p qjZbtmWeQiUaON9nakCBeBVepmGECBc+cSpKmS1pPqYlRb+My0DohaccrNIzgMocBMs0 BERVtEJ35+2veCbgFwsQISDue53j3fIhlrIk9B843TKalWJPW9LaX+5PkTmNnFHdFkKf BSbv+W/kFJrSiXxrJY/NJvfPwSfbmL+9mK/81i1/r5v5p7J86z08tUOY9ObHCH7ng6e9 sroQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239967; x=1733844767; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=b3exVnDMp/IOhHIDjd3iXAsXvMKnv5qXwhAcrafsuM/t2/OcYThj6DzSTKwVido8OP gj/r0jMOiaCkC4uFA2eZRKlQfxpvMg7/rgsbJx7pUjghe3bU/shUAx6/x3jU4VkE6wic uVIiZb08fkP1yyFytenak6i/X+6BCvnsgA2TFnOTbh9XVeULnQgazTB3t4NTfdWRfWFC ILQsCtADa0rrYg3r35YvZ2+vRKIQGHJawK/Ra0KbRp+UpQIeMs8qmsf5vpPBji6HeASU qz1txB/HCe/nR0mWLUcr5UuPgt6tZmTsLOBjT4TgV/4SiS3bU7JSiS+dQuWkx+EHN3zQ AusA== X-Forwarded-Encrypted: i=1; AJvYcCWdC8B6ALR9O0LfPAkyAJ6XPBcX7hVr7XM7YSbAtOKB2UrVlm1AhCKycTd0aSKCzC8wTRTurvJkYEw3kG5X@vger.kernel.org X-Gm-Message-State: AOJu0YwDno7JiUHPutmfbgqgLYnJl17EZ+y0EHpZDlICzulsdd3OyvX9 hoU4vDGDphmp5G8qZHHXVXT7ep7At9m3BF7MLgyClFijjJn6rvfFktqMPo8Hfog= X-Gm-Gg: ASbGnct5Iitw11Su7NUpI3cTMHaX8Ta/pTe+8e4Bg+w9E+hWQ+f0ZMN7ZMRxkSjiAWJ pcLDOwlPdR6hGKgFdBYAM7fo6faI1t1GJi2ve0VIC8FdU+5KRwXh5YDWnmsxJ/pyc/z2XTxb4NK PI7kaGdJANodycjI4RdnI6s+3iCexe/Gsus4rVe1kzYE4DsaWqQg2kFGxidwx2HnYx1yn1Sm4wB T0UbL6z79AvlUXdKeqsTEfNxSDrzd1RoIycFb0U2jYv3aKwgs7m2E9h4oo= X-Google-Smtp-Source: AGHT+IEjGwc7GFJjqyvYWDgJBqgGGBcMfQCEEJJRf9YUZZ1W6BgQFHNlpJWElLeXDmEpw+9XPHnUvQ== X-Received: by 2002:a05:6808:1406:b0:3e6:1ea5:6b30 with SMTP id 5614622812f47-3eae4f9162amr3792647b6e.24.1733239966815; Tue, 03 Dec 2024 07:32:46 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:46 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 01/12] mm/filemap: change filemap_create_folio() to take a struct kiocb Date: Tue, 3 Dec 2024 08:31:37 -0700 Message-ID: <20241203153232.92224-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than pass in both the file and position directly from the kiocb, just take a struct kiocb instead. While doing so, move the ki_flags checking into filemap_create_folio() as well. In preparation for actually needing the kiocb in the function. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7c76a123ba18..898e992039e8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2459,15 +2459,17 @@ static int filemap_update_page(struct kiocb *iocb, return error; } -static int filemap_create_folio(struct file *file, - struct address_space *mapping, loff_t pos, - struct folio_batch *fbatch) +static int filemap_create_folio(struct kiocb *iocb, + struct address_space *mapping, struct folio_batch *fbatch) { struct folio *folio; int error; unsigned int min_order = mapping_min_folio_order(mapping); pgoff_t index; + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) + return -EAGAIN; + folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; @@ -2486,7 +2488,7 @@ static int filemap_create_folio(struct file *file, * well to keep locking rules simple. */ filemap_invalidate_lock_shared(mapping); - index = (pos >> (PAGE_SHIFT + min_order)) << min_order; + index = (iocb->ki_pos >> (PAGE_SHIFT + min_order)) << min_order; error = filemap_add_folio(mapping, folio, index, mapping_gfp_constraint(mapping, GFP_KERNEL)); if (error == -EEXIST) @@ -2494,7 +2496,8 @@ static int filemap_create_folio(struct file *file, if (error) goto error; - error = filemap_read_folio(file, mapping->a_ops->read_folio, folio); + error = filemap_read_folio(iocb->ki_filp, mapping->a_ops->read_folio, + folio); if (error) goto error; @@ -2550,9 +2553,7 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); } if (!folio_batch_count(fbatch)) { - if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) - return -EAGAIN; - err = filemap_create_folio(filp, mapping, iocb->ki_pos, fbatch); + err = filemap_create_folio(iocb, mapping, fbatch); if (err == AOP_TRUNCATED_PAGE) goto retry; return err; From patchwork Tue Dec 3 15:31:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892606 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 432171F9418 for ; Tue, 3 Dec 2024 15:32:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239970; cv=none; b=i42mc3z9nFbbCmTk9Y5T/a8qsbSPAG+nkiSIaDWRsjZ+EFo//luaPvPZeofYZaULB8//nX0fIcFdvqbt6y1ddSEevVW77kQAhgr6SpEEJHyHP53toYamD5W2N6Tu+hKnpBL6Hrzkl8yaAFMFGBGWS93iD1rAvDhrFV+4IdO8Q/o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239970; c=relaxed/simple; bh=8Bb0MfS/J32mJVu/hCpTfSTIXsQUeZKlcznJ/+p92u4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J/FCSpcd7YV6WcaOrcu0z9UnZSDueDqmHrCniihuCCLzdD8yJxx1tPNpAvKrAC31zU6UobWiGgumMGRLLKWNAVyj5IIAohUTa6Vd77iJchTGPkpUAlpikV3yLa6HMESjU+JDf1J8osG5sYblXNJV06yEsb5lwJ7q3OGhBqq0XjU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=vI3+LtaN; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="vI3+LtaN" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3ea47869282so3048698b6e.1 for ; Tue, 03 Dec 2024 07:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239968; x=1733844768; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=vI3+LtaNq/TnQ0w9S52h9H8vdWp5zjIWm4vmECGM4OLUaQkVfjYLWSZeBzwm8RkN+t QZBOBOs37SFN0AXY4lgL8QvsA+LIbBOvJy3jYzYBEf0DFjOW2Or/4MqeT4sUsqcdyKxH krxdvKTFDTUK4+uMvXVKGrP0QdooVKLGOdsjyPFAGiaROrSAHqUKPQqb2Esm7zmu755q L3snlWcVxQRYPSKR05GHn/anLu77M0zcXS+LauovAaCJGM/WjuKY9xah6ct2oTdUJ2vf 7xoXj9DChXxJKIhOnduYaKYh1Aiuwk/HeB70IdkRDyMFoj6lCcMhzdBpqWm16E2K/Wkb URNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239968; x=1733844768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=nRvbfH523Rw3aQpTfdxVUYtuzoa9n8GosVTojJHarC7kY2ERMO1H/ZwIRYtf5MdL8F S0ovucfYjqD3VPeJpdu/xzxqXyHsyjUh9e7IRdrsEzzRLp0sOXm02sG+Hqg8fUaSuR6O MQF2yT5GXddJClnCBoSdSyFFXjfIcKKubHW7Wn0yuUNB1Nsg6V5jG4zZ3zadhUCl5Lsa x/8mX7ZmH27Q/Q0EjnaCKAfyM8gIJsnOEcjrRVj7o4i9rTs2fpydqVKcsIR1N3FYLM8T djcjY27S6ILtVjFaVT6DYldckO4iPxrM1j2hXqTDo9Rm7WWxPMpxnn1uL4x2PM9BHAC/ 9aLQ== X-Forwarded-Encrypted: i=1; AJvYcCU4hfpXAF2x32mFk28ReMErE3Wf1BBIyQmkh0EYPQmtQVNvMqAjupARDxBONpioHuEloX8U167U6ZPxhSfM@vger.kernel.org X-Gm-Message-State: AOJu0Yz2ZMVR/d7X0vts5rrfJxD3SO9tSDH1cdK06ve0wA5/4t5tE2vm cQ2yoDtb3PW55ru3mskwKH0qU3U5SMtOt/Dp4rWVHegwoYZjbmaFHzz0uwmmPzkFnmw4BR3BGxk 3 X-Gm-Gg: ASbGncsajcP2UujNb7A+Upk+1IlW2cybxgjzxS70DkyPf9xb8VTAcHFKBygsA/iWrie 0ZKjs/zrsE+Sx3Tka50gp7PsSJDBEpNqCnfhncHGON4iOazg59bVvolIBz0TGZkul6aliBjzVhq modlQRL2NJi5TJM5aShJ4TOMFHdO2kX7+JCdTYteJPe6lxfrZ7RFi5zLmIWqngGFY6qjeGHEtnu V3RPSSWBjyT5GMF3Yf6tuglynQ/2cSQWo/We2+fYdEkR9wyYAo5PaINkcM= X-Google-Smtp-Source: AGHT+IFPC3QIjfGmFueYpArbFjHsCE/nJs1cTRfjgQkZc7hdX+t3EH5eMgGO4h5vvPNqW5LyAgQfEA== X-Received: by 2002:a05:6808:1454:b0:3ea:45d1:de13 with SMTP id 5614622812f47-3eae499b2b5mr1606169b6e.11.1733239968341; Tue, 03 Dec 2024 07:32:48 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:47 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 02/12] mm/readahead: add folio allocation helper Date: Tue, 3 Dec 2024 08:31:38 -0700 Message-ID: <20241203153232.92224-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just a wrapper around filemap_alloc_folio() for now, but add it in preparation for modifying the folio based on the 'ractl' being passed in. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/readahead.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index 8f1cf599b572..8424bf1b67e2 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -188,6 +188,12 @@ static void read_pages(struct readahead_control *rac) BUG_ON(readahead_count(rac)); } +static struct folio *ractl_alloc_folio(struct readahead_control *ractl, + gfp_t gfp_mask, unsigned int order) +{ + return filemap_alloc_folio(gfp_mask, order); +} + /** * page_cache_ra_unbounded - Start unchecked readahead. * @ractl: Readahead control. @@ -265,8 +271,8 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, continue; } - folio = filemap_alloc_folio(gfp_mask, - mapping_min_folio_order(mapping)); + folio = ractl_alloc_folio(ractl, gfp_mask, + mapping_min_folio_order(mapping)); if (!folio) break; @@ -436,7 +442,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, pgoff_t mark, unsigned int order, gfp_t gfp) { int err; - struct folio *folio = filemap_alloc_folio(gfp, order); + struct folio *folio = ractl_alloc_folio(ractl, gfp, order); if (!folio) return -ENOMEM; @@ -751,7 +757,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; @@ -780,7 +786,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; From patchwork Tue Dec 3 15:31:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892607 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FE961FBC90 for ; Tue, 3 Dec 2024 15:32:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239972; cv=none; b=GrBqgBFQs+yhy2OtHyS0N0h+RXfeCkFKGDJFJxJw29q7ZvkQ/4tpiS+2TYaUjrJrJmZjESblt+ElIbZ+oFDaEjQgZsVrnk4i1BF0ALZR+9NPcYuog7vgW4rgNnrqLLoo+76vdPDSzy9vzz+61m+yZpWz2S2wcf8bafK0OIMVl44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239972; c=relaxed/simple; bh=q5+uRl+k/fn3TorBhDOZ+b9X6VZ/V2geJNnascX9Tuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QdcYAS80V3EluPddSj4dL2DczDCP3UxerEN+NzOCcSrzA/Kx+HpYfjSxZejW4sM5QV4fiRckSNA000TCiOl/rnYDGDK4gUVuDkZI9K8iYbjU5smnCIp1ua0xq8CUihwPg/fPgytzT3H7Ri04b4n61yKSnZSGBtiwazAbXhNtOcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=VcxAfTER; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="VcxAfTER" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3ea33a922afso2780357b6e.3 for ; Tue, 03 Dec 2024 07:32:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239970; x=1733844770; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=VcxAfTERfrNCIjV4y56ah0/NIhjSnwZk+jEWqkNNz9Hga7fwuUrtk8VHECMIOKYbVL lP+1zgU68BxKo9y3CM0lkavVEVfY0mNlIKEMhg+OsemQxjokI3iXWRiamfutr66xSjgh MCQ1iJ+fWAqoKWCPNFG3jGqEgU3VLkSPeGFEjbRb2ATyMOrCZpztKXDftsKZiRzzVbDB 4PygPEUE0cYiNgP2yH/zNZI/JJi2VmviFY+rgGzyQaWZfbs1u3ukrWOdYkaASV6u0HZP JBAKdEucpfgW8VqPv2uAv8mKGuD7KR5bxdyoqP3S3xEnZMUHgCpLs5Lw5qE8R7UI41Rf zYGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239970; x=1733844770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=TvjbYcLgovSCROtQ/E0I/tMLwnXv2RSyc1ummdRL/C1KTojbgPQ/mSX/QeDPTtiQVd qL+GWQmUI7YNvidNRh2GusPag3QO1tePUtncP16E4tAV7JDYUWY54IVW/h9Bb8AD4GYb WOn7subTbBCMPTH440mxiHPlaSsrzFvaFLd5x3FM4jSuM6pfEoxn46KlZtrOy2lZwT2/ +5rCvdKMdOMxWH6SEHWzatSlXfH97gKO3LLaNRc/ObNsMi0vm95ojjnZELXG26wcKKIi /jGrbSDQbW2Sbm6vOWOxeSFOL+Muj853jLw83+OPsQ7pYGHr4qNSOiaNL5oyyOPQSjxu FF3A== X-Forwarded-Encrypted: i=1; AJvYcCUTMd3d/6+G9AXi/IrtRCgk4GYh9olSMkvNTDITHTxht48pTBQzVATgIlkSapLltfg9MRHHqu0OmWTX0G05@vger.kernel.org X-Gm-Message-State: AOJu0YwYkope8JHeKcv9+u/kZC9Jp+aTMBm48hKYDr1jaVdpRvzWvP/d johSsnUAm1eEmDsnDGZqlsT5Hryj0AB/QUB4fBJ1CLvB65OB45GJTI+8V0LhJZQ= X-Gm-Gg: ASbGnct+KYkT8UEXqbKIPVTLWG6GhzJEWBTCQthNeDMJy7VOW+pvnXNk341A6/zBlOh +5tchOSMOxitJMNLPVtFFkl1v96Kd6JREbEUi05ShAFCZuHykgV4DX70NfhH10mxApD8/arX+nm EtsFBG8T6WgT8+kUZNhm/aHngs3FsPdciDmz7UrrbGaqjlVjfw1AAW1tKY8QvP081Lu98weFySu YAaQ3lyGL6S+52kPoOJn3xsC9HMAb1ySJu28maX1wCRrncRR94H/8qmPSM= X-Google-Smtp-Source: AGHT+IF59+hbl1oxKnagTF4XdkSvu7+brogdJGlmHp3dV6C6q4drpkxZSCaanT7IxLge7RXlKXRl5g== X-Received: by 2002:a05:6808:1795:b0:3e7:9e07:432b with SMTP id 5614622812f47-3eae505e041mr2729493b6e.29.1733239969773; Tue, 03 Dec 2024 07:32:49 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:49 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 03/12] mm: add PG_uncached page flag Date: Tue, 3 Dec 2024 08:31:39 -0700 Message-ID: <20241203153232.92224-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed. Signed-off-by: Jens Axboe --- include/linux/page-flags.h | 5 +++++ include/trace/events/mmflags.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 2220bfec278e..14346fa2470f 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -110,6 +110,7 @@ enum pageflags { PG_reclaim, /* To be reclaimed asap */ PG_swapbacked, /* Page is backed by RAM/swap */ PG_unevictable, /* Page is "unevictable" */ + PG_uncached, /* uncached read/write IO */ #ifdef CONFIG_MMU PG_mlocked, /* Page is vma mlocked */ #endif @@ -562,6 +563,10 @@ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) FOLIO_FLAG(readahead, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(readahead, FOLIO_HEAD_PAGE) +FOLIO_FLAG(uncached, FOLIO_HEAD_PAGE) + FOLIO_TEST_CLEAR_FLAG(uncached, FOLIO_HEAD_PAGE) + __FOLIO_SET_FLAG(uncached, FOLIO_HEAD_PAGE) + #ifdef CONFIG_HIGHMEM /* * Must use a macro here due to header dependency issues. page_zone() is not diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..b60057284102 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -116,7 +116,8 @@ DEF_PAGEFLAG_NAME(head), \ DEF_PAGEFLAG_NAME(reclaim), \ DEF_PAGEFLAG_NAME(swapbacked), \ - DEF_PAGEFLAG_NAME(unevictable) \ + DEF_PAGEFLAG_NAME(unevictable), \ + DEF_PAGEFLAG_NAME(uncached) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ From patchwork Tue Dec 3 15:31:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892608 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 464AC1FC0EC for ; Tue, 3 Dec 2024 15:32:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239973; cv=none; b=edazf/fPlUh4A+3UhTURDqUbhgGVPqztZVGp4Py8zGdOhTVP1sCm7cBhmsChl/B8XT8BFaaajkMBPB00a4EWvAMOZz4XX3axy5TdPzY718+C22/uYUKyVZsi1LFHRFGXvKwdKr2re3p8PdQvat8wxt9der6oomnJ/qu0USi08pc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239973; c=relaxed/simple; bh=ZgRCLG7Q7PSJRuEf4S9CorJsUog68v6zrN1kia1bNYk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=miuAwY8KODoywbwaVM4P/C62NkSZ+XgBKJh6BKtecDMhVPkxlBsz0DWxP2MVHxWVpfx/EqdDJl94cGCqi5meMzD2FHdJxpwMnfpNxMf8BrRf7A76bi603qPx8rnfy2wYTtd6i4xx7QEdDEXS9jHUtieNu1FL07+nUtdHMjUwFoA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=HlbZFfOb; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="HlbZFfOb" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3ea4acd3ed2so2426260b6e.2 for ; Tue, 03 Dec 2024 07:32:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239971; x=1733844771; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=HlbZFfObU5xPz5v+qTCn8lR2qgozqdYGlOM6PCCDOyas9I5SUWycbDVa/ntlFyBsP7 Tacye40HugiN1g7Ivs1sfvOw6hcUV3WWcMJIa4YzOCvQSZOgiYlXKyqJMkv2bJ2jx6Mp rFeFWrfv8hwWjtYJDXSIazfJvDE3BP9wj7ZW08hxgjnTWTueZ8F3F7q2OURNwyjlcId6 +kyiwEtLqI7qmrkhimyRvSTNrFnjKXMkBKJMyFWkvPnoZZO2uMtWhL+VlgUSCNaNLTib N1p3geFG+eZWKu4UwIl2bfFNsZC8pmu4MC6DYRvW1kvVQZ5my8P3Xc64YrxHhdW/27Qe mUxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239971; x=1733844771; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=PE/dr5VD8u12qMFLAls9P+PEmR4iMXGk5a3c1Q7LBGVW+58pPR2MVW8N35Gck+yvrX /RGhH369fDOcWbsKuB3k3dnmA9Uo7Qp6L8OPWkRaZJsFhpyfW4z7ZOupZPI+z3DyNNmG DG/oYslGF41+5uAEMLahkeCE4SWjb/OnCkYksZ5BfqTiBE0js2dDmfM1+DBvGEDDpmUZ PK+YA3fRXcgxF/RV8PonLrbtdbWPnpcZlPuim74W+OX68LY88UyL9fBfwi5f8lHJqYPY IZ+Dnl7lkdnbmADNYH38+pv6bIlfLREjALJ8m84rVOFXiWgupmzBBomtuKb0be/xG9s2 D8Zw== X-Forwarded-Encrypted: i=1; AJvYcCVdvsQA+vUwUbMltozKPbBbq196RZozoMTNjXZyy8d0GpXI9QY9RqKuk0+76Nabjo/iiFK5S6Das9xpUvCa@vger.kernel.org X-Gm-Message-State: AOJu0Yz8CXTGEloNYU3QzKs81yWpxKypYXX26N1jV6Iwn3GVxWbjcN8i yn2X4Y0iCIcO20N3GahlYsHFcptEsdZYoR1hBDVyHeJmjZ7H4BKXrS27QFM3dfc= X-Gm-Gg: ASbGnctpmfUjfZXysX/ElA/RNiLbsWSW/ZjEZqX1iCVT1+b5mhU8+RgUCrkIKfwBPYZ vRQPvNrEOO6YyitmtQYs4CL71XEnKrZn5/bom0PRFG4anVNAsL6BaavzdN7fo+Fg0EgZ6goOcLq nPmNWjEh05HkkBp3wQMrjQ6d0/sW1Ma5Cwrx9wNktLCGEU+lNNOBRaXz4TaTTq4xl1ZqBVF+F72 8j6iGpHfP0mZd94dmm2BkoIUWCFvgk6HDAutn137Oq5WJM6HwQZSyFHI44= X-Google-Smtp-Source: AGHT+IF9UmPrglgGWAW3JFFN5BU2DOVhhB7roaz8QIV4CUJFyvDEFcQOUcjiZ4xJCcSYYgdgH9P94g== X-Received: by 2002:a05:6808:1405:b0:3e0:c13:9837 with SMTP id 5614622812f47-3eaf041efccmr453615b6e.37.1733239971292; Tue, 03 Dec 2024 07:32:51 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:50 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 04/12] mm/readahead: add readahead_control->uncached member Date: Tue, 3 Dec 2024 08:31:40 -0700 Message-ID: <20241203153232.92224-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If ractl->uncached is set to true, then folios created are marked as uncached as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 1 + mm/readahead.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index bcf0865a38ae..72b03b37c265 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1353,6 +1353,7 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool uncached; bool _workingset; unsigned long _pflags; }; diff --git a/mm/readahead.c b/mm/readahead.c index 8424bf1b67e2..33a2d0feae14 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -191,7 +191,13 @@ static void read_pages(struct readahead_control *rac) static struct folio *ractl_alloc_folio(struct readahead_control *ractl, gfp_t gfp_mask, unsigned int order) { - return filemap_alloc_folio(gfp_mask, order); + struct folio *folio; + + folio = filemap_alloc_folio(gfp_mask, order); + if (folio && ractl->uncached) + __folio_set_uncached(folio); + + return folio; } /** From patchwork Tue Dec 3 15:31:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892609 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E3C81FCF41 for ; Tue, 3 Dec 2024 15:32:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239975; cv=none; b=NC4gVMEXzjyT91Hlc87ldgwFCe7BNgM4rb5bhsfcMOOCE0urj2O8E01MlgJ4YCtoJ7v3Gbe7XV1xCnMkNSh46J0PIJ2ftY+/KbBB5oBj+qytuLlAIcArX65n7SQCRmQYVpWSNLsr0OKAc/jkCQDA+AQlUYD/qxjNkouBYsiYifY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239975; c=relaxed/simple; bh=SQI+ulnw+CRtDMqT4Q5a3RM/yVgI6AvOV11bMjB5isk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eU48wsTepPVK00X+mFMTjARA33zBy6LsruyjEV7tBsYfLK3wlE7eCiWiR5plTx/gnpyuk4ur7y8z/1Lj8Z7DgR0oNlikkftFPO5sdS5Dk6WOC0OJQJr4Jw4XKzxxxTcPZOyFy7dE+IwMM0vipjoqMDEc5gSMnSbMaq8ZOIX3LU8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=RuMwze1b; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="RuMwze1b" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3e786167712so2728080b6e.3 for ; Tue, 03 Dec 2024 07:32:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239973; x=1733844773; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=RuMwze1bubbkloeBcqqhIIKdhgXrelfY/tzbUVkN72jNmqL1Pu1EcLkjbX/221s3Ec gC2rfqKx9VyjuhWcbokwU5l6TMtPQ7D1KDHv31FBP0RgjchCEVvWU4qS9CUYNHcaScAm iAl6bJwhqxTCuPouyfBG9Yw44cwPJhpJjgEZ282vfGnD2YOlL83R1ZlgCiU2f+AhwtcY ViAk7w1NRl8Q+6m/mMvDaeDm5OGC8TyhI78JxVnjwC/4fwBcfY/FJNdBKtQONWx5YzTb t6mS9rkO5MPiEZghVzftLG2OUnEoZ78TmE80CO/jUbE8N/RArWNWGqmfE85hBajtwZjY W0DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239973; x=1733844773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=R3liiuUae835+GsfHOTMMDzkix8CFUphSOlAPiTzNmnoVwqhjMNEeMZ4zgtJbPujCL n64NuldaOgUyfYp4WpJSzHGsP+Fc+CuMHNJINSwmA3E4U4DaeEL5/2d1Iya03NFpxZwD KwUvXd87EORyIoSD4RAwsKiINkxMallfiFLWdHVx+GzqqgLjc1ojYNxQXDbfGp8m47xI zXbHCztqGjs2TbpfRRkukrQ6jSKX9e09bVnqUyImUQeeBwiH8OmhcHIazw0+d/wvkgry L1MC3TuwhmfQZTUv6Cxsu1l6el51i8dHty6JwzAB8Ntq4odY+hjxJer+RNR2nVIJJlJ0 LEbQ== X-Forwarded-Encrypted: i=1; AJvYcCU4VZHlOWv1bR+iRfAa0up/acGAVFiwErr+WDLcpNQJ31tkW0PQ8hqhb5szZxBdfM11pCcL/IKWiUoJZM81@vger.kernel.org X-Gm-Message-State: AOJu0YwZPWrbX0tAKctHfe7bLqS0zR8r3PwORl0X3IikW2556zz1CulF CQ2olic0sXzcK5FxNvt0bcTaLwX4WYZCtMqJU4ccEkCd6zPxd6ZZjM0Ikg/V3WY= X-Gm-Gg: ASbGncv6k0edUD7+CK1wj1WSYkO4mzRT2iJxDQjkGCvfrj0WmjT79ZO2Ub5UtCn5aXS a/YZLXtLSLUndfLiCHRVB1RfwPyZNwKIQqUeyVVVQnkoR+BBJwg6Toze7b1WX9lDt8FvFdw1aUB 7oLsOZp57h0hPRZdxf+asw140EvHet06KLbjcgjxljtf0pt/2v95q91LAQiLvkXO8nJcHirAml8 X4gjlhY7dQJMrg0q7Dkh4P3ILyVAheCBB6kEScKajDmAMOflHzViPo+Ukg= X-Google-Smtp-Source: AGHT+IGEqY5mPPeclzVfJP1sqwpK2NEo6ZfmI2zg3SEWdjZlSMpJ+brY6rhQdeJiLlo5jFiEsUF9Rw== X-Received: by 2002:a05:6808:1795:b0:3ea:4c23:daf5 with SMTP id 5614622812f47-3eae4ecb2camr3172373b6e.8.1733239972824; Tue, 03 Dec 2024 07:32:52 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:52 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 05/12] mm/filemap: use page_cache_sync_ra() to kick off read-ahead Date: Tue, 3 Dec 2024 08:31:41 -0700 Message-ID: <20241203153232.92224-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than use the page_cache_sync_readahead() helper, define our own ractl and use page_cache_sync_ra() directly. In preparation for needing to modify ractl inside filemap_get_pages(). No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 898e992039e8..dd3042de8038 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2527,7 +2527,6 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; - struct file_ra_state *ra = &filp->f_ra; pgoff_t index = iocb->ki_pos >> PAGE_SHIFT; pgoff_t last_index; struct folio *folio; @@ -2542,12 +2541,13 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); if (!folio_batch_count(fbatch)) { + DEFINE_READAHEAD(ractl, filp, &filp->f_ra, mapping, index); + if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); - page_cache_sync_readahead(mapping, ra, filp, index, - last_index - index); + page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); filemap_get_read_batch(mapping, index, last_index - 1, fbatch); From patchwork Tue Dec 3 15:31:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892610 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 313481FECCF for ; Tue, 3 Dec 2024 15:32:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239976; cv=none; b=nYo9CUk4MC33YGcb29zC3tvleTJU7KdHKubMQm1rq/+qTOhWaWmqK6o0bIismpgySj4MBrHLybOuy1irLVpNAIbgMu4T0QrpREKHRGKQt4P1Z31zDMkeviTAmXOXID1kR4DF53XYydQgwQfu/Oo8k04QJ49hDZnHOQf0ndJwjxA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239976; c=relaxed/simple; bh=YX3XCyHjYzyVsTXZV7qYbvuii/Uh6uyZoNv1uBKQubM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AteHWEhSfzFO7hH00JXM1zSmN+5s7S5n0KEzDq+Bxix/p2q9+3NgPADmooBkzPL0dbokUCXkF4Aij1X/zTREwjJSCKV1SXkB/mJfntyzotb9kW225zCkDQrHi6mFzXJoNCKpb8JBr5nE9twBCQDQsgDJCZLqjhlVGC+C38c5Qag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=0jbp1Iuu; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="0jbp1Iuu" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3ea369fdb0cso3320120b6e.3 for ; Tue, 03 Dec 2024 07:32:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239974; x=1733844774; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=0jbp1Iuur9naulsin3kmi+SybTDNMm5eTWmCVsvE0jKeqIrtgcbOc1lJt0ws5oRJuq Sjv3tz5r650N9b6YtmMmXTU/yhqB1dKl7xEj6mVG/KngfILqIJIOFzd9YlgGmSkU+gyK TeUfX5VOGwkJBc9Ycn3m5ZfdCG3bRYGBofsi0clxloal7RcEbypUuH3TDPa0pGqGOFC8 V86km6rLFWOz0gKPgtUnNMOnEcYsJ2UdA6endKWApYrCMpd21bSCCux3aOaEdnldy/vc MJRbv7eFoXmkwI/YMFMwW5fMiy+E8DwSYOvkaGYfcOfUO69SCaIXXsO1QEulBG4c9lpk mT4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239974; x=1733844774; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=BY/kmXOpVHhRWVnFSRQ/pxUF4+4oMP0/B3eD+6Svz94W+RU3tPE7Ns5VRrDBNfwTFc CHcKDz9Lwq9nPlq3giURj2beFXT7TvXzlxloFvcBZHACS8JKpIcHWeU522IueJvAMk8y X9ckhjJeZWrOu9e2UDiQUmGOFDkSWabNwdZNIaSRPUiP86H6rKuVRak5dYch0qXoewzR tnRdOH9yyApFAEhV2AJ8TrC7pN+D2Ob2J7ybZlXnjoSu+4Ilvo4mLwlLIO7eZIQYzqO6 WAIijOSFaMEFKXDrNYigN/vTdUnRHX0yTQEfphTzO2KFsSssN3wjeKUpqr1gQ+FbywG9 6X0A== X-Forwarded-Encrypted: i=1; AJvYcCUcag+TYhwD2szZn4NUF8qtwXSXB+l+oQreBskbxMrx+blKTYbPt3VGqujqErGaVq2vGhoyxze5EFDQcVra@vger.kernel.org X-Gm-Message-State: AOJu0YyRMUSXp5Umj4i004pKlc2SXnUVVU/vzaAjqeARiOGk9AZ/6qn4 bNxnypHsgJmH75krsqvSk0Dm17fzMp+v4s17InY+gl21bMlwD5NdVHqo2jVGnq8= X-Gm-Gg: ASbGnctarApU3ZiO2Nqc+wLzz8tE/aujmTtkFMeGADv0JGq7hPxm2n5NAUDKfDFGZzN tnj3fQWaR93X2wdI6K1RCb2A4t+aJ20HHFwhthRFYl6bpC6gfAciTfrbMrMg7yJNYBm4HuVzsdR YF2BptTopTQU4DDq3FW3NlGVj3kejVfn/S2ZROckv3bwVjCkb3pxqYOssRAK/pw6xD/+NnD71co lLAwtrq4fM/cjWAm9NsGdx0WkZcrMNDb+xgJyojv7ta0vuNwysZPzlhviM= X-Google-Smtp-Source: AGHT+IGHTQ5LeGksRhzqMID0Kqh5YSoxN8Gylam5Ojf1AQcflOZOTzp2gKozYR7Jx08uJ7hl5NcXoA== X-Received: by 2002:a05:6808:3843:b0:3e8:1f5d:b7f8 with SMTP id 5614622812f47-3eae4edadc4mr3375768b6e.1.1733239974385; Tue, 03 Dec 2024 07:32:54 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:53 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 06/12] mm/truncate: add folio_unmap_invalidate() helper Date: Tue, 3 Dec 2024 08:31:42 -0700 Message-ID: <20241203153232.92224-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a folio_unmap_invalidate() helper, which unmaps and invalidates a given folio. The caller must already have locked the folio. Use this new helper in invalidate_inode_pages2_range(), rather than duplicate the code there. In preparation for using this elsewhere as well, have it take a gfp_t mask rather than assume GFP_KERNEL is the right choice. This bubbles back to invalidate_complete_folio2() as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/truncate.c | 35 ++++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 72b03b37c265..f2d49dccb7c1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -34,6 +34,8 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); int filemap_invalidate_pages(struct address_space *mapping, loff_t pos, loff_t end, bool nowait); +int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, + gfp_t gfp); int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/truncate.c b/mm/truncate.c index 7c304d2f0052..c1dfddb1122a 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -533,12 +533,12 @@ EXPORT_SYMBOL(invalidate_mapping_pages); * sitting in the folio_add_lru() caches. */ static int invalidate_complete_folio2(struct address_space *mapping, - struct folio *folio) + struct folio *folio, gfp_t gfp_mask) { if (folio->mapping != mapping) return 0; - if (!filemap_release_folio(folio, GFP_KERNEL)) + if (!filemap_release_folio(folio, gfp_mask)) return 0; spin_lock(&mapping->host->i_lock); @@ -570,6 +570,25 @@ static int folio_launder(struct address_space *mapping, struct folio *folio) return mapping->a_ops->launder_folio(folio); } +int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, + gfp_t gfp) +{ + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + + if (folio_test_dirty(folio)) + return 0; + if (folio_mapped(folio)) + unmap_mapping_folio(folio); + BUG_ON(folio_mapped(folio)); + + ret = folio_launder(mapping, folio); + if (!ret && !invalidate_complete_folio2(mapping, folio, gfp)) + return -EBUSY; + return ret; +} + /** * invalidate_inode_pages2_range - remove range of pages from an address_space * @mapping: the address_space @@ -629,18 +648,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, folio_unlock(folio); continue; } - VM_BUG_ON_FOLIO(!folio_contains(folio, indices[i]), folio); folio_wait_writeback(folio); - - if (folio_mapped(folio)) - unmap_mapping_folio(folio); - BUG_ON(folio_mapped(folio)); - - ret2 = folio_launder(mapping, folio); - if (ret2 == 0) { - if (!invalidate_complete_folio2(mapping, folio)) - ret2 = -EBUSY; - } + ret2 = folio_unmap_invalidate(mapping, folio, GFP_KERNEL); if (ret2 < 0) ret = ret2; folio_unlock(folio); From patchwork Tue Dec 3 15:31:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892611 Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD6DA20371A for ; Tue, 3 Dec 2024 15:32:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239978; cv=none; b=Lr5yol3b4O7vsxgjW6tknOMhxUW9JNsXyWYTD/fXYNXyY0FVPVoJ+TdwirC5jsxOvzqp/nG4idET8e7bNtoz16fWfmG4dLm24Dyi6iB4ZTc8KqVSIsvz/Dw/iw1T8OVRPMvMYMK1dOG9m7ehDFSHXLweAGsWz3Y3dUmgDCVj5dk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239978; c=relaxed/simple; bh=mhIaz7b2lyC1oOrpy7fcg+5AljdReMhMByTLtm2p4EM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qP/CODif2lgBncwAKDF7IaDiTILzlu5ZUA5sHYxrUmq1rZvSUdMA2JDhSX1Hy/KPHunXWd3JjSWooLpTQ87gd6IHGot8d7v16XROuB9cFwozgUIKlQnqY8ECN6QgDSYZnqLz7HpVqgdUjPQqHsx81SyYSrJtjWX3++FW0WX0t2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=fe79W8Nh; arc=none smtp.client-ip=209.85.167.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="fe79W8Nh" Received: by mail-oi1-f180.google.com with SMTP id 5614622812f47-3ea5a7a5e48so2650324b6e.0 for ; Tue, 03 Dec 2024 07:32:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239976; x=1733844776; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=fe79W8Nht1SNccoLINB3gun47a42Nyg7qCWlAyUHn7wod5WZJVvfj5epG7rdRBk3EL IHI6Fy0gmAc/hoo/Rbj2tnqec0JQnK3TI8eslyBHs0GimM6Ve/bXBH8t1EsM0F/r4nfY H4Z6eo24b7YiLr6tn1N0IfuVI7ti0ebkN27qDrvk17HP0T/yfIErOWX8/joQS+PO8D1K XKVccScUJYUVpHgO01Ep1qUIMQLv5tSbIZAIOTweoUPn82mbpQd8wMNBlhJx/ax0jvx7 +36yZ/NEp7WyKD9QrNsMJu9mb7b1Ypg4OQD0Xftl71coTjK/xpubRFXvsgbVSogeKAJY L2kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239976; x=1733844776; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=nNtChhc1sI8QGpUsogKOYI7cIbaAXLIe8ArMaTKb2rn20PLZEwQoTiD9ReZoWYgN0i hjrrhTbjXPdHjTs28c4LVfZg03PgD5fbJchaDR+fVzG95DAJm3L7bVHG3k2D8dcTyEY4 jaDrOOIVd59NR3vyariqu0eRCNSdsiGQ+w3NHRHqXBaP+MxNdL72qRWyQIxjuE/uW9XN qSsYdKb2Wc0tV+9XE86jqxQJCRP2WtTAAF3bxKi6Kv1LDWaacBO2s9sbh5MBUiMAionq huxGqM0H9mmiabd1AdQZglAzxtf0cyX0QtJYx6HfY5uXSbyoH+8I2YXm+5et+luSEGP9 F9xg== X-Forwarded-Encrypted: i=1; AJvYcCXaKOKiDqKQFjeeV1B1FMmtD6ZUWU6wWUdXEVH9y+TaBuhA6jE0WKJ98haWRUX3BM4XO8v1hVK5B5uBZvf+@vger.kernel.org X-Gm-Message-State: AOJu0YwTZSilMP7N5VLBsTB4WqCyRlwAjn4mGGETlQHvCeWcYOrS+UGn Q/saqZsyUiNgJpADuUi+hhDkma7idCeBIzx5+Y+gPoyqgQpj/n1JoZw4MCM3uwM= X-Gm-Gg: ASbGncte4iOrszwXUKYPpEqiG2FfYNHTuFDRA7CKp1lV+BPw5VUfgH4/R/lYqnaINLE lzqh8+rdEF4fahUN6Zon1H01pcrGcWtzmmAm2FQOxcq1c2Z48fJG1cvLx06rNl1ZU9uEGhseHN/ UGS2afIsX8VlsTSchWI+gnfaSGKyLadNdJCjF70Zo/iOtmfKAsGt1gPFLA2/WrbQDXGRVEWYqjB xYaNSsuBjOyplySxrmLi+0sk4cYSXjl2vDEl2/vdWh4c8R2pSaSRbLyMPs= X-Google-Smtp-Source: AGHT+IHMJnWLWXZesvRG8H115i2+SnEyGT/XbMmAgLxKcYoNaa8g7OGxqlE3KOV7Yd59LM9TrdKc0w== X-Received: by 2002:a05:6808:152c:b0:3e6:14a6:4288 with SMTP id 5614622812f47-3eae4f366f8mr2926476b6e.11.1733239975822; Tue, 03 Dec 2024 07:32:55 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:55 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 07/12] fs: add RWF_UNCACHED iocb and FOP_UNCACHED file_operations flag Date: Tue, 3 Dec 2024 08:31:43 -0700 Message-ID: <20241203153232.92224-9-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If a file system supports uncached buffered IO, it may set FOP_UNCACHED and enable RWF_UNCACHED. If RWF_UNCACHED is attempted without the file system supporting it, it'll get errored with -EOPNOTSUPP. Signed-off-by: Jens Axboe --- include/linux/fs.h | 14 +++++++++++++- include/uapi/linux/fs.h | 6 +++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 7e29433c5ecc..b64a78582f06 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -322,6 +322,7 @@ struct readahead_control; #define IOCB_NOWAIT (__force int) RWF_NOWAIT #define IOCB_APPEND (__force int) RWF_APPEND #define IOCB_ATOMIC (__force int) RWF_ATOMIC +#define IOCB_UNCACHED (__force int) RWF_UNCACHED /* non-RWF related bits - start at 16 */ #define IOCB_EVENTFD (1 << 16) @@ -356,7 +357,8 @@ struct readahead_control; { IOCB_SYNC, "SYNC" }, \ { IOCB_NOWAIT, "NOWAIT" }, \ { IOCB_APPEND, "APPEND" }, \ - { IOCB_ATOMIC, "ATOMIC"}, \ + { IOCB_ATOMIC, "ATOMIC" }, \ + { IOCB_UNCACHED, "UNCACHED" }, \ { IOCB_EVENTFD, "EVENTFD"}, \ { IOCB_DIRECT, "DIRECT" }, \ { IOCB_WRITE, "WRITE" }, \ @@ -2127,6 +2129,8 @@ struct file_operations { #define FOP_UNSIGNED_OFFSET ((__force fop_flags_t)(1 << 5)) /* Supports asynchronous lock callbacks */ #define FOP_ASYNC_LOCK ((__force fop_flags_t)(1 << 6)) +/* File system supports uncached read/write buffered IO */ +#define FOP_UNCACHED ((__force fop_flags_t)(1 << 7)) /* Wrap a directory iterator that needs exclusive inode access */ int wrap_directory_iterator(struct file *, struct dir_context *, @@ -3614,6 +3618,14 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, if (!(ki->ki_filp->f_mode & FMODE_CAN_ATOMIC_WRITE)) return -EOPNOTSUPP; } + if (flags & RWF_UNCACHED) { + /* file system must support it */ + if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED)) + return -EOPNOTSUPP; + /* DAX mappings not supported */ + if (IS_DAX(ki->ki_filp->f_mapping->host)) + return -EOPNOTSUPP; + } kiocb_flags |= (__force int) (flags & RWF_SUPPORTED); if (flags & RWF_SYNC) kiocb_flags |= IOCB_DSYNC; diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..dc77cd8ae1a3 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -332,9 +332,13 @@ typedef int __bitwise __kernel_rwf_t; /* Atomic Write */ #define RWF_ATOMIC ((__force __kernel_rwf_t)0x00000040) +/* buffered IO that drops the cache after reading or writing data */ +#define RWF_UNCACHED ((__force __kernel_rwf_t)0x00000080) + /* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC) + RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\ + RWF_UNCACHED) #define PROCFS_IOCTL_MAGIC 'f' From patchwork Tue Dec 3 15:31:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892612 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B8D6204F87 for ; Tue, 3 Dec 2024 15:32:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239979; cv=none; b=oXNovNeLgMcDZetsUZi8wE/5cOts8AAoPgc7SUxO/eaKL9RGqV/XXlDKAaCtAhhll0I48OpcNlGgdglwlHdKxQ/JYZ5FQFWxFdPnlshWfD04hB+7y99Cq25gDa0WAynb/hZutUiaAvDKiziP42XabYYBHESbvCQvgYspCHKEbmE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239979; c=relaxed/simple; bh=LUYFIqzYiNC8iDjcYjzRM/NzCPA2BcvlCTGVyApeMyI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o08FJmCBs/4IfILxipcOgenysBz9Gh0DhKTOMNHPKx7HEjYHgyuFNaPEkSSQtSYz5LlK5stX2J8fmRSgu0lV5GP+F1GUz0+896s/cD4PV5IyeNxFcxouFcpCq5PI/z9jTzDoPjF36DBVqhMVMqc1mNX+0aLs7gnqq2Mdh94df7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=U5y+hE8A; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="U5y+hE8A" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3ea3cc9a5ddso2808066b6e.3 for ; Tue, 03 Dec 2024 07:32:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239977; x=1733844777; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=U5y+hE8AlQ5RNTn1wvZKs/tYic8sjv4/RNNQaa6i4OIGS/hkyTlJsqcHVHm8gQKasG uQc1yxKexzBJUXxKM6+qwa7sl2c2c/eqF2yvp1x0Nfuz9seX1h1j4qsTluEBwxqdisVl S7KI2OpHt0VItCdaP1cnGq0lhnG2DxcXu9P8ClotpHeJcN5O4o9JUN9vgG6/hI9BDfza 2h+ylaZur13T8T8Jd+2SngaZkG0PWAYuAS7me3Xq4IomLG+cOlx1rM52I5pW/3AuauG0 bMfvd2DtFVfmc9lJazpc2CV9UeS2n3WzBba1I7DCHT9wNrfT9XBCYQZAZkRu51JmZSHh h9fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239977; x=1733844777; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=AZ1v/lwSrJN2ONLjvApUKS2pqBd3Fr+cRg1ynRkTudSdfWX/oxyKZjqn6UIQGOz9Hg n8lCSV7SOXhYRaPAWxus9t8n44jb6TQXp8OV0wBxp05s8vPQp4iCbW4AedaZFkWb0Lpe XVIKAQDGmAvIXHVHo3nmFVxyw6aTPoEZkYDQxC/lO7xfLYOf9SKTRAtH/BpnS50oltl1 YCSNi1Wb/4467NNmqhoQsIoAlgd3x6fN7SDsMKZtrrm9uDSJ1Qk1GwlOf6IBgNfaLf0x mP36b82bI7QFFBMbCUru8KVux553cLjuhowAIqzaDwVicmLRLgzjS8oFbIzyYpxTcukU 94wg== X-Forwarded-Encrypted: i=1; AJvYcCVOYatbOFJA2tIRWdcrcWPtk7FtroPkpdMn/SzY/qVp11R/KlAH6L6b3qfOuip2PgIPWPK4GtLembrpHSi4@vger.kernel.org X-Gm-Message-State: AOJu0YzNiwPq+ldCqIc4pQfz6OwWZ9JXWQRCsfOgzn5sv711JEZvVMvC Tda/XBkU5p+zXH/etABCamwj1tQAFyzL5OmruBxzFEpTjYrZwIYNJZfmQ0QqeU4= X-Gm-Gg: ASbGncuEeVpSbVzfg+nvzFLCaq4eW8lL+LORSDjQ21PPxKzEPcgHuhUJRbpSDm8wnIk aONmcBCufl0S0rPEV40tPpqntnjZv+kNzhGdrjvUOL/E/Tn1PgHCMx3AFZLNwyV1mb9KJOtEKQA QbADKv8SO8MDqp9RfLjCy7BOiVKul6IOdHHUVe92yqvMxxXRO/f2lyglN2b7ftqhOg0ThiZPnys 3Klrc0afM0xDIu/wtmqg5ZTuGyAqGyUk8MijxwuvJcH2mGdv6V8eDkf1hw= X-Google-Smtp-Source: AGHT+IEn2a1k9EtjOpfVet1ysm6MWr9jkiUyE2X2SH8cUoAjtJnT3D5sVQSXdc6UVYtxwSGjKa8I4g== X-Received: by 2002:a05:6808:2e4e:b0:3ea:aa8a:c115 with SMTP id 5614622812f47-3eae4f8c2c0mr3705224b6e.21.1733239977325; Tue, 03 Dec 2024 07:32:57 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:56 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 08/12] mm/filemap: add read support for RWF_UNCACHED Date: Tue, 3 Dec 2024 08:31:44 -0700 Message-ID: <20241203153232.92224-10-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add RWF_UNCACHED as a read operation flag, which means that any data read wil be removed from the page cache upon completion. Uses the page cache to synchronize, and simply prunes folios that were instantiated when the operation completes. While it would be possible to use private pages for this, using the page cache as synchronization is handy for a variety of reasons: 1) No special truncate magic is needed 2) Async buffered reads need some place to serialize, using the page cache is a lot easier than writing extra code for this 3) The pruning cost is pretty reasonable and the code to support this is much simpler as a result. You can think of uncached buffered IO as being the much more attractive cousin of O_DIRECT - it has none of the restrictions of O_DIRECT. Yes, it will copy the data, but unlike regular buffered IO, it doesn't run into the unpredictability of the page cache in terms of reclaim. As an example, on a test box with 32 drives, reading them with buffered IO looks as follows: Reading bs 65536, uncached 0 1s: 145945MB/sec 2s: 158067MB/sec 3s: 157007MB/sec 4s: 148622MB/sec 5s: 118824MB/sec 6s: 70494MB/sec 7s: 41754MB/sec 8s: 90811MB/sec 9s: 92204MB/sec 10s: 95178MB/sec 11s: 95488MB/sec 12s: 95552MB/sec 13s: 96275MB/sec where it's quite easy to see where the page cache filled up, and performance went from good to erratic, and finally settles at a much lower rate. Looking at top while this is ongoing, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7535 root 20 0 267004 0 0 S 3199 0.0 8:40.65 uncached 3326 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd4 3327 root 20 0 0 0 0 R 100.0 0.0 0:17.22 kswapd5 3328 root 20 0 0 0 0 R 100.0 0.0 0:13.29 kswapd6 3332 root 20 0 0 0 0 R 100.0 0.0 0:11.11 kswapd10 3339 root 20 0 0 0 0 R 100.0 0.0 0:16.25 kswapd17 3348 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd26 3343 root 20 0 0 0 0 R 100.0 0.0 0:16.30 kswapd21 3344 root 20 0 0 0 0 R 100.0 0.0 0:11.92 kswapd22 3349 root 20 0 0 0 0 R 100.0 0.0 0:16.28 kswapd27 3352 root 20 0 0 0 0 R 99.7 0.0 0:11.89 kswapd30 3353 root 20 0 0 0 0 R 96.7 0.0 0:16.04 kswapd31 3329 root 20 0 0 0 0 R 96.4 0.0 0:11.41 kswapd7 3345 root 20 0 0 0 0 R 96.4 0.0 0:13.40 kswapd23 3330 root 20 0 0 0 0 S 91.1 0.0 0:08.28 kswapd8 3350 root 20 0 0 0 0 S 86.8 0.0 0:11.13 kswapd28 3325 root 20 0 0 0 0 S 76.3 0.0 0:07.43 kswapd3 3341 root 20 0 0 0 0 S 74.7 0.0 0:08.85 kswapd19 3334 root 20 0 0 0 0 S 71.7 0.0 0:10.04 kswapd12 3351 root 20 0 0 0 0 R 60.5 0.0 0:09.59 kswapd29 3323 root 20 0 0 0 0 R 57.6 0.0 0:11.50 kswapd1 [...] which is just showing a partial list of the 32 kswapd threads that are running mostly full tilt, burning ~28 full CPU cores. If the same test case is run with RWF_UNCACHED set for the buffered read, the output looks as follows: Reading bs 65536, uncached 0 1s: 153144MB/sec 2s: 156760MB/sec 3s: 158110MB/sec 4s: 158009MB/sec 5s: 158043MB/sec 6s: 157638MB/sec 7s: 157999MB/sec 8s: 158024MB/sec 9s: 157764MB/sec 10s: 157477MB/sec 11s: 157417MB/sec 12s: 157455MB/sec 13s: 157233MB/sec 14s: 156692MB/sec which is just chugging along at ~155GB/sec of read performance. Looking at top, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7961 root 20 0 267004 0 0 S 3180 0.0 5:37.95 uncached 8024 axboe 20 0 14292 4096 0 R 1.0 0.0 0:00.13 top where just the test app is using CPU, no reclaim is taking place outside of the main thread. Not only is performance 65% better, it's also using half the CPU to do it. Signed-off-by: Jens Axboe --- mm/filemap.c | 28 ++++++++++++++++++++++++++-- mm/swap.c | 2 ++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index dd3042de8038..139d1db79ff8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2473,6 +2473,8 @@ static int filemap_create_folio(struct kiocb *iocb, folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; + if (iocb->ki_flags & IOCB_UNCACHED) + __folio_set_uncached(folio); /* * Protect against truncate / hole punch. Grabbing invalidate_lock @@ -2518,6 +2520,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_async_ra(&ractl, folio, last_index - folio->index); return 0; } @@ -2547,6 +2551,8 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); @@ -2594,6 +2600,20 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio) return (pos1 >> shift == pos2 >> shift); } +static void filemap_uncached_read(struct address_space *mapping, + struct folio *folio) +{ + if (!folio_test_uncached(folio)) + return; + if (folio_test_writeback(folio) || folio_test_dirty(folio)) + return; + if (folio_trylock(folio)) { + if (folio_test_clear_uncached(folio)) + folio_unmap_invalidate(mapping, folio, 0); + folio_unlock(folio); + } +} + /** * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2707,8 +2727,12 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, } } put_folios: - for (i = 0; i < folio_batch_count(&fbatch); i++) - folio_put(fbatch.folios[i]); + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + + filemap_uncached_read(mapping, folio); + folio_put(folio); + } folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); diff --git a/mm/swap.c b/mm/swap.c index 10decd9dffa1..4019ab371759 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -427,6 +427,8 @@ static void folio_inc_refs(struct folio *folio) */ void folio_mark_accessed(struct folio *folio) { + if (folio_test_uncached(folio)) + return; if (lru_gen_enabled()) { folio_inc_refs(folio); return; From patchwork Tue Dec 3 15:31:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892613 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9266D20371A for ; Tue, 3 Dec 2024 15:32:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239981; cv=none; b=eLkaHckM2qH3zyxbdxsGWaNMNjauMx/WV3o+wvj9LwQcKbPjoca7G/KzY55hjGG3lHfCUedxM6NdZqaPfEgHBuxnAfWu615vDd2Igh+QBEo3wijfa9kxBj9MmerTD8fZGKKAZhROZUdFgsRoQTTKXSXLsv4pa8I0srKgTDCLiQQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239981; c=relaxed/simple; bh=vs/kjKbKvWVyON8cIhVh+OmQAXll4qsNA3yrIFPC0B8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TqSzHT6W6vbNvzvf34n3VKrA6t1pWH9/eQXRqbxoHMpZ7aohqFcnpV26UNSus/I6WtiEXkWDm/aKu7Pft457/eYdPEt8DUPMF2J73QoUVfBrakQtd+rw+hCuM5dmrYTsJ951WH+3rB+3+dAVRnZZPDUY0mnFnqGbcsQ1uPwni7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ZoXjuGJ6; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ZoXjuGJ6" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3ea3cc9a5ddso2808082b6e.3 for ; Tue, 03 Dec 2024 07:32:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239978; x=1733844778; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=ZoXjuGJ6LW812fJZgfsUYAW0o3hUhJ1ECO3V3B3mzS9aIwU9AjwvE85cw46HRAI5RZ epkxvG7nOJM1hP2wfxke+5kgfquRsWWR65Lk0pAEnMXQ2+gMnh4ysevXl+0+Vx0QEp1h cCYRdj3tlbvMIc6o2QzxKIA4awRVNLJiNLD3bcLulb1skvEHqI3ICwDZNgmnR+XJGlta PzOYkueg0mzh2ofTcsjh3eUtmS35rpWUUBzZIURqsN2uVxdy0dU5Be80xisMA3sZgCts XoQKMOK2tYHYoMpMmSRSQmG2SLD6J2Cx+284FNdBXzFv+ywcAQBF8/n9i4cgFz1lrk6i sjjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239978; x=1733844778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=NH+dEYRXUnsNrbcXBgpkihcpBvoWYQaUSK3NhUTPBycnuPdby5cAOKGW5E7yy3/gBJ 1jSDXoRZyoRTsZeiai6wfyeHi37DYgLw8QLAkV3VUEQJMhUdRVKv54+wK7+NB5i/3qQw toqZq7P4M88Zu8wbRi9O7QOYSn0beqkqNSWGh/w7BtP2DJ8HrGsh2tSqLUi+BHyoatj5 5QYrvTHrXpqAheGxftmMJmmHOgtjtGSzWH/xunTIcZ2952un1lXPnjupar96geRAhCci cNrc/PoW9N0VD0ovnW6ewoWyLWK1a42hzWQme4/H1c0rWcGXN/FjydULNjVMhICAP0IC efVQ== X-Forwarded-Encrypted: i=1; AJvYcCXBIJbSmBiylYipyFd0btwpX9e0orQMdr8nzouxEIRFoegV3lWHXFlW3wMS9yQL09yjB7VrF8WZiF28LqmR@vger.kernel.org X-Gm-Message-State: AOJu0YxQ46nGuFKL+9Gg0Y2HGUok6vlR0Ow/EsKkuM6LSYokQuijDUEf AW+336G6Ga5h3U9VH0ji94G2+R/S/fjL0SKVvGlYOCugi/6vD7oz8zs63nYM7F0= X-Gm-Gg: ASbGncukkoGgCLx5WrRUajRXjc7GkCBPRCQxJxmDpAkYHvIpJscMGc0QD417fzfEM93 7nzVkp1H9y9bs1571GRvidNm58RJORJK9sm4yw4DaMhe1EePyfzn6fQkCr6RqH2/nSYvC5wr3n/ Y/AS2eKPLfwup/BrPgJtDMYELi3cTSiOlO6yCQHT5tl0sIWGhFGWa4GzPjTEW3S8IOgeJfSzPkC 6DgpN3rVuWWqsMkyvG/hVTaJcU77xAidxbyz/Rx5qWEi9ImXPq6Sa6qozc= X-Google-Smtp-Source: AGHT+IGmoU9sxAuWmjwWvkKLQ0cqG9D7TpsEEAnWsGzkgwInxbMdKTegKVNOgiZahNH2vDY40Qhwdw== X-Received: by 2002:a05:6808:218d:b0:3ea:4aad:a877 with SMTP id 5614622812f47-3eae50d7cd4mr2734954b6e.43.1733239978709; Tue, 03 Dec 2024 07:32:58 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:58 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 09/12] mm/filemap: drop uncached pages when writeback completes Date: Tue, 3 Dec 2024 08:31:45 -0700 Message-ID: <20241203153232.92224-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO. Signed-off-by: Jens Axboe --- mm/filemap.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 139d1db79ff8..eb6a8d39f9d0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1599,6 +1599,27 @@ int folio_wait_private_2_killable(struct folio *folio) } EXPORT_SYMBOL(folio_wait_private_2_killable); +/* + * If folio was marked as uncached, then pages should be dropped when writeback + * completes. Do that now. If we fail, it's likely because of a big folio - + * just reset uncached for that case and latter completions should invalidate. + */ +static void folio_end_uncached(struct folio *folio) +{ + /* + * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but + * can happen if normal writeback just happens to find dirty folios + * that were created as part of uncached writeback, and that writeback + * would otherwise not need non-IRQ handling. Just skip the + * invalidation in that case. + */ + if (in_task() && folio_trylock(folio)) { + if (folio->mapping) + folio_unmap_invalidate(folio->mapping, folio, 0); + folio_unlock(folio); + } +} + /** * folio_end_writeback - End writeback against a folio. * @folio: The folio. @@ -1609,6 +1630,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); */ void folio_end_writeback(struct folio *folio) { + bool folio_uncached = false; + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); /* @@ -1630,9 +1653,14 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake_bit(). */ folio_get(folio); + if (!folio_test_dirty(folio)) + folio_uncached = folio_test_clear_uncached(folio); if (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); + + if (folio_uncached) + folio_end_uncached(folio); folio_put(folio); } EXPORT_SYMBOL(folio_end_writeback); From patchwork Tue Dec 3 15:31:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892614 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18F6E207A02 for ; Tue, 3 Dec 2024 15:33:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239982; cv=none; b=TWdsK6nfIjCyShoRZUhmCW32Mik2UbUE75N7EX7iyZm7UJ7cxRlZP4ap9UbqvRmvTBb1rFqev1hMVVfrpW3Pww8JOZDcp4neoLhggUL5/Wkxsn+HpeYZFHo6/J2lmdlymqxVg9bQMXl7XYBDCiL5euVAlix8LGNsmlJpSdEFpLo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239982; c=relaxed/simple; bh=+Kcl03qnXK2xysxVuDL9GwHkxgxUkvV623M1SHc2Ce8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dHQ29sIP+IzpbNjvDcPpjg7LW584XvZKOwvIlXBk/VN8LqusZ40Vu0YO+fYwbYHkRItbrkhVbY3qg3geb/ERgb1faoky+OMXop8tSrmPBaPAASRRGFEmiKfT4hWynHx2WZVT2cOg03PclzYH4X60EHsZRNrWRjG8zGqxBWmewKk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=zawtuFyk; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="zawtuFyk" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3ea4c550a3fso2766374b6e.2 for ; Tue, 03 Dec 2024 07:33:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239980; x=1733844780; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=zawtuFyksJZVwTNlOSWuLl0I7/NrJnGwM6osw35w7RF+X4LrOQBsIpdgqB21N2DG/L hs2QdLfiE//80U5o1IqKVnvGP/5rJciwzgJESHGtoiAGmd2A8D07PvYIoD/qq09Z7Hqd XvzlipIh94GKK1kJnwgRE3/19xA5Q9ihvdRUc1Slmyqe7J5l9co69/51Jva6CzI66LUW 9bMDGOXzlwMfzaK28tdx1giFwMEZS3GcXInjHWOOzqqbNyxG/kcGgC+tl1F3JdjMIycd kJ6BWTh3DFaDiIFxiFYz+iLsRlVvoYLhAzfZBlP49LBl3KnRdlxMazAvHXaCyDlSJdFv fJxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239980; x=1733844780; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=JOp8/3QHb39MKzVT6iGasr7jc35m71sySE/anNZOYK1z+7sXwt1mDL+edUJrohDHHk O2WV7g3iFDD9Cqgt68HA3Nj+Pi8hGc5NCnX987n/NhZwALCM3PUf/LjW7DNk2+Oq4U94 pFUeWPNcC/CMZm621Il41FUHyIx92Nrqcmxt52U+QYNrO+w7bP3XLORP0v/fqPCN3ivX IyN/QHm1Hd9aKb+TNwOApUeTmhvGckIupKp7xz5m1v4bfO5lnVH0c4ZxPzezAiC6/mHl z9lIJ04UJrcVK9siS/tff1X7WrPS9zXDJBuPa7i5NJQ+38yE5IquK6T+SuNEK4Dx63G5 w4jg== X-Forwarded-Encrypted: i=1; AJvYcCV3rzkp4PWtNxJ+QPzhMczS293Yq6VZRgBapkPBAT3isNejIfeNCN75i1nyNqANTkzS+dxVwZ+xTvSD+0L9@vger.kernel.org X-Gm-Message-State: AOJu0YxTPf5+l//+14W1DiT+gTAmByjQ9AagvRtDVOIPSLtbi8ZaCCYI /fMQW+UT2lmhztfgScfClafeveXM8UWIyfktH5tZr92J27K3m66G2FmylK3zykc= X-Gm-Gg: ASbGncshpz2UKVGg+FMRiVxsud+JL1a3cGVaSvThafX2VHlrJHKTAD1BE5XxbwGaCcL 5B/9xvdpg3PeXAnM4E9xNu0iGmy3GecgoolqgTS0r9aESwPi8Tc230/8AofIh1ZPVHM+zIQi1mS Fwd8TOYRMoaF83OdvcbFP3ySYA78Q/8rdSqjp5Cp67XS4ABBwM38ogPtNns7Y9HPNZ/NS1LM9hu i7diMFZE0ZIdU3HiSiCYgAE2NeWP6NxSNjaRK78g0ozeHuqu7OKhCwxHSM= X-Google-Smtp-Source: AGHT+IG43FQHTFqL4PqM6F3qojzARV1JxsiCZ9D3Zgsdfa2TEHjYK26CmkPy2H7KTh+zkEYZWpQaaw== X-Received: by 2002:a05:6808:3085:b0:3ea:52b7:ebdd with SMTP id 5614622812f47-3eae50c51b8mr2904392b6e.42.1733239980115; Tue, 03 Dec 2024 07:33:00 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:59 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 10/12] mm/filemap: add filemap_fdatawrite_range_kick() helper Date: Tue, 3 Dec 2024 08:31:46 -0700 Message-ID: <20241203153232.92224-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Works like filemap_fdatawrite_range(), except it's a non-integrity data writeback and hence only starts writeback on the specified range. Will help facilitate generically starting uncached writeback from generic_write_sync(), as header dependencies preclude doing this inline from fs.h. Signed-off-by: Jens Axboe --- include/linux/fs.h | 2 ++ mm/filemap.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index b64a78582f06..40383f5cc6a2 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2878,6 +2878,8 @@ extern int __must_check file_fdatawait_range(struct file *file, loff_t lstart, extern int __must_check file_check_and_advance_wb_err(struct file *file); extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end); +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t start, + loff_t end); static inline int file_write_and_wait(struct file *file) { diff --git a/mm/filemap.c b/mm/filemap.c index eb6a8d39f9d0..826df99e294f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -449,6 +449,24 @@ int filemap_fdatawrite_range(struct address_space *mapping, loff_t start, } EXPORT_SYMBOL(filemap_fdatawrite_range); +/** + * filemap_fdatawrite_range_kick - start writeback on a range + * @mapping: target address_space + * @start: index to start writeback on + * @end: last (non-inclusive) index for writeback + * + * This is a non-integrity writeback helper, to start writing back folios + * for the indicated range. + * + * Return: %0 on success, negative error code otherwise. + */ +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t start, + loff_t end) +{ + return __filemap_fdatawrite_range(mapping, start, end, WB_SYNC_NONE); +} +EXPORT_SYMBOL_GPL(filemap_fdatawrite_range_kick); + /** * filemap_flush - mostly a non-blocking flush * @mapping: target address_space From patchwork Tue Dec 3 15:31:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892615 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70FDF208964 for ; Tue, 3 Dec 2024 15:33:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239984; cv=none; b=iYO4g/CfkzoKZm77ZMMqH2hGQGKxsKJlIPWxfF3hqGQWpnCwuHOS0jzVtvPorCvyQJSpgK3tEtL1bq9FDRUw08kmrLFQLOvk7UGQp2W5IGBrEaXpHHFpXVJOZ59YEVeVVg+Bax5U3+AZJav42JCzIhhjHgumDUD1sklSH0O+pD0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239984; c=relaxed/simple; bh=H+SJrhFgkMARuJzf6zQzkW6swewkOc9KT1deZMH8OhQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PJLu5bUZouaxqAzhweO2G8KQLbIP9xGY/gRVLUKuhFtQndyT9LiF942FEe2zrxBEY4vo+r/9n8Dmc2lQbDNCyozwwYqr/L6X7Gz1a969VuVQqtsNe6dlq0+fKkFon5Jwab0tNiEJj8/q/HZVJCW9sBc5z+rQf6gqMve4TGFDB5Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=NvPvt5h7; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="NvPvt5h7" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3e786167712so2728156b6e.3 for ; Tue, 03 Dec 2024 07:33:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239981; x=1733844781; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=NvPvt5h7l8lngjBGNuxy1XfMurd28jtD71TE5JEBJN8m0imAbUo5S3bdcA1KRVni0K adBG9tVcZH3XkpLeOYkphqHUsl8KthkdbhxTVwzY/AYvhuXK8Y1+2DFvCv6V01nqbG2e 3oUcfMTQszBNztI4erRKjKFbzvlrRZNoM57g/od4rXo89X13JlAbyf7VExy+ZafmwNUN TmnqZ40rMEAHhACUFoJrPsE14XGVVZYsZnLmmGH3lDZfNy7cR9Ctj/KxowZFTmsInge0 Bvoc8xEaEU5E2ZG/+QlgV/0AKK+JojGNNqffpdz0vhhgRX0cutoayz9fj9wDcLTW1kct 3org== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239981; x=1733844781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=uk+RwACfSQKHEV9M5XO9sxsaey4/h2vjUrd54lsr3y1DNfeb9UxiZw8yRPEQkyMKHo 55OzejKPy0y/BzeZswpOYsTJl9seIgn63Ltgi5Z3k+TGPYRGXS2U5pOZUaEycqY0sM4g OjgNv67CpcxYA4d8/OOk+Im8fQJvaEZSZt9OrNR2wihGsfHmcPxDZt/2CYpCaTEJQXPe Q6tXR9SZZUZEpWBQMVw7laOsg8gXwAQISUziw4pZW8uEKgWK9/Xq3a2rOiG9OJR8A8O+ EShQQW/jBWD7UMQeiA3ihj/ovF+TNak0R4NukNxKl0Uv8ag+Ly/igJhdAhThO59q7PMp A5GQ== X-Forwarded-Encrypted: i=1; AJvYcCUnXUp2wYb+KhA5lm5+Ju69xe9NgY2kIHrDt4s3TXBLY0nZoyG+ptZR2S2uRc65QlgTjDy433JBtxDNrDDk@vger.kernel.org X-Gm-Message-State: AOJu0YxsjOuYsF7nOp4eA/acpGRHon2b48oxx4Xhm7dW7ib/H7luF2uk DWBK8e+WOyon1erexIR1mD3I6zKSlOMl2vcxEqSdVtHiE85GKGZnSrsr7xRdcnQ= X-Gm-Gg: ASbGncsVWdAFoMHyFCNXfWTG94swPshDmKS2K3k3gkc6RFMy+Kg7dYl/K+lX7rmiDgL ZIUeK+yBafZOvoQs8lu9JzXkB4LP8i3yMxVYiYcfMn466A/mp/fr2qrst4lNZlQpJjgoFL7WDk7 dK/uzfsXXpz3hGVtOwBJtuFuTwQO/pVpAXaGchWxXFR2cItGWqt5cllb5FuQpbtpaRmP5g+uLcr uV3Dt7ve7sL1Eclh6M1YZFulk6OQ91P3zQ+6nLJFvJjbb2Kf97+yfWVdAg= X-Google-Smtp-Source: AGHT+IGnmoFBWMRz09BuY3oNVVI/1OH4qyI0X8Y48goC/ICz9U7NJeNOcoRPtjmNOo7azkeA+JUnjQ== X-Received: by 2002:a05:6808:200d:b0:3e6:6208:b737 with SMTP id 5614622812f47-3eae4f3bce5mr2757240b6e.17.1733239981545; Tue, 03 Dec 2024 07:33:01 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.33.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:33:00 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 11/12] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Tue, 3 Dec 2024 08:31:47 -0700 Message-ID: <20241203153232.92224-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary. Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec 2s: 132308MB/sec 3s: 132438MB/sec 4s: 116528MB/sec 5s: 103898MB/sec 6s: 108893MB/sec 7s: 99678MB/sec 8s: 106545MB/sec 9s: 106826MB/sec 10s: 101544MB/sec 11s: 111044MB/sec 12s: 124257MB/sec 13s: 116031MB/sec 14s: 114540MB/sec 15s: 115011MB/sec 16s: 115260MB/sec 17s: 116068MB/sec 18s: 116096MB/sec where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe --- include/linux/fs.h | 5 +++++ include/linux/pagemap.h | 9 +++++++++ mm/filemap.c | 12 +++++++++++- 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 40383f5cc6a2..32255473f79d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2912,6 +2912,11 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count) (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); if (ret) return ret; + } else if (iocb->ki_flags & IOCB_UNCACHED) { + struct address_space *mapping = iocb->ki_filp->f_mapping; + + filemap_fdatawrite_range_kick(mapping, iocb->ki_pos, + iocb->ki_pos + count); } return count; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f2d49dccb7c1..e49587c40157 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,6 +14,7 @@ #include #include #include /* for in_interrupt() */ +#include #include struct folio_batch; @@ -70,6 +71,14 @@ static inline int filemap_write_and_wait(struct address_space *mapping) return filemap_write_and_wait_range(mapping, 0, LLONG_MAX); } +/* + * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write, + * and the ->write_begin() handler on a file system supporting FOP_UNCACHED + * must check for this and pass FGP_UNCACHED for folio creation. + */ +#define foliop_uncached ((struct folio *) 0xfee1c001) +#define foliop_is_uncached(foliop) (*(foliop) == foliop_uncached) + /** * filemap_set_wb_err - set a writeback error on an address_space * @mapping: mapping in which to set writeback error diff --git a/mm/filemap.c b/mm/filemap.c index 826df99e294f..00f3c6c58629 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -4095,7 +4095,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) ssize_t written = 0; do { - struct folio *folio; + struct folio *folio = NULL; size_t offset; /* Offset into folio */ size_t bytes; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ @@ -4123,6 +4123,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) break; } + /* + * If IOCB_UNCACHED is set here, we now the file system + * supports it. And hence it'll know to check folip for being + * set to this magic value. If so, it's an uncached write. + * Whenever ->write_begin() changes prototypes again, this + * can go away and just pass iocb or iocb flags. + */ + if (iocb->ki_flags & IOCB_UNCACHED) + folio = foliop_uncached; + status = a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0)) From patchwork Tue Dec 3 15:31:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892616 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B0041F892A for ; Tue, 3 Dec 2024 15:33:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239985; cv=none; b=aHYnBlVneWc+ApLd6/rzmNzi1AiL6J9T0w/VdHluEVqY9liu32DjIdt6rn7ModslhzhYw21hIhjCy/oLLtxetzNV+Sx2ZFkfc0ZKiKudHSO3R0y4SCdbgArypodufzU1loJL0TWHI+HXC1l1L1fH23I+ZS9IwjDD/oLiqsExJPs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733239985; c=relaxed/simple; bh=uuqM4dJdCE0QCImMtNfF8QapqPv5QHbWROfY8tXB1So=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j2jAM7LNZAJlDVC0/kLWgvTBT1DKFgWrf9BVED7Jr0z6Eqq57MfUm40a+QUPZPGfAt+6u4/e8qy8EJBOfOkTIIEtlVgr3AMx04j6sdxaxGLOsU7JZAVjcoCXDZKNxoy/sXvd6vNXsvsSXEnZnUCWGd9b8ojVhOYR3u5GUL1pJUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=QOc0N/NU; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="QOc0N/NU" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3ea5a7a5e48so2650385b6e.0 for ; Tue, 03 Dec 2024 07:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239983; x=1733844783; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=QOc0N/NUqzgyQBFL3YcsFrekttyd/1YnrV1H2WXWVqq8rRuDr/fMbYTz0t0nSvjZsZ PM0IJhaPyr1XFGkB+CgN5rJGrQo594jDOQh7cePyS+sg/JBAs1GtxQqcsovONgFLy5Op 1+ljoiONKHAsiMjXtSjgd+p1IUQBCGfFdjwWv68j6hNyma/YiG/+fSzPcVvx21A7TXKZ J8YTAo4MNYwj2TozO7JRy9v50QYghtb7lOuUhpepBkAbV48PxgCkaesFHij5OlaLIewn r6GeI3nXSYx2eMFHBv35VFSCGjE7XwwK8w5vv5wtuIXoUTRr94mCjgtrTWpg37lX2KUa Jc9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239983; x=1733844783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=OwzzHzMLyCEr/ibjz3BDlk8OQAxo6mdW+E7Xtx0I7RZ6hwG40dK98lEiTJ59s/Y7gN NzyQ8J2hu+qpum7U3F60XMy6PKNO2sAKezW0IFb1hoJAbWiSoX6sKLOgF0hiVUp4gK0w mGyvnC0aA1ZmBYRwqpchHKPnAX2LhaE5/2f8PYTdgy27J/jGO8aAMSU2fmSoqUNrgnWb AFpkUQo9sZQPhDXOs9kcYnnx9Rgg+BzNrbykAfblPQNXyeMC6puAG29304aNF5jboBWO f6F7KU+SnsJ5FDCx9LcswO+GcJ4kl5hFzV2oFU4cSHIuzk0tP15Cr23+PTp63Lh+oLi9 tLww== X-Forwarded-Encrypted: i=1; AJvYcCVL7v4lHM5MENimCxKcUQU7l6b1sZhsknA+tANe+pkEXNq6vY+KTE7QXXSMT0u5BKr1UU3Uh3GxhhL16PvP@vger.kernel.org X-Gm-Message-State: AOJu0YyU7w44qEbYu3Jx44IEX8UahGYa75DqLHiB3wXeufuNNke97nsb gU0c6fI4AJ3wRrLsAaJpwTuZq3m5zXpBicBILaqjxuqRAA476DAJyqKhdTxOxVA= X-Gm-Gg: ASbGncsvFJvGJVFhMPqHY5uJutzzoT7raavTMqLyZy/DTLRqNmDONG9hjkRs2ABWB2g QK29RtVHjxZmuS3P0ZpiMDCj5Rqi9XK3m250kF97PQIu43GVdJ+SHRyfLgPt89+virMUqdCWp5V Gx25juLbeh8Z/lWYDonFyZHbHojymO1SsntuniCqc8svfvluqNb5Q0JKopSfkG8JxIQtzYsonPQ BwoUd2BK7xpgTju8kQDLJ+b8fDUJNqfxp/cniBGvU0hFdOpa667Sc32KkA= X-Google-Smtp-Source: AGHT+IFgNffB59kRtGSa279D+oxFDG6GGmZlBVzfslJYY485yS6Rp8valFBlgcp9p+EP83czXVtQIg== X-Received: by 2002:a05:6808:199e:b0:3ea:6533:f19d with SMTP id 5614622812f47-3eae505a0femr2960000b6e.30.1733239982974; Tue, 03 Dec 2024 07:33:02 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.33.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:33:02 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 12/12] mm: add FGP_UNCACHED folio creation flag Date: Tue, 3 Dec 2024 08:31:48 -0700 Message-ID: <20241203153232.92224-14-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Callers can pass this in for uncached folio creation, in which case if a folio is newly created it gets marked as uncached. If a folio exists for this index and lookup succeeds, then it will not get marked as uncached. If an !uncached lookup finds a cached folio, clear the flag. For that case, there are competeting uncached and cached users of the folio, and it should not get pruned. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e49587c40157..374872acbe1d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -721,6 +721,7 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, * * %FGP_NOFS - __GFP_FS will get cleared in gfp. * * %FGP_NOWAIT - Don't block on the folio lock. * * %FGP_STABLE - Wait for the folio to be stable (finished writeback) + * * %FGP_UNCACHED - Uncached buffered IO * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */ @@ -734,6 +735,7 @@ typedef unsigned int __bitwise fgf_t; #define FGP_NOWAIT ((__force fgf_t)0x00000020) #define FGP_FOR_MMAP ((__force fgf_t)0x00000040) #define FGP_STABLE ((__force fgf_t)0x00000080) +#define FGP_UNCACHED ((__force fgf_t)0x00000100) #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) diff --git a/mm/filemap.c b/mm/filemap.c index 00f3c6c58629..a03a9b9127b8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2001,6 +2001,8 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, /* Init accessed so avoid atomic mark_page_accessed later */ if (fgp_flags & FGP_ACCESSED) __folio_set_referenced(folio); + if (fgp_flags & FGP_UNCACHED) + __folio_set_uncached(folio); err = filemap_add_folio(mapping, folio, index, gfp); if (!err) @@ -2023,6 +2025,9 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, if (!folio) return ERR_PTR(-ENOENT); + /* not an uncached lookup, clear uncached if set */ + if (folio_test_uncached(folio) && !(fgp_flags & FGP_UNCACHED)) + folio_clear_uncached(folio); return folio; } EXPORT_SYMBOL(__filemap_get_folio);