From patchwork Wed Oct 4 16:53:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13409067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A91D4E7C4EA for ; Wed, 4 Oct 2023 16:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C9446B0270; Wed, 4 Oct 2023 12:53:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62C6B6B026F; Wed, 4 Oct 2023 12:53:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4143C6B0273; Wed, 4 Oct 2023 12:53:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0CE886B0271 for ; Wed, 4 Oct 2023 12:53:25 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D578240397 for ; Wed, 4 Oct 2023 16:53:24 +0000 (UTC) X-FDA: 81308374728.14.754A69D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 429761A000C for ; Wed, 4 Oct 2023 16:53:21 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WlYR7SVy; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696438402; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VjQq5vIRQg04dws7PyaymhwSTjm3kTO8DTKw00f2iiE=; b=Et2cQ66LnRSRzrfKpMX01ZINYBzQxl2xtDxv68A+26Kt3DmxEa8rvklSzc+ZzHxLaKtwE9 h8eoKlp1ZPGqXKRwEwmXQK35plGFfcJJLRBeJl565u/cFkH7UHBEDJzuld9p4tMSDBnz1v TtS8cHUE60aCbic5hJTvd0+TngK0omU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=WlYR7SVy; dmarc=none; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696438402; a=rsa-sha256; cv=none; b=eAiDJyivr+6ypIFjFap+tIB/4yvSvwIKfRVCBzymnAS+CPimQq6c7ZD/662Lm1MB/aeBcC Mnp+a3DWML7bP0/t9ccl8MIRyNbvSsB3hid8rxJxZu7k5MXUl48h406VElcRbq9OrN1TQm WfXubScCMdj3ZK6rraGXXRXgwp7XIbw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=VjQq5vIRQg04dws7PyaymhwSTjm3kTO8DTKw00f2iiE=; b=WlYR7SVyJnVNGNohBfF4SgXWK4 eiC2zmNKFtHyl0x9NHoD2Qj1GbLuvT9EfwgRXBH1bL+XUgJojOeePsBlj7eT+NoeJgfEXHAG7GTYQ oEZxQCk9OpT6d54J6R4sPQVYiJF8CjbPIXDfQ5fDNIaV7apCBlDD45D/7PZfXW3GGnYexS6Kergtq y7T4LA9JNaQ10uu2a1zCent9hOjzxoZi1Hsu2yv63P1xWF0B0ohpBTJRPXqYq9fdYxS6Xvw08aGHc 9cYF6pUKSBX41H5rDJac/ghEXLdcGM5d5+ihGxsw7ysvqGc4z7KoWUcGQDspm4pNnmE0KD9tDht/U HUI2ig1A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qo57g-004SG6-Og; Wed, 04 Oct 2023 16:53:20 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, npiggin@gmail.com Subject: [PATCH v2 15/17] mm: Add folio_xor_flags_has_waiters() Date: Wed, 4 Oct 2023 17:53:15 +0100 Message-Id: <20231004165317.1061855-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231004165317.1061855-1-willy@infradead.org> References: <20231004165317.1061855-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: jb46m9m1cd88ihqtxytf4d47jbnbek67 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 429761A000C X-HE-Tag: 1696438401-961857 X-HE-Meta: U2FsdGVkX19GrEtaSfepzY5f0LAUmYX5RDUs3eI64mmxoCxDnxnt+dpbVkSXLasNtAIO7UAI9o1Y/n1t2CUKJbpCPRBpaRd2zLbO7n6DaN/JkKkpdBmFMaDVISJYPVudcbkJyor18SfDzzQB3YJAJbrQ5VVjkp4SUuH5wr6i/3UTvUpYmE4sexleeLLDTwH5M0d+FX5giC3M6V2fK53HVcaz5sePu82QnYaonUSgvPrydgkjdLaMsKp5rRbOtZwHt8hy3af/3OsOvZF/HS9Cj9KK52hJ2ff8IQ/q/da8LuKBLwrC3oO00Zvfim9Crh4M7BEFGRKnxUvEh5c0BPXanni8LCY2ASvWmOnHDXP3YbOvx+1TZfnTBkGcx1hyNew3wI6ahfrj6DEh8LjD6MUvwMdIe7amO/MyNE/c5dTeJQgyNyezxYM8hDCxckNxumi/uM3M50tu1+LYG5HZgRMV1vKvPocRouyuxEPtSklr5qCBsZnBFouh6YnrOTjsH8gVgZeZY3CIkyvpbly2sdm1lTfYb5JgcAEIBaWvS+PKr48GIqCMhVsMkM848nyXLa6Zce/93zHtdLdx3S1aOp55vKq5CyZ3sY0cTKn863RZK1OBv3SPh608KspCkxxNLNVT6j7H4+yiquGe2MoVIzVnzEAhgPCbpXsPMV2ia0WfWKA3NE3QobVa1C1lfvNfU70zwCEu3KcXpIIq7H7x/124TP9obty4Cj6eMFICmTLWLnnJ5KZAxi90jSbWQW6poxx2Ba2RZH24XRd19mnD9MjiRiGwDFzz0s1C5CQIppbiKpaQq75UHDr0Aq6+c994aPdR2z09vtwSrvYoVmqirvlP7GFx9dpBUWEGXJNBNRWkVF0/M1dEgh7qZchxdlJG6gLxaLkpQHshC6DNoVL1ZrcFocuDmifWUIWjKInZUB0Ybl9bcFEh8+uFvNyk7+n5b6910nm2PHjyEO/qlDxZnhZ hXcJsezo XUeSj93+FZTjKIyUOSe6E9Sn4XbAunEW55sBk3A8V783Hz9LdwU6GzlCiPJeVgko4QdNiiW17uLqorQ7VQekLfJwxQRyRQAKg/VE1la4zi+eUuMsdwhA5YEXCv+pfk1m3lu83MvjRuETcG2d7xNHr1Vnj4Sni30/A8aFLaoCFb9kKFG/DPcyGhy1Hjtpez1UhwfKx5JRvGR+4ix30bzdKirHcJkpRVep1VWBpOFsnvk/ySHaIiu1JHLyupUfWb9dEGKdp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Optimise folio_end_read() by setting the uptodate bit at the same time we clear the unlock bit. This saves at least one memory barrier and one write-after-write hazard. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/page-flags.h | 19 +++++++++++++++++++ mm/filemap.c | 14 +++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 5c02720c53a5..a88e64acebfe 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -692,6 +692,25 @@ TESTPAGEFLAG_FALSE(Ksm, ksm) u64 stable_page_flags(struct page *page); +/** + * folio_xor_flags_has_waiters - Change some folio flags. + * @folio: The folio. + * @mask: Bits set in this word will be changed. + * + * This must only be used for flags which are changed with the folio + * lock held. For example, it is unsafe to use for PG_dirty as that + * can be set without the folio lock held. It can also only be used + * on flags which are in the range 0-6 as some of the implementations + * only affect those bits. + * + * Return: Whether there are tasks waiting on the folio. + */ +static inline bool folio_xor_flags_has_waiters(struct folio *folio, + unsigned long mask) +{ + return xor_unlock_is_negative_byte(mask, folio_flags(folio, 0)); +} + /** * folio_test_uptodate - Is this folio up to date? * @folio: The folio. diff --git a/mm/filemap.c b/mm/filemap.c index ab8f798eb0af..3dad2615af41 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1499,7 +1499,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - if (xor_unlock_is_negative_byte(1 << PG_locked, folio_flags(folio, 0))) + if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) folio_wake_bit(folio, PG_locked); } EXPORT_SYMBOL(folio_unlock); @@ -1520,9 +1520,17 @@ EXPORT_SYMBOL(folio_unlock); */ void folio_end_read(struct folio *folio, bool success) { + unsigned long mask = 1 << PG_locked; + + /* Must be in bottom byte for x86 to work */ + BUILD_BUG_ON(PG_uptodate > 7); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio); + if (likely(success)) - folio_mark_uptodate(folio); - folio_unlock(folio); + mask |= 1 << PG_uptodate; + if (folio_xor_flags_has_waiters(folio, mask)) + folio_wake_bit(folio, PG_locked); } EXPORT_SYMBOL(folio_end_read);