From patchwork Thu Jul 11 02:13:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13729934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DED1C3DA41 for ; Thu, 11 Jul 2024 02:13:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D62CF6B009C; Wed, 10 Jul 2024 22:13:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D11FA6B009D; Wed, 10 Jul 2024 22:13:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B651B6B009E; Wed, 10 Jul 2024 22:13:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9635D6B009C for ; Wed, 10 Jul 2024 22:13:38 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 132A0C02DE for ; Thu, 11 Jul 2024 02:13:38 +0000 (UTC) X-FDA: 82325850516.02.E9331AB Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 5AA92C001B for ; Thu, 11 Jul 2024 02:13:36 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="sUnObo/h"; spf=pass (imf10.hostedemail.com: domain of 3zz-PZgYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3zz-PZgYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720663973; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NwEn3rgyWV9MRh72z1BbFtM83NQgy5RTa5P2H131skI=; b=6/XP31qZdYxdKXbgXlGrg4PIPSTpGen5kAh/dsewSJT9KXAFUgJ2LeMWqMlzAfb8pDPiN/ HqWz04F99ZA3AVhgmgU2X6EHWrCt8qyoldwgVlbCw384tgHaOBiumIryzfEM2qDhsK9OSV qKmXB+nqOL0cAGQu7FCwh2xFOM7GVOw= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="sUnObo/h"; spf=pass (imf10.hostedemail.com: domain of 3zz-PZgYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3zz-PZgYKCMYA6Btm0s00sxq.o0yxuz69-yyw7mow.03s@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720663973; a=rsa-sha256; cv=none; b=lmciyV973sv0SeAruBknEPT/y+Kia1ow0V3sJMv98g/cu9RsTrEFDhaJEd+PN2JrCMdTT4 /ALTbUH3amX0rRFeA+bmzbAiJLXuXC6vQnTuuMt5qaVErkK5GJBQFJrdz7bjxt9JmfyBC2 Ft4BxuVyAjP53Xqhn4adFcVw6u32EQM= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e03623b24ddso736889276.1 for ; Wed, 10 Jul 2024 19:13:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720664015; x=1721268815; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NwEn3rgyWV9MRh72z1BbFtM83NQgy5RTa5P2H131skI=; b=sUnObo/hpXTzBOy2ANdEeq7/YyRj016QVlYP2LLTeoCuSDJjSzNxI/fgBCsQRWidfz E4lTZafyc/aH1lGZiqNgRtCHEcyUuJGYP33YEqBzK2Pm+H0xhUjoejTRpYLkvFQWfyJB AA2fWhSF2j2BljLz3W6rYqYY/5bmhMpxveiXDNWDEIMobn4giyKT25J3HsUcasdNoZbS UmBdBl8lh1ymXiDtOhGDQbYzXLzcksD8Ij8YjHxjppGamZhIvOpLZYZO2Uidui0NZS2f u95XI3BTxguv0Ce43gT6bCZhpBbOB5GpX93Ilkt6JjiYMV7Gn5K/JLYdK+4NVVp0y0+5 FXGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720664015; x=1721268815; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NwEn3rgyWV9MRh72z1BbFtM83NQgy5RTa5P2H131skI=; b=iEhDzHbueUrefxDEWd4Vt8t+psgi+jx8BEN7I5PrPYgy+3IJEKddL5jtxA3/Jjyu9o Sr1zrcUsxrPw5L710cwohRNMdqVbH80FtSw9c7y9Wmx5JUzixNMfu80wJ9wVxT8sAKo4 PY60zoIr3ElAnGCy78dXpu/XUXRMKkPkrYpjKp7ofJsd/duxaY7wJczzKNjw1jpOp2Ut st7uyFe6lcXJjjODXeCv5nh9Jptbj3Cp3bl3R3cde3mq8fjkOD13LIdVQ7uaE/kjr5AT 6Xv4cqVv+mZequV1OXHdVCVkcZrRsROy3/9VGqqkT+xuUYVWWyZAKrPcvLRWq4tD+7sL tMfw== X-Gm-Message-State: AOJu0Yy63z6qz8G3Pg9F55F0k7Oq5D/gWBJ59mBoLMGvFUWhtaR+zoVv 6Y6hJFkDFaw4YWb06hIB6uzJ5JTn/J0P075z8U1HSNi45keiHJUMP5tt0OAoKfhGcDpABxDjgdW mAw== X-Google-Smtp-Source: AGHT+IEbVZmYW6DpRm49c6xD0RPJKOVNNtKwNJhKE+00HKB2nAyQq5npS5jud2Z6RbIqjV4/yQeeTz+gpoU= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:9b06:2f28:6675:a98b]) (user=yuzhao job=sendgmr) by 2002:a05:6902:158c:b0:e03:a0b2:f73 with SMTP id 3f1490d57ef6-e041b05c952mr151870276.6.1720664015346; Wed, 10 Jul 2024 19:13:35 -0700 (PDT) Date: Wed, 10 Jul 2024 20:13:17 -0600 In-Reply-To: <20240711021317.596178-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240711021317.596178-1-yuzhao@google.com> X-Mailer: git-send-email 2.45.2.803.g4e1b14247a-goog Message-ID: <20240711021317.596178-6-yuzhao@google.com> Subject: [PATCH mm-unstable v1 5/5] mm/swap: remove boilerplate From: Yu Zhao To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 5AA92C001B X-Stat-Signature: pjcuwc8acbjdixcxqcrjk3qf4trkwsek X-Rspam-User: X-HE-Tag: 1720664016-423998 X-HE-Meta: U2FsdGVkX19eF0Xnxr7qAe046CDgJ1wbc9KC6sVwR0j502ihPBK/p5FR4hVG2+nEYmygnzRtQgOsp/YHI/DXFZK2wf2Jnm2fLYYhcuou9D8WNsi9l/IN7Nm80GiFLCcMXFJIsg1y0lDrlOe+WvrdYUxmYQGCazJjmmTnwMeRIt8zxzIBPHhYQ6kEAILJe0cnPO2G/e+62qlRqlYcXe50pXxCnsYxPyGIr5PheWzK3WwFWbHllvk0frnmeFZSnDxlQ6cN5aKww4e4Ygwd7QOEymCDWl7TMVXMaU8OeOjBzt8/OzGHMWvkqnodzN4fDK+Xp8J8CXACc5pyr5A/EgqXEjEEXxQ1LHVYSa9Sm9PSL0ef/yRxWXTfcb5QqoDOzntl1wR70FWXgE0GgYPohEqj4DV0AB2p2S+omvA7Pb7Xi7ILA0kHDiuiwH5Hm1RSXpi68bNxSOSpiAXSyh/HAFHPB3BCm55It9S1hW2QGPDgdkPrhbWfbgkuI5IC6M+6qFsYN7AdCyqIBj5LOKAkjvNpvBPTYCc8i2o64uN6R5kP/isJ1tqNnTiz2LfKOp1vcgzGZrJkvSzs5T9677yBsZILfiPuEjGNGSUU3hTZ8hSwP6GTMg1Gr1a/CcrV0BRUIdwPKiBpvKVBcBK8eVYNTnKhf4t7u74289C2FE1m+3x9oSbDpEk5vzwZWv/vaIpYydgF34oHfHXk+9uPmD91kFABZTuQ4bFpOGoWPQZfEOn145UXy063szoMctT4bdTocvWJ5na6TUeo9gUvjLUUkmhUgBAQsOe7gKUf+AlfP0IylnZqEhH4efU31RFzb7lgDIlpR0M+ye8IGJMraNVdImnwz6L/36I4cDe5wZp20qHDr5ZzQ56gkkgw1KngFulXXQnTdQwwhsJj2TAHg5IfVantWP/zJ5CwOGXignT10+73x8/EOTPWbpmmfMRzWEZDjq8I2HIX4fnM71bOPZZtedu 5vhcxprg N/bp43P8RVD1h9XlLJQkcvyh9D2DVbzhDyJGdCq3BMmdnt4sf/PFk7L7xaSxKef9nQvamWfDhnO5YpqnSUa0ubVpx9aXV/rqZ5mE0pdVbl14k+g+KiCjLqof/NaK20Bqwt45NcMYhjo08aQ5Aq2Cl1uyEIUVhnCdqR5hYedthIVi6LI+Gckoo8co7fEf2d5Vo1QfV9YU1Dl+P8cj43kxSartPpbJylGNn0VPbKqJDjEh7l+xdUezQwO5FujksuYj5xwdUb09vCvqG/g6SBgwhv3EJ2zQpW6hOY3fb33MlbS3zYpPL2bXKywY8GmLGEGDGQliUl+2vCHikQD4HJCeGdOhrCs5cH/D4a1p3TBdAxFIp7l0rSHo06VTdsXl6XFsN+iePiERQH+LJEVoMH14XyQfM1/02DwwU1QtxQwY+p0m8Te/28MW0CwnmF5NiaQmtmaVhN1R1CpFJkU2EHV/4EMlaMiK54RYt9quWj56L6zDwF/i2ZdKlyG0YM9kVWe2lQANH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove boilerplate by using a macro to choose the corresponding lock and handler for each folio_batch in cpu_fbatches. Signed-off-by: Yu Zhao --- mm/swap.c | 107 +++++++++++++++++++----------------------------------- 1 file changed, 37 insertions(+), 70 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 4a66d2f87f26..342ff4e39ba4 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -220,16 +220,45 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) folios_put(fbatch); } -static void folio_batch_add_and_move(struct folio_batch *fbatch, - struct folio *folio, move_fn_t move_fn) +static void __folio_batch_add_and_move(struct folio_batch *fbatch, + struct folio *folio, move_fn_t move_fn, + bool on_lru, bool disable_irq) { + unsigned long flags; + + folio_get(folio); + + if (on_lru && !folio_test_clear_lru(folio)) { + folio_put(folio); + return; + } + if (folio_batch_add(fbatch, folio) && !folio_test_large(folio) && !lru_cache_disabled()) return; + if (disable_irq) + local_lock_irqsave(&cpu_fbatches.lock_irq, flags); + else + local_lock(&cpu_fbatches.lock); + folio_batch_move_lru(fbatch, move_fn); + + if (disable_irq) + local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags); + else + local_unlock(&cpu_fbatches.lock); } +#define folio_batch_add_and_move(folio, op, on_lru) \ + __folio_batch_add_and_move( \ + this_cpu_ptr(&cpu_fbatches.op), \ + folio, \ + op, \ + on_lru, \ + offsetof(struct cpu_fbatches, op) > offsetof(struct cpu_fbatches, lock_irq) \ + ) + static void lru_move_tail(struct lruvec *lruvec, struct folio *folio) { if (folio_test_unevictable(folio)) @@ -250,23 +279,11 @@ static void lru_move_tail(struct lruvec *lruvec, struct folio *folio) */ void folio_rotate_reclaimable(struct folio *folio) { - struct folio_batch *fbatch; - unsigned long flags; - if (folio_test_locked(folio) || folio_test_dirty(folio) || folio_test_unevictable(folio)) return; - folio_get(folio); - if (!folio_test_clear_lru(folio)) { - folio_put(folio); - return; - } - - local_lock_irqsave(&cpu_fbatches.lock_irq, flags); - fbatch = this_cpu_ptr(&cpu_fbatches.lru_move_tail); - folio_batch_add_and_move(fbatch, folio, lru_move_tail); - local_unlock_irqrestore(&cpu_fbatches.lock_irq, flags); + folio_batch_add_and_move(folio, lru_move_tail, true); } void lru_note_cost(struct lruvec *lruvec, bool file, @@ -355,21 +372,10 @@ static void folio_activate_drain(int cpu) void folio_activate(struct folio *folio) { - struct folio_batch *fbatch; - if (folio_test_active(folio) || folio_test_unevictable(folio)) return; - folio_get(folio); - if (!folio_test_clear_lru(folio)) { - folio_put(folio); - return; - } - - local_lock(&cpu_fbatches.lock); - fbatch = this_cpu_ptr(&cpu_fbatches.lru_activate); - folio_batch_add_and_move(fbatch, folio, lru_activate); - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_activate, true); } #else @@ -513,8 +519,6 @@ EXPORT_SYMBOL(folio_mark_accessed); */ void folio_add_lru(struct folio *folio) { - struct folio_batch *fbatch; - VM_BUG_ON_FOLIO(folio_test_active(folio) && folio_test_unevictable(folio), folio); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); @@ -524,11 +528,7 @@ void folio_add_lru(struct folio *folio) lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) folio_set_active(folio); - folio_get(folio); - local_lock(&cpu_fbatches.lock); - fbatch = this_cpu_ptr(&cpu_fbatches.lru_add); - folio_batch_add_and_move(fbatch, folio, lru_add); - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_add, false); } EXPORT_SYMBOL(folio_add_lru); @@ -702,22 +702,11 @@ void lru_add_drain_cpu(int cpu) */ void deactivate_file_folio(struct folio *folio) { - struct folio_batch *fbatch; - /* Deactivating an unevictable folio will not accelerate reclaim */ if (folio_test_unevictable(folio)) return; - folio_get(folio); - if (!folio_test_clear_lru(folio)) { - folio_put(folio); - return; - } - - local_lock(&cpu_fbatches.lock); - fbatch = this_cpu_ptr(&cpu_fbatches.lru_deactivate_file); - folio_batch_add_and_move(fbatch, folio, lru_deactivate_file); - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_deactivate_file, true); } /* @@ -730,21 +719,10 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - struct folio_batch *fbatch; - if (folio_test_unevictable(folio) || !(folio_test_active(folio) || lru_gen_enabled())) return; - folio_get(folio); - if (!folio_test_clear_lru(folio)) { - folio_put(folio); - return; - } - - local_lock(&cpu_fbatches.lock); - fbatch = this_cpu_ptr(&cpu_fbatches.lru_deactivate); - folio_batch_add_and_move(fbatch, folio, lru_deactivate); - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_deactivate, true); } /** @@ -756,22 +734,11 @@ void folio_deactivate(struct folio *folio) */ void folio_mark_lazyfree(struct folio *folio) { - struct folio_batch *fbatch; - if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) || folio_test_swapcache(folio) || folio_test_unevictable(folio)) return; - folio_get(folio); - if (!folio_test_clear_lru(folio)) { - folio_put(folio); - return; - } - - local_lock(&cpu_fbatches.lock); - fbatch = this_cpu_ptr(&cpu_fbatches.lru_lazyfree); - folio_batch_add_and_move(fbatch, folio, lru_lazyfree); - local_unlock(&cpu_fbatches.lock); + folio_batch_add_and_move(folio, lru_lazyfree, true); } void lru_add_drain(void)