From patchwork Mon Jan 29 17:54:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13536157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EF60C47DB3 for ; Mon, 29 Jan 2024 17:55:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 288E66B0081; Mon, 29 Jan 2024 12:55:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 238996B0082; Mon, 29 Jan 2024 12:55:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08BD96B0083; Mon, 29 Jan 2024 12:55:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E7C636B0081 for ; Mon, 29 Jan 2024 12:55:21 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B7ABB160A23 for ; Mon, 29 Jan 2024 17:55:21 +0000 (UTC) X-FDA: 81733100442.30.4894C3A Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf08.hostedemail.com (Postfix) with ESMTP id D5324160016 for ; Mon, 29 Jan 2024 17:55:19 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xh0igg0c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706550919; a=rsa-sha256; cv=none; b=XLJlQvPf+rmHHTEovD1DFtbT7q/YjuN3WZ74mkdzWVpZXqb+38dv2G+4ul+O2nWDjKPe/K vhI0fYgjtoTBx0FlLz0h/SI5AIkhJj9XYNXnDplbWZYrycUNqdVGnNJ3tuqMO+g6i+ZaAT xYpFjvLSPEw+M9a/LTzz83wBSzYu7+0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xh0igg0c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706550919; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZEaJsg6FaNgmWF+uyO/aaFj/erl/Uc0eXE/63t3Htcg=; b=B76+H8kU+S6ZUC8zKGIdxUItE+ugksML7ZKaRcpkeoU6Gz6wesaAVInlSVUaOyDJVWKvrQ 0EKjgub0qDMcsCxJJ3hXmXpsOj5w7f9dY4kXOtfnQ6fgyPTWnhZU+SxXFfTb4SiRlvikVL RXTwrKO8jH/4Il/JZZdk5ycpKQxIGX4= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6dddc5e34e2so2065607b3a.0 for ; Mon, 29 Jan 2024 09:55:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706550918; x=1707155718; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZEaJsg6FaNgmWF+uyO/aaFj/erl/Uc0eXE/63t3Htcg=; b=Xh0igg0clZJayq0Ujr3eantOKRTW7MqEfCpi0pJCTsbmzy+WZzKXnG9kk/ODvP6974 3yT2LKcDjBxizV5ncbneeAuRmsYh9aysUClPo7YzuFf05EeXKSH5sdeoBeE/I0XNZoMF 5LQe1rsjP9H4T3JdLiWqv+KDzWgIvmKvi8dtup6egNGXx45oR6hDWXsaF4fAiE3wzF0D ORMskrypTIv+jkzT+v2deQEi9F3zMIGy2EZlImdqogrZ/DkNKcq6Jey75VWhWSqRYxBD yI4yhm6KhGI1nmTFmX+2RoYpV74/DdiDNDDcydiDc67gydyU89XGf3TDMLEjiBVg9g6e Ez+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706550918; x=1707155718; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZEaJsg6FaNgmWF+uyO/aaFj/erl/Uc0eXE/63t3Htcg=; b=wuaHvc5KBBCWYUF/ptL+/lOIzQDd0oJXor1fKPrW82hAXLnhS3bDetBTpm7Eg07BF6 ftDT3DOwsUm2oOZ2REGjbPgGUGqHdTU0NE7K9OENp7KWYxnyV3f+QE00Xhzvhrj5Rg9y Vs/aQmfV09qs41/Rf0rGZciFk/Se1tju/JN2FsGbtsAy9fyN/w0yly+UA1RejIbQbzFi PtJD2AZ+rBM2Xe3rIt9YcZVarYavMqf2lbgbXf6SaHB1VzP8Ds+tHe4cOg3RK97xAhWD 9cXerHSxrD2UudYwwnDTVlKt8LO99fKeD9pX+stY3LcyvNOvCYcznber/VTtFhGqVdkI 8nWg== X-Gm-Message-State: AOJu0YyLdSR9daTmvOMO8Z+Nck2j+Ax2cC8tq+e93mcqG2KMmWSpOxjg KyBLzv6Vq3bhHEkA5/RjcSQfYpE+bdtaqgEjEldRMySGQRGrgldOHT1EtZ8gf9pD9Q== X-Google-Smtp-Source: AGHT+IHYv2LHBpglkk4ViPSn6YlurzgUjDbn2zS3N+K1h38tw1crAQqtxAPY8gv9wn+50sarKhsTLA== X-Received: by 2002:a05:6a00:2d96:b0:6de:2470:92e1 with SMTP id fb22-20020a056a002d9600b006de247092e1mr3531769pfb.15.1706550917544; Mon, 29 Jan 2024 09:55:17 -0800 (PST) Received: from KASONG-MB2.tencent.com ([1.203.117.98]) by smtp.gmail.com with ESMTPSA id h8-20020aa79f48000000b006ddcadb1e2csm6116676pfr.29.2024.01.29.09.55.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Jan 2024 09:55:16 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , "Huang, Ying" , Hugh Dickins , Johannes Weiner , Matthew Wilcox , Michal Hocko , Yosry Ahmed , David Hildenbrand , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 4/7] mm/swap: introduce swapin_entry for unified readahead policy Date: Tue, 30 Jan 2024 01:54:19 +0800 Message-ID: <20240129175423.1987-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240129175423.1987-1-ryncsn@gmail.com> References: <20240129175423.1987-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D5324160016 X-Stat-Signature: p5famw3aoxtbqhb9dmbu4onjtuqyjipe X-HE-Tag: 1706550919-60004 X-HE-Meta: U2FsdGVkX1+bospvjMeUERl+pE1GBK+dsW4tWnQh+uAzOCJdTMNU9cus1y63s4x5hFE+zG3MLwyvYVaznrHINrX+v9tnTOEiJxyorsvU8wc/M9jDZ7uXLwVxSl5J3VBEd8znV7VtN7kBSo1bd49O3MU8SWtrakqzFN8x2JsYbyUMfl/DdYfkZrS9hLlb1HuVz7udLSR83grDVBmteVYGnx4oRro6RDKGk90GuNvo5lQTTsBAY1JrDVbWX9EMp5Z8OJPkC/tVk1+Av7nVlE08LChHnnpU5xMCfvwfaUDENcUWCQCvqKkaYTU2/xTyx7UlGB440jsm4lsCPKOzrMZ6rM7u0lrsfS8U7/h19CcO4TBIc7a9nXd47gvd0VwVYgqyKd488ymx7zKdQxmc3IWkaKzMRZQjhEQOyRkXGXroBLpbi5HTrhj4LS7qPshPktbB4ZacvkyoeNcSe10Y+G6OgEoUH9gDH00+bqB71GV6VAIa87KD8siQ11MnH1qOdvjOmloBFqHo2d370G372QinVW8QlXByD7xIqPAP47zr/1NJwvy389C33CrauqIDOM2yFPVPLUeL2zg9QPzRjkfTPd7T5Fw11UfhCeKJapdmrS/OXT83LtBo8xv1mpUbB4tedYucPPuapOuHSFGDGavCn4bIdot1lu2h2SbCYR9HXQ4rv3GiLOnT7E7QnO/VdNa93Gi9Qgb1G5VavFlABLI3RJAZjmoPWFkpqjgwpxOVmsuosx4sa6alZEFwn+z3HLcsE4ylSf1QojafOBjgFTL6MRdqWbERg335MA6yX3A/1RBCNLZY5vjms4vH75rd/W6mci7YbTOkIwpjyvjmaK9itdR7OWQhRW7gRXxR0oww9bfOSVTVp1r3cq1PU+FqBStVC4a6yE6S/LBALxO1A+taGnQ7vt321BSLdE9NYlkl9cOWa1G8OcBOqnSpwnfy/ca516i/WoqiUebSn0Otg0n 3UoxxsDh k/cMgFshYdCb1HUv32rjN0VLtlHI+AoNSjRMLpx7FtZwI5FrFX4idD2PEbh3rA/b02InqcQJlLMdl9DIXs6Ys1tMWATxRXZpsTp0YduPx5KGLMgJYv0rGheX6olYR8TcaviyC6iEusQfYOAA3YBWCCk7dyygo7m08Y2HZUabdChy94IteIWoyLHAFC/IoWR9g1qaA5OIz+5AtolQ3xCjMmU5lNw3LWRa5jNFr+iaqanO2RvBbNa2ukWKb5iJzmGUNo7pWpfJ2fL9aRc368YWuUBFa5hwHOOLjPJN3jfsjGDGN9+BwSwt4O6xn8CEFGZA4U10/hIGnboIcVf1nFsNRC38tcuLmJM9XsjwK/xJToO0bLn9zScI8qM8k4YGZQVMcM4gw5hPQX3ioI9xH0rMlg4j9BGhutk62D/CSeb8fR43uqlYJun4HF51i5TSewUpdoc+WWgETPnQJarSx0h0EBXaF2gx/HKThiNKHHdlvZ66riuweZR525zNT8G59RgNcIXnARAsjW783V8nlUi5oyIdbh7Yza+fSMKOPg4oKeCmqzMC+HgbUp9X3P8Ytjv3PTHkDazwXNB23YjOWwSvy1IGpYtd7+9n0zV1aDt2ahIedC0RPVc/0q9w21cEr0RUmQ/UjtXlVCfJzWg5B0ecHQSOTY2j2g6f5IUs+HacODdDjxms= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Introduce swapin_entry which merges swapin_readahead and swapin_direct making it the main entry for swapin pages, and use a unified swapin readahead policy. This commit makes swapoff make use of this new helper and skip readahead for SYNCHRONOUS_IO device since it's not helpful here. Now swapping off a 10G ZRAM (lzo-rle) after same workload is faster since readahead is skipped and overhead is reduced. Before: time swapoff /dev/zram0 real 0m12.337s user 0m0.001s sys 0m12.329s After: time swapoff /dev/zram0 real 0m9.728s user 0m0.001s sys 0m9.719s Signed-off-by: Kairui Song Reviewed-by: "Huang, Ying" --- mm/memory.c | 18 +++--------------- mm/swap.h | 16 ++++------------ mm/swap_state.c | 40 ++++++++++++++++++++++++---------------- mm/swapfile.c | 7 ++----- 4 files changed, 33 insertions(+), 48 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 81dc9d467f4e..8711f8a07039 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3864,20 +3864,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swapcache = folio; if (!folio) { - if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && - __swap_count(entry) == 1) { - /* skip swapcache and readahead */ - folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); - if (folio) - page = &folio->page; - } else { - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - vmf); - if (page) - folio = page_folio(page); - swapcache = folio; - } - + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + vmf, &swapcache); if (!folio) { /* * Back out if somebody else faulted in this pte @@ -3890,11 +3878,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) ret = VM_FAULT_OOM; goto unlock; } - /* Had to read the page from swap area: Major fault */ ret = VM_FAULT_MAJOR; count_vm_event(PGMAJFAULT); count_memcg_event_mm(vma->vm_mm, PGMAJFAULT); + page = folio_file_page(folio, swp_offset(entry)); } else if (PageHWPoison(page)) { /* * hwpoisoned dirty swapcache pages are kept for killing diff --git a/mm/swap.h b/mm/swap.h index 83eab7b67e77..8f8185d3865c 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -54,10 +54,8 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, bool skip_if_exists); struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); -struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); -struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf); +struct folio *swapin_entry(swp_entry_t entry, gfp_t flag, + struct vm_fault *vmf, struct folio **swapcached); static inline unsigned int folio_swap_flags(struct folio *folio) { @@ -88,14 +86,8 @@ static inline struct folio *swap_cluster_readahead(swp_entry_t entry, return NULL; } -struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, - struct vm_fault *vmf) -{ - return NULL; -} - -static inline struct page *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, - struct vm_fault *vmf) +static inline struct folio *swapin_entry(swp_entry_t swp, gfp_t gfp_mask, + struct vm_fault *vmf, struct folio **swapcached) { return NULL; } diff --git a/mm/swap_state.c b/mm/swap_state.c index a450d09fc0db..5e06b2e140d4 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -870,8 +870,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, * Returns the struct folio for entry and addr after the swap entry is read * in. */ -struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) +static struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; struct folio *folio; @@ -908,33 +908,41 @@ struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, } /** - * swapin_readahead - swap in pages in hope we need them soon + * swapin_entry - swap in a folio from swap entry * @entry: swap entry of this memory * @gfp_mask: memory allocation flags * @vmf: fault information + * @swapcache: set to the swapcache folio if swapcache is used * * Returns the struct page for entry and addr, after queueing swapin. * - * It's a main entry function for swap readahead. By the configuration, + * It's the main entry function for swap in. By the configuration, * it will read ahead blocks by cluster-based(ie, physical disk based) - * or vma-based(ie, virtual address based on faulty address) readahead. + * or vma-based(ie, virtual address based on faulty address) readahead, + * or skip the readahead(ie, ramdisk based swap device). */ -struct page *swapin_readahead(swp_entry_t entry, gfp_t gfp_mask, - struct vm_fault *vmf) +struct folio *swapin_entry(swp_entry_t entry, gfp_t gfp_mask, + struct vm_fault *vmf, struct folio **swapcache) { struct mempolicy *mpol; - pgoff_t ilx; struct folio *folio; + pgoff_t ilx; - mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); - folio = swap_use_vma_readahead() ? - swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf) : - swap_cluster_readahead(entry, gfp_mask, mpol, ilx); - mpol_cond_put(mpol); + if (data_race(swp_swap_info(entry)->flags & SWP_SYNCHRONOUS_IO) && + __swap_count(entry) == 1) { + folio = swapin_direct(entry, gfp_mask, vmf); + } else { + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); + if (swap_use_vma_readahead()) + folio = swap_vma_readahead(entry, gfp_mask, mpol, ilx, vmf); + else + folio = swap_cluster_readahead(entry, gfp_mask, mpol, ilx); + mpol_cond_put(mpol); + if (swapcache) + *swapcache = folio; + } - if (!folio) - return NULL; - return folio_file_page(folio, swp_offset(entry)); + return folio; } #ifdef CONFIG_SYSFS diff --git a/mm/swapfile.c b/mm/swapfile.c index 606d95b56304..1cf7e72e19e3 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1867,7 +1867,6 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, folio = swap_cache_get_folio(entry, vma, addr); if (!folio) { - struct page *page; struct vm_fault vmf = { .vma = vma, .address = addr, @@ -1875,10 +1874,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, .pmd = pmd, }; - page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, - &vmf); - if (page) - folio = page_folio(page); + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE, + &vmf, NULL); } if (!folio) { /*