From patchwork Mon Apr 8 12:14:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhaoyu Liu X-Patchwork-Id: 13621035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFD28C67861 for ; Mon, 8 Apr 2024 12:14:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 627E46B0088; Mon, 8 Apr 2024 08:14:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D85E6B0089; Mon, 8 Apr 2024 08:14:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49F6B6B008A; Mon, 8 Apr 2024 08:14:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2BD426B0088 for ; Mon, 8 Apr 2024 08:14:51 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B1B1E120AC6 for ; Mon, 8 Apr 2024 12:14:50 +0000 (UTC) X-FDA: 81986258340.13.AD8290B Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf08.hostedemail.com (Postfix) with ESMTP id EBCFE16001A for ; Mon, 8 Apr 2024 12:14:47 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YYLvEEaw; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of liuzhaoyu.zackary@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=liuzhaoyu.zackary@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712578489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=PkNlcb91n/5YmjH6EI5qNjc9lwvi5OJF+TEyy7RKmos=; b=aI0yx1AOpDLdGn4sxidT+LBRBEt8O2TabTfJbbDBEiryJT6WCFjaKbfHJEX7um04V5tKF0 6eu4TY+839I10lTxTRkO+uStcFEtFXZwOTNEXnD34bjooOnaEFXnlubdgoSkoEI3HmEtLU vtLUf6nNhlT3btp4//amUspBk3VA2IA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=YYLvEEaw; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf08.hostedemail.com: domain of liuzhaoyu.zackary@bytedance.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=liuzhaoyu.zackary@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712578489; a=rsa-sha256; cv=none; b=DYJyElYz9wpxP0TNHqQ3S5oCTYhgiuLVSYu2DkFexx339XZE3l6/COsrf5hnlZw0ZdzHmh hWzvz7TueG4TBAqeU/zcGQHoPloXpkawO9tRQ+mp/0cR4pVHM0NDF+iXVNcqYwqCaO8Gt0 QVgSNxzl0yFU1bWvkueuCH1ZhyqPz1E= Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-29e0229d6b5so3564503a91.3 for ; Mon, 08 Apr 2024 05:14:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1712578486; x=1713183286; darn=kvack.org; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :from:to:cc:subject:date:message-id:reply-to; bh=PkNlcb91n/5YmjH6EI5qNjc9lwvi5OJF+TEyy7RKmos=; b=YYLvEEawrukRIjab9slSzTkovmzkrLx82JwDBx6Z7ca+u6/jEYHbmMRAkbBs5RLLqM 9vU+83VoZdl4j3nfIUk7IXoHa5cqZHNjNWMH2B+y+xo+voloyEtk0jiwC9g+hoZxw/YF B+ikxhrdtVHJINfuAciwQTGKjN84PnagkAok+N2PKMrZNndcwYvqaEgD3CKay4BvLc2V /3ljaCYCgxEvnEXF2O3QMo3F+4sfXAx7l1Ohq3U+kHQoWLiC4kChj4eHngm74sgN4qda ZKExHZrVRmGy6RoMYVtVva/maC2AUA8dVhy2Uu3hz7sTsHMLAvAVMXvgsydFq6BqzN+v psrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712578486; x=1713183286; h=content-disposition:mime-version:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PkNlcb91n/5YmjH6EI5qNjc9lwvi5OJF+TEyy7RKmos=; b=GmJoqAdUMMJ+K1hZW4yI8OYUNNHftLixzaeNHI+BBIJj6CpBv8w+QONlHxkjOvUuPs N8ZOZXv9d8LidIDq6Dx/VDavbU9hgVaDvmiGeDrlsKdYarNyqR/fZyDSpAOmi2KQ+tQ1 6pmiyWt8ZCww+p9tiR1OfP4z/qtu3LIbRsgXtDRelRhzzskHnUjFZkeb9QglJG7mCV1t h332fx5SK5u15mX0NM0vgbg1CTHcko7YXM4oJIZZqYjT7FXL9IrzOi1rZeAyK7Sxnlvz bawSoMqcZlyzD3uC8B6qgSWzT3KXqifXyINl31hnd8r3qMJ7KrXoguziO/xXIX30ybcn cjxw== X-Forwarded-Encrypted: i=1; AJvYcCWGls9ndvS1STSnEIihSZLp46LmXqJDRsrgcHwIKirbg2lifyB7Om5tM2+N41il6ezrjVlcTySIAv/efB+iSFt7Ulo= X-Gm-Message-State: AOJu0YxmI9bimob9gD+UOxUkfgacliKDU5eJj9Awwmq2eVUfPEzL0cJu SJoGjpjqxjsJb+yknj+GQ95Fy8Nt0+ioB0rjZ8x9x9iqkASmPG6/b8cZan+scW0= X-Google-Smtp-Source: AGHT+IHZn4xNPFft1trJmRLs125/y+NhZFRZEO2EJIN/Viow45Qbc63czU6NPOJZBdzamMEkO23U5g== X-Received: by 2002:a17:90a:43c4:b0:29b:c9ac:c563 with SMTP id r62-20020a17090a43c400b0029bc9acc563mr6824401pjg.19.1712578486691; Mon, 08 Apr 2024 05:14:46 -0700 (PDT) Received: from bytedance ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id x1-20020a17090abc8100b002a28f39db9asm6268087pjr.53.2024.04.08.05.14.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 08 Apr 2024 05:14:46 -0700 (PDT) Date: Mon, 8 Apr 2024 20:14:39 +0800 From: Zhaoyu Liu To: akpm@linux-foundation.org, ryncsn@gmail.com, nphamcs@gmail.com Cc: ying.huang@intel.com, songmuchun@bytedance.com, david@redhat.com, chrisl@kernel.org, guo.ziliang@zte.com.cn, yosryahmed@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2] mm: swap: prejudgement swap_has_cache to avoid page allocation Message-ID: <20240408121439.GA252652@bytedance> MIME-Version: 1.0 Content-Disposition: inline X-Rspamd-Queue-Id: EBCFE16001A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ggahedeunemekm8nqamydqi7ehiqa8wm X-HE-Tag: 1712578487-349850 X-HE-Meta: U2FsdGVkX1+rYGDMccXv+msriRSJIQ8kn0u/HaHkuiSah0Nt8dm30n61RQrNOUR4/ZRRcHzt87ief4jK+c2TA4LzJdHkeJA+2JYWJBYgtJMWrrP7uyP2rrGVAxmFGzWmJFUb/5LJeB+QTPbuo3mH5Ch9q9xIyrxnfCH7YsR6+PqLGHwJ71hqEN5W5hNc7stCXVuuHnkKXEBCGGhl1OOviTLfXaOdANTGVgEDWwFs0voQzr5ni+GNdPMuArfjHcmZn31777gb4G1TB0bL2Inz0UAO6b3XUrqkVBEnmv5v6/ABGQ8FRZstr/YarJO1BNdYU+oIPwE4swR88Jyck6B28k30SV2g1mUzCYtjpTWvR61Y9WsBFvxYVna0Ztr7qfmyTdl00s7eKbebYv4mnKamKaiPxM5ZYGgxNWiqdEdLla6l1b7m6I5qcosbyzHcaChAYr9cSn34lHPxHZiYlbW8q5TeMmP3Mazr8taPjtd6NpV6EDZa1oKq3PTyplEqoI/5KDGsxJXHE0B/GYahO3ulVM2wX13l+6L42BP+qQG0Dp8B/nGBAMNGVAqo0yV0J+upuglnjyiJR4lydDbN6IgE7T4R1SBJmsEJiCskXOrxAmuO6RAVHrF1/TgtZaHTTf++M/GMD7lJlyo2Lp2DicoT22aoFtwPqvXzUghRChR+VqTr/BZp37FPTKk+qIS8hTuYIixB+rPXFnwWOP1aeyZXCRKY+mDS9jcxd5UIt1PaF1ODziEhNnA8cF/xbc2sYHTUSXKTPoijzcBsADfYDgdqdVybUoDSTVNl13W9d3yVRpbzq3GGL/xfKXl4jv2mt0qmHK6I6kx2GBjyCxhfK4i5lqpYyu3dzPGbgjnJk4p8z66hYj7zdE4HP4vuxlY+kJhfg26YLoEAUpUPX9qQMOcVxqatfEFhDodXA9PS8lrpb6yswrsYuxdmnSBgPEJ2SaGScnNNmE5jB3RIq37Ds6G M/3pSj3t WyF6wxfWkWpRcf9KhMZJp+bvkBShyXYrZfP214aOLtJpJRehz28nKiXMxqrMxOzFc3paQ91Vj8+Vvt+b/CL16oBH1zmb1cncBcTF8zgNx/lexuHIzHPMUIaEXzey6Tkztt9Y6xoExNvc/49eBKQ1ruxDgzNVx/HdSXfGhlvyFrkj0s48RQgb5usT1M0lYDkwIhj70Xxecv061Ao3EIkn180p7xmXTA6MTOHIhL4WxrHqwkCnEBJQpDu3miB8G87vUs8tWVLwYZauDeWomQ75m6bLEk+Q64ZbStzXTQX/L/AeLhSUIAGqHkqKhPavLyjeTTsKzg3BZA2qNW7Jkl8iScg0WURoPgwnRN1kRs3xsrapDZJoPWNekpDLrQtCGVVtMJ33jxxub/qJ+gAuxquiF3AiwoN75shRm0vxjDCq44pGD11z1TadTE7euMesqAKHrBW1UXMB6LfoxcnjgGfIoF4kfEQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Based on qemu arm64 - latest kernel + 100M memory + 1024M swapfile. Create 1G anon mmap and set it to shared, and has two processes randomly access the shared memory. When they are racing on swap cache, on average, each "alloc_pages_mpol + swapcache_prepare + folio_put" took about 1475 us. So skip page allocation if SWAP_HAS_CACHE was set, just schedule_timeout_uninterruptible and continue to acquire page via filemap_get_folio() from swap cache, to speedup __read_swap_cache_async. Signed-off-by: Zhaoyu Liu --- Changes in v2: - Fix the patch format and rebase to latest linux-next. --- include/linux/swap.h | 6 ++++++ mm/swap_state.c | 10 ++++++++++ mm/swapfile.c | 15 +++++++++++++++ 3 files changed, 31 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..a374070e05a7 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -492,6 +492,7 @@ extern sector_t swapdev_block(int, pgoff_t); extern int __swap_count(swp_entry_t entry); extern int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry); extern int swp_swapcount(swp_entry_t entry); +extern bool swap_has_cache(struct swap_info_struct *si, swp_entry_t entry); struct swap_info_struct *swp_swap_info(swp_entry_t entry); struct backing_dev_info; extern int init_swap_address_space(unsigned int type, unsigned long nr_pages); @@ -583,6 +584,11 @@ static inline int swp_swapcount(swp_entry_t entry) return 0; } +static inline bool swap_has_cache(struct swap_info_struct *si, swp_entry_t entry) +{ + return false; +} + static inline swp_entry_t folio_alloc_swap(struct folio *folio) { swp_entry_t entry; diff --git a/mm/swap_state.c b/mm/swap_state.c index 642c30d8376c..f117fbf18b59 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -462,6 +462,15 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, if (!swap_swapcount(si, entry) && swap_slot_cache_enabled) goto fail_put_swap; + /* + * Skipping page allocation if SWAP_HAS_CACHE was set, + * just schedule_timeout_uninterruptible and continue to + * acquire page via filemap_get_folio() from swap cache, + * to speedup __read_swap_cache_async. + */ + if (swap_has_cache(si, entry)) + goto skip_alloc; + /* * Get a new folio to read into from swap. Allocate it now, * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will @@ -483,6 +492,7 @@ struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, if (err != -EEXIST) goto fail_put_swap; +skip_alloc: /* * Protect against a recursive call to __read_swap_cache_async() * on the same entry waiting forever here because SWAP_HAS_CACHE diff --git a/mm/swapfile.c b/mm/swapfile.c index 3ee8957a46e6..b016ebc43b0d 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1511,6 +1511,21 @@ int swp_swapcount(swp_entry_t entry) return count; } +/* + * Verify that a swap entry has been tagged with SWAP_HAS_CACHE + */ +bool swap_has_cache(struct swap_info_struct *si, swp_entry_t entry) +{ + pgoff_t offset = swp_offset(entry); + struct swap_cluster_info *ci; + bool has_cache; + + ci = lock_cluster_or_swap_info(si, offset); + has_cache = !!(si->swap_map[offset] & SWAP_HAS_CACHE); + unlock_cluster_or_swap_info(si, ci); + return has_cache; +} + static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, swp_entry_t entry, unsigned int nr_pages)