From patchwork Thu Oct 31 08:13:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13857703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6885DE68944 for ; Thu, 31 Oct 2024 08:14:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F172B6B008C; Thu, 31 Oct 2024 04:14:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC7E66B0092; Thu, 31 Oct 2024 04:14:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8FE26B0095; Thu, 31 Oct 2024 04:14:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B47576B008C for ; Thu, 31 Oct 2024 04:14:11 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3C21C1C70BA for ; Thu, 31 Oct 2024 08:14:11 +0000 (UTC) X-FDA: 82733183904.22.8627FF7 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf18.hostedemail.com (Postfix) with ESMTP id DD2531C000C for ; Thu, 31 Oct 2024 08:13:56 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=NYTIVbHL; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730362235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=tkQ9WZUCsC8z89qOXEr8b5vtEe7JN52Ek+ujHmE9f44=; b=boPK68CyrCBygUXSAFh1AThtcuAAKsXFRE2EYwskQrtMkFHwESYzDjpCa/Xp5dvyBScPIE oFttLGt+RykfRLIelPhlQDA+J3bAI+Sb52KDG2w3oUF+zvzzsC3ErJFKNQ4nl9SzOhMBTk kEyNkBAi9zR2CX+Ajyg3iP+ymk7KgUY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=NYTIVbHL; spf=pass (imf18.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730362235; a=rsa-sha256; cv=none; b=ps+yoQOgqA/GoZL/gUjx0KsMfMnOXhmjV/g3pgz4IC+M2SNBPyJemisswfTZQhPE/CvBjd N61iU9IfJw8IeSVGWMIOrIievDu7zyPPej+CZhMyTu+kChE5s0PqYmwrLmaVqdKlb5/blV kBqWHQM4hLKVIMmny5Oasm+KzDIdilI= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-20c7edf2872so13543055ad.1 for ; Thu, 31 Oct 2024 01:14:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1730362447; x=1730967247; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=tkQ9WZUCsC8z89qOXEr8b5vtEe7JN52Ek+ujHmE9f44=; b=NYTIVbHLT6RCyxKA3seDtuJiXP9forvvqj3xWWGYAOI6RH/aGJ8dwd+7K5/HLmtyIX dvxz6tTPwzE24F4iXWOHeJQj+9SES1zesGtJt0COLuziBLf+Mzf9mBG2aIwMjTE4tsT/ DTBAnPCQTuwSR63DMUzHUWyelLD0kko2IXSI8/gYw1mOu7pFp3eA3GN9qLtraxpALsQC NXlRPH5vVVqfCG44z+Vu+zLrCxY/f/LGLy4LRQ1HX9SrPKNBhigtHu5fLHU5U7iabuY6 Y1OS7XAEnudtPWI46LHbvFyPdBfeOz7B6higChZ1aQvVy6tC2T5oTrGrhpah+rFcEgHr nDsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730362447; x=1730967247; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=tkQ9WZUCsC8z89qOXEr8b5vtEe7JN52Ek+ujHmE9f44=; b=iuIm55t6AmXp0vDFRyApk1BHhBEVtwfdFt+5PhD4f5chWO85sjM+7r/9VKGGTx/Y8n JEWFFRMhN6Qx5d8Wx27+Uw9tWpLJM/OXAEczQ/Y7MPmvJXkLJnxkn10F2etAVK6ub0vG Gw6Uy8xLwRbBJHNAkLuJYDZ7JUWqx3pc7FUtvlxQ8u9jthwa+7/sh/4piig0OKPF0Pnn ln+vOCiExPp+pSy68Ve6Uu/Kq2R/1Lm1NHU+1FKD5cVLeaFqKOt5x0icVwOYf1yvlx44 poVQHmsLBmlsJaE6c9aCi7HI8IaZk9KooVSrMXOCDqq0OsBEkmEgc+V/K7T7bfJOEkCz Ac+Q== X-Gm-Message-State: AOJu0YwgEns00ap6T2QlT56p7exchy0jr6YJ9YZU/g4uFgs99WmYg4Ez Whece8eRc9ADKUCbBcTxNBikAfesEuY4s6yAcdwxvHjaEJtZALigiJQL3QUYKV4= X-Google-Smtp-Source: AGHT+IHNqTGZLYppxBamLvQthBLNg7bIYE2zQEjxS+ujin+/siUJIbRU2Fk3XijmqC60zqrcFDmIrA== X-Received: by 2002:a17:90b:1d0a:b0:2d8:b510:170f with SMTP id 98e67ed59e1d1-2e93e0e5ee5mr1974810a91.20.1730362447078; Thu, 31 Oct 2024 01:14:07 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.149]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e9201b2586sm5163996a91.0.2024.10.31.01.14.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 31 Oct 2024 01:14:06 -0700 (PDT) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, mgorman@suse.de, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, zokeefe@google.com, rientjes@google.com, peterx@redhat.com, catalin.marinas@arm.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, Qi Zheng Subject: [PATCH v2 0/7] synchronously scan and reclaim empty user PTE pages Date: Thu, 31 Oct 2024 16:13:16 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DD2531C000C X-Stat-Signature: hzsk3muouje41ity3qhi3mocs9zpcnit X-Rspam-User: X-HE-Tag: 1730362436-646081 X-HE-Meta: U2FsdGVkX1+r3hPyJH9Ca7HHK+p8J/dlAMkgF0Eoi9FQMNAUs4IrI8U2D19wBWmuql3omukjtAG3ZZb4bhIMsOU9i7jlTOfyq7EmUjFVDzp4yX/IYAiSkh6CBYsNtdPoW6j9WucW49jQX88LGcHIsSW5pSImaqbShNVnPhbWpUbUUwPtYSUjXcYMQt9msYPaHZGGHVd3nOk1sLNMWgxWbTDuYfz21e+PcxWSCKYjEOjcQc8WEv4dS5wrznNX3RNoE/BzYGprxlQw36X9JayYBlJnUxgoityNMQrVzWqjsxsDYu/ntPJG+7RoFIZqh+G41PeIxRUn65F3XE1YQ3aWw8iqUS7LzJtEyp5U6YR8AbUvx5FiPHdmO4TzjJX7Lt41I+rrY/YaiAUIlxvj9gx0RWp9HW0ffFPlg4rqmxSHye3huu3DnPQEqE4gnb1rVflbzsXV/JXRwZRdYVlj43vAMrAYHNyzPHiFti3keKEXJRTc0E9mGG12aj7NmEKMbCd0RAPcvP4+6swmvphBNa1/d/nQHg2TMmNAQ+m9x+BCI4Hoqjf849dn7CFRfLSSOgaij8Lafyp2fCSM79/lxC9QyP5VBwkOXmeorzLMatZrz5tJlfU5mh4MyRAyWKq7vwLlY2qr/njMpmN1hgQWfMb7EWxJS6njyAmFt0VmhLmUW5BolfJ4GWP3ZYuLFRQBzSHkxrxiP5xv2FJgTz/6WlScaVMufVqhA7hVTD1fSZ94hth8X604jEJQMu9n+wWMBJ44qy8agdRqRMwJmDygAn2X2jTr7QkyCg7VFO5u068A9X0kbLMSVW2k3rZtbHYXrLvRif/wBLQrKIHa2p9w1EIj7Ylp+xq3dsDdx01bhieaMe2Bo1O1Rn/QdQvsB7NfmRglk3FEjhlC/fUGSdcKlEY+RX/Rc2cOzpVjAajDR5A3IZAlQAuomx7KINGv7ZdLpvx/OqRGCcV7wRlJ61Z3a+N cPAht17B 28fxPAKwkY7UIy5XnI0g5iNZOb2flSPe7HkXC7nsbSCvkT7X6LEyj3UhWwb+hrKX6gXwYcWlkb4swhBk3Z68EKVSg2AKhGLjB31tGTlhxQUH6PChqF7w6GE5afPNbpcyIy0KV4tqRBtk7/pvRNERSb9lYTiCHL1jMW9V3yuoTA5Fugr7hQWwuD4uW5dIp7j6asDJJwKXvohYSTKokAl4K7LQL1tUqtgueoXndqwYBHT5ane9tiqWm5j7iw0j3hwMeep0Vvc6rBe5I09Vrv0oiwhhkdVsK1mtPWwlNf8AxS89L/3tILEVkfhNDK59EK0k6LsV9z1ESIkapyY9rNOUpxKiPIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changes in v2: - fix [PATCH v1 1/7] (Jann Horn) - reset force_flush and force_break to false in [PATCH v1 2/7] (Jann Horn) - introduce zap_nonpresent_ptes() and do_zap_pte_range() - check pte_none() instead of can_reclaim_pt after the processing of PTEs (remove [PATCH v1 3/7] and [PATCH v1 4/7]) - reorder patches - rebase onto the next-20241031 Changes in v1: - replace [RFC PATCH 1/7] with a separate serise (already merge into mm-unstable): https://lore.kernel.org/lkml/cover.1727332572.git.zhengqi.arch@bytedance.com/ (suggested by David Hildenbrand) - squash [RFC PATCH 2/7] into [RFC PATCH 4/7] (suggested by David Hildenbrand) - change to scan and reclaim empty user PTE pages in zap_pte_range() (suggested by David Hildenbrand) - sent a separate RFC patch to track the tlb flushing issue, and remove that part form this series ([RFC PATCH 3/7] and [RFC PATCH 6/7]). link: https://lore.kernel.org/lkml/20240815120715.14516-1-zhengqi.arch@bytedance.com/ - add [PATCH v1 1/7] into this series - drop RFC tag - rebase onto the next-20241011 Changes in RFC v2: - fix compilation errors in [RFC PATCH 5/7] and [RFC PATCH 7/7] reproted by kernel test robot - use pte_offset_map_nolock() + pmd_same() instead of check_pmd_still_valid() in retract_page_tables() (in [RFC PATCH 4/7]) - rebase onto the next-20240805 Hi all, Previously, we tried to use a completely asynchronous method to reclaim empty user PTE pages [1]. After discussing with David Hildenbrand, we decided to implement synchronous reclaimation in the case of madvise(MADV_DONTNEED) as the first step. So this series aims to synchronously free the empty PTE pages in madvise(MADV_DONTNEED) case. We will detect and free empty PTE pages in zap_pte_range(), and will add zap_details.reclaim_pt to exclude cases other than madvise(MADV_DONTNEED). In zap_pte_range(), mmu_gather is used to perform batch tlb flushing and page freeing operations. Therefore, if we want to free the empty PTE page in this path, the most natural way is to add it to mmu_gather as well. Now, if CONFIG_MMU_GATHER_RCU_TABLE_FREE is selected, mmu_gather will free page table pages by semi RCU: - batch table freeing: asynchronous free by RCU - single table freeing: IPI + synchronous free But this is not enough to free the empty PTE page table pages in paths other that munmap and exit_mmap path, because IPI cannot be synchronized with rcu_read_lock() in pte_offset_map{_lock}(). So we should let single table also be freed by RCU like batch table freeing. As a first step, we supported this feature on x86_64 and selectd the newly introduced CONFIG_ARCH_SUPPORTS_PT_RECLAIM. For other cases such as madvise(MADV_FREE), consider scanning and freeing empty PTE pages asynchronously in the future. This series is based on next-20241031 (which contains the series [2]). Note: issues related to TLB flushing are not new to this series and are tracked in the separate RFC patch [3]. And more context please refer to this thread [4]. Comments and suggestions are welcome! Thanks, Qi [1]. https://lore.kernel.org/lkml/cover.1718267194.git.zhengqi.arch@bytedance.com/ [2]. https://lore.kernel.org/lkml/cover.1727332572.git.zhengqi.arch@bytedance.com/ [3]. https://lore.kernel.org/lkml/20240815120715.14516-1-zhengqi.arch@bytedance.com/ [4]. https://lore.kernel.org/lkml/6f38cb19-9847-4f70-bbe7-06881bb016be@bytedance.com/ Qi Zheng (7): mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() mm: introduce zap_nonpresent_ptes() mm: introduce do_zap_pte_range() mm: make zap_pte_range() handle full within-PMD range mm: pgtable: try to reclaim empty PTE page in madvise(MADV_DONTNEED) x86: mm: free page table pages by RCU instead of semi RCU x86: select ARCH_SUPPORTS_PT_RECLAIM if X86_64 arch/x86/Kconfig | 1 + arch/x86/include/asm/tlb.h | 19 ++++ arch/x86/kernel/paravirt.c | 7 ++ arch/x86/mm/pgtable.c | 10 +- include/linux/mm.h | 1 + mm/Kconfig | 15 +++ mm/Makefile | 1 + mm/internal.h | 23 ++++ mm/khugepaged.c | 17 ++- mm/madvise.c | 4 +- mm/memory.c | 228 ++++++++++++++++++++++++------------- mm/mmu_gather.c | 9 +- mm/pt_reclaim.c | 68 +++++++++++ 13 files changed, 319 insertions(+), 84 deletions(-) create mode 100644 mm/pt_reclaim.c