From patchwork Wed Jul 12 04:27:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13309557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D995EEB64D9 for ; Wed, 12 Jul 2023 04:27:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 240BE6B0071; Wed, 12 Jul 2023 00:27:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F16A6B0072; Wed, 12 Jul 2023 00:27:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 092E96B0075; Wed, 12 Jul 2023 00:27:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EE49D6B0071 for ; Wed, 12 Jul 2023 00:27:25 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B6C70404AB for ; Wed, 12 Jul 2023 04:27:25 +0000 (UTC) X-FDA: 81001675650.17.DFB9C1F Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf07.hostedemail.com (Postfix) with ESMTP id 0FD6440008 for ; Wed, 12 Jul 2023 04:27:23 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=YjT2iggk; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689136044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=wiHFwa+qInqC9Ufc7qLGqz2T4+XJI6K3K6pkdSvL2BE=; b=5/YzWwnXspZaVTM7ATovfqhtNQUT6v/NBlZ+LHe/LNbOMhTpQ1PsRkIBZKvoLFIBKXOTmV GSp2zsBap9g6o9PTWOHTmMKK0j0hhKFj9R1Zn6LIYOiF6MpAxbqUC6B+OC5AILC8TdcJH5 lFh5UprFqEYocC+9VKyG31scr8uzTVY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689136044; a=rsa-sha256; cv=none; b=cff6uYNaG7Gqka09ftw4BmViDUtNpZLOu3NKSVv3T50reCF+qCKHAPFI5nWI+srnQhQXB9 2o0CT2/pRHSEe/b9Vc1Q16DqEek8oYHb+tXrGKSiRtVWwBp9MTDqn4F0rebDY15aADd9fs m2yUbuQEHj+Ol5NFTyASRZ+8eugVIQY= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=YjT2iggk; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-57eb0df5265so1528497b3.3 for ; Tue, 11 Jul 2023 21:27:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689136043; x=1691728043; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=wiHFwa+qInqC9Ufc7qLGqz2T4+XJI6K3K6pkdSvL2BE=; b=YjT2iggkRTiXtZMR1jMLANF/DCx6aBNBHMVDVVl8Gr0GinPbSloyq6adfQMkbwXuDy 4LOhxp9lsDQkNYPLAl5k89Ck7UzmU2gS51lf58HBHMNkZxeBrwzXKOfpjPMpHU7Xhn8z K9b44iVOIzPUbQ6ymuTwqdYb3uxH5ZNNlppLByjfI27io4ZaUeNDCqifG61+Enltfq1W Q4VeG3xJt0Bk+AP9vkRr0OjwSSW1svxOnjAGiUGPKap1kin/J6ow8NcPSugD6S/XzNDL Pw7KzGLDDMHF/NqW9CZBI1VUfB0aeFWeRbgCBvTEtczkxjQmMIk8cik2x0kBWOU3eCTM DkUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689136043; x=1691728043; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=wiHFwa+qInqC9Ufc7qLGqz2T4+XJI6K3K6pkdSvL2BE=; b=Ro0fHkLYFS1DuDf9cYWNYQ8IXnljoEas5tmZBfC35fpA/S30qwb6kQdOWGmXfxQmVK J4b+fVU3sm1tl2YexET/o4WcVSk3ylPlWJ0GTSyJ640KjA/rXOEx6Hb2N7aKXgYNVkvD JzY1esdhWjff1lxeO5m06R5seeiTRzz2IzdnXyGDHMiv/BRafxZML9+XvTuuDn1/Xay9 dmPdVhqi9DQ41WnGFYuMZFvi33nCy5w2aveeV9s2RL66jPlOxVCF8q7aoal4KOlmhU1x HuFpZbG2TMmIYwubxnI/70wyLOu9BTX4Gb2Ng/rmdDjlM8F8+Zx08zZ0MEKWsTcbrjx9 KWfw== X-Gm-Message-State: ABy/qLajToGwWSFHGh+hvP3JkcO/sVpZ5bJ/qRuSC3PK6X1jDmhlmpPL 0X3I93k9fs8sha1qftVtmHi2lg== X-Google-Smtp-Source: APBJJlF5lEOtFts7UwfGvvEUizjbQpUSNw8Id5rb3Vpn8mt/ge/Q6rrPqXyVhuX/GSF3C2Mj1ANz1w== X-Received: by 2002:a81:91d2:0:b0:559:deed:f363 with SMTP id i201-20020a8191d2000000b00559deedf363mr16696703ywg.2.1689136042885; Tue, 11 Jul 2023 21:27:22 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id i67-20020a0df846000000b005772f3fc0acsm972965ywf.44.2023.07.11.21.27.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 21:27:22 -0700 (PDT) Date: Tue, 11 Jul 2023 21:27:09 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Gerald Schaefer , Vasily Gorbik , Jann Horn , Vishal Moola , Vlastimil Babka , Zi Yan , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 00/13] mm: free retracted page table by RCU Message-ID: <7cd843a9-aa80-14f-5eb2-33427363c20@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0FD6440008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ntigsscni9ags7ymns3qsxpjfeg6an1r X-HE-Tag: 1689136043-893828 X-HE-Meta: U2FsdGVkX19OYwGiDunjmGwl1i6+yxFHgmVH2bvyF5DisjkfBO8EV1hRr0PDafE7urKgsjNkLi36JEvPvhgW2gu0jb3T/nIB6kdFdDArCQuiNqw7IlocDt9UlMfZhEqyyx9IIwg2AmhVOOG7bWq3qc93GxaSx8LK1KRdnVcHR/vHb8a5vX0QwTcODkIXuvWZEuwkXMBTrSSU5HFaap1yytgVQH2TbbZ+YQFGZYktbzStT5hwq9KYlhZjnHUZY3UePG/7oJt60zld9Xt82j2OzG9eolbNNHdKvPdTzR6Av8VAq61YwXFtkO2g7rMkU8zbX5mcyaP0KgZVY4de5ZyKkwVlrsBTI93+nuqyRf1fIYLyaCK3FZdgHPsc8aQb8bksI16QBXl8WFn2PrjXKkHEFW3QsMe0g9DhG9XFh0I4QZZCarC64tkdMwCtX4C7Sz5SviMWdmLkjaCC/ZBYfvWc5B1DzUmsaW9wXyTEhMAdd7k68lcDuSKD9TzNccaW7VZfhUHWw27ILeiRMRrolIa7DwTrYRdhuOTcB2Jv6mw1waCARb1B4GHUV4EwQshrceMgJxonqTBknhLgqx2KPwzNURBKeW4wfgHqYcvmNuGm3RJcgkID3nnxuGASj5ppYgJ3k+UtdGsjJKRZYEZ7Ivb7IYzRpcR5hQLASJVgnul3v9ILLj0K2DKAuGd6a7t6hOolNkNLZ19znjVXzM0NR0GaGEu0bqca6kV/YFvgvFQL0EjHeuJJsQTKgj7/wv5zH4HA+ce6XzmOmJTVckWQLKGS2BC/6/PJD1twXcSP5BMzHtlqgZClG5o8IM+1GH4VMm4RxPMCwkIDO9xyRmn9+YIq0AQuOqwG/0zHsFO1tqO+AWmQVpFl/jvz70Ch7U0sHweP0e8ynnbV/vacGDEJWHXPvXoIQLiSagg1KLd+ZFwT11lAKLxbK5e50fw4obvA31wIhsu1EmpqUBuhsxpszpP y1x9QnV0 RPCwLmULks55GBqoj9DYYfDBQ4p+CYEYCLgzd+euE3E6sRcWLYWHoQOftmBmt7OzRm9O2+2c+KCFCOFB5AfN5PgjW/R9ztbCt1OXcuXeMSrDVh0tox4E/t2ciAE9gWf99eL1LHFwDRLBkj2y/qmak8kt2ppF9Y8dFfDRRKdij/MMLJTy1pdhUnKr/qICT+/PSTAJxzrxFCI/m6A3S/ukzEON6aJXaBD2sPf4AlurVuYpKUkjssXH4+isjcQPn93576GhGWIQ2ghZ1GFmgTxSZ81wdv2kLCn38OR6jbigg8pmUGCRJ+vDslGD2ixoUd5akmhQGeK4uI0YP9knctv1Sdrvc0oKhVFhB5r3grqgUR+a9RrgKjHPHobG80cgpFEnvuMdPrlqjvmaLfXCLDJNW1sJ+SO/gOug81+A1SYfbtStSGDDv1gvO96p+7EzxcTYygJSbKb4O5G2GbvyS9Ue3u/7XkItysXJ8jovBl876ZyaHkfmq4xPEGzZGIhyT/gzwrGbLOUKN46D98JI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Here is v3 of the series of patches to mm (and a few architectures), based on v6.5-rc1 which includes the preceding two series (thank you!): in which khugepaged takes advantage of pte_offset_map[_lock]() allowing for pmd transitions. Differences from v1 and v2 are noted patch by patch below. This replaces the v2 "mm: free retracted page table by RCU" https://lore.kernel.org/linux-mm/54cb04f-3762-987f-8294-91dafd8ebfb0@google.com/ series of 12 posted on 2023-06-20. What is it all about? Some mmap_lock avoidance i.e. latency reduction. Initially just for the case of collapsing shmem or file pages to THPs: the usefulness of MADV_COLLAPSE on shmem is being limited by that mmap_write_lock it currently requires. Likely to be relied upon later in other contexts e.g. freeing of empty page tables (but that's not work I'm doing). mmap_write_lock avoidance when collapsing to anon THPs? Perhaps, but again that's not work I've done: a quick attempt was not as easy as the shmem/file case. These changes (though of course not these exact patches) have been in Google's data centre kernel for three years now: we do rely upon them. Based on v6.5-rc1; and almost good on current mm-unstable or current linux-next - just one patch conflicts, the 12/13: I'll reply to that one with its mm-unstable or linux-next equivalent (vma_assert_locked() has been added next to where vma_try_start_write() is being removed). 01/13 mm/pgtable: add rcu_read_lock() and rcu_read_unlock()s v3: same as v1 02/13 mm/pgtable: add PAE safety to __pte_offset_map() v3: same as v2 v2: rename to pmdp_get_lockless_start/end() per Matthew; so use inlines without _irq_save(flags) macro oddity; add pmdp_get_lockless_sync() for use later in 09/13. 03/13 arm: adjust_pte() use pte_offset_map_nolock() v3: same as v1 04/13 powerpc: assert_pte_locked() use pte_offset_map_nolock() v3: same as v1 05/13 powerpc: add pte_free_defer() for pgtables sharing page v3: much simpler version, following suggestion by Jason v2: fix rcu_head usage to cope with concurrent deferrals; add para to commit message explaining rcu_head issue. 06/13 sparc: add pte_free_defer() for pte_t *pgtable_t v3: same as v2 v2: use page_address() instead of less common page_to_virt(); add para to commit message explaining simple conversion; changed title since sparc64 pgtables do not share page. 07/13 s390: add pte_free_defer() for pgtables sharing page v3: much simpler version, following suggestion by Gerald v2: complete rewrite, integrated with s390's existing pgtable management; temporarily using a global mm_pgtable_list_lock, to be restored to per-mm spinlock in a later followup patch. 08/13 mm/pgtable: add pte_free_defer() for pgtable as page v3: same as v2 v2: add comment on rcu_head to "Page table pages", per JannH 09/13 mm/khugepaged: retract_page_tables() without mmap or vma lock v3: same as v2 v2: repeat checks under ptl because UFFD, per PeterX and JannH; bring back mmu_notifier calls for PMD, per JannH and Jason; pmdp_get_lockless_sync() to issue missing interrupt if PAE. 10/13 mm/khugepaged: collapse_pte_mapped_thp() with mmap_read_lock() v3: updated to using ptent instead of *pte v2: first check VMA, in case page tables torn down, per JannH; pmdp_get_lockless_sync() to issue missing interrupt if PAE; moved mmu_notifier after step 1, reworked final goto labels. 11/13 mm/khugepaged: delete khugepaged_collapse_pte_mapped_thps() v3: rediffed v2: same as v1 12/13 mm: delete mmap_write_trylock() and vma_try_start_write() v3: rediffed (different diff needed for mm-unstable or linux-next) v2: same as v1 13/13 mm/pgtable: notes on pte_offset_map[_lock]() v3: new: JannH asked for more helpful comment, this is my attempt; could be moved to be the first in the series. arch/arm/mm/fault-armv.c | 3 +- arch/powerpc/include/asm/pgalloc.h | 4 + arch/powerpc/mm/pgtable-frag.c | 29 +- arch/powerpc/mm/pgtable.c | 16 +- arch/s390/include/asm/pgalloc.h | 4 + arch/s390/mm/pgalloc.c | 80 ++++- arch/sparc/include/asm/pgalloc_64.h | 4 + arch/sparc/mm/init_64.c | 16 + include/linux/mm.h | 17 -- include/linux/mm_types.h | 4 + include/linux/mmap_lock.h | 10 - include/linux/pgtable.h | 10 +- mm/khugepaged.c | 481 +++++++++++------------------- mm/pgtable-generic.c | 97 +++++- 14 files changed, 404 insertions(+), 371 deletions(-) Hugh