From patchwork Tue Nov 29 15:47:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 13058731 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2482C46467 for ; Tue, 29 Nov 2022 15:47:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 403F16B0072; Tue, 29 Nov 2022 10:47:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B5D16B0074; Tue, 29 Nov 2022 10:47:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 235586B0075; Tue, 29 Nov 2022 10:47:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0BC4D6B0072 for ; Tue, 29 Nov 2022 10:47:45 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D50BC1206B4 for ; Tue, 29 Nov 2022 15:47:44 +0000 (UTC) X-FDA: 80186910048.15.7605070 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by imf23.hostedemail.com (Postfix) with ESMTP id 7968D14000B for ; Tue, 29 Nov 2022 15:47:44 +0000 (UTC) Received: by mail-wm1-f41.google.com with SMTP id h131-20020a1c2189000000b003d02dd48c45so8831269wmh.0 for ; Tue, 29 Nov 2022 07:47:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kT8IZrjdmDIMAqdPXo5e6xGOYanI26oKv2vQ1WDKtQk=; b=BL0l4dA+hwXaxDagId779Kfqnra35IhbXewFqGa9/iFcvBB3etkTunBBW79/i36Z7W FfYpfWUnFoFz59NOcvGwEnZS33GiDNsps7XypTQfxe7OCHjHXk+z0tOPFjYOE+cACcsZ BXOa2b+J8eBTp5BN0CToCCnb2NeAqGP9snzI7hhPMzlf2jIo2SFynaVZjlfQYYi0iCSw GPCs7ry1zENWaAccRHHdPcJ6eAZekj2Pr5KT7h76W9ibjnORNwIxH00ODyhVgzvlN1vh CW16wRyiAnWnHq7GeBxH72cDAsZOijNEWdnWbcouFDuLz3dJ+2kRu6/4vmBtPncSAits V0Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kT8IZrjdmDIMAqdPXo5e6xGOYanI26oKv2vQ1WDKtQk=; b=YXV7q/dZ0GaXBTYrNxyaruw6SFeBq32wwfhBC3e6vy1gp6AWqOJtaJn8td4yhc6AdN fB9GlFXkuaw/z+bEacCRTqbaUb46T1coVYw39+UhpI3NenTxo2NkD02lJBz8XBZkf3bs H46x0RwNPX4O90JJqYCavC7TMXrLWvHQ61ljAXwa9Y6uucEFGPUf/sT13RkJQPPViyo4 A0aLqIb8aYXMNvEy65wUOAjPzf4HceMArsRm5PDQ1qbFvotN5fJUSfTAV+OZWtiqGlqH d+B3E22Av52+5aNqBmlzTC7X0aTwJ/Fcx751mQ9JmT8rv3Tf5PivNRhf1NP0gWGmZaWs u9UA== X-Gm-Message-State: ANoB5pmWcB/gvnSUpTV87AfQqfsH+FtQO6KcDud+DZz+NzUksXK0mBOb U9JSz32KSE21O2rNzS/pa3kxaA== X-Google-Smtp-Source: AA0mqf6a5G/8CLXnb0SAy9z118CKG1gCpO4zhtBYmnvQa4oO/gHafcMunPZk/HVikU3D8gdKcQu13g== X-Received: by 2002:a05:600c:19ce:b0:3d0:5160:e0e2 with SMTP id u14-20020a05600c19ce00b003d05160e0e2mr11570874wmq.147.1669736863055; Tue, 29 Nov 2022 07:47:43 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:5011:adcc:fddd:accf]) by smtp.gmail.com with ESMTPSA id o9-20020a5d4a89000000b00241fde8fe04sm14090930wrq.7.2022.11.29.07.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Nov 2022 07:47:42 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 2/3] mm/khugepaged: Fix GUP-fast interaction by sending IPI Date: Tue, 29 Nov 2022 16:47:29 +0100 Message-Id: <20221129154730.2274278-2-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221129154730.2274278-1-jannh@google.com> References: <20221129154730.2274278-1-jannh@google.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669736864; a=rsa-sha256; cv=none; b=cJ8eUkR067NDQu9mCrJX8ViiSDsKnDYIFDURLLC1pSGWYQTgslKcggXwxfTutoefk8ySwi /ZHj8QMGYSzq+QpbX+eUqCAJ8wY2R/0cVH+JR3a/5WpSXtiSZYuBcaWCDnTlTF0w5C23G4 t4PlB/LZq3gp2DLgvqjN44Z05nb65QE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BL0l4dA+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of jannh@google.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=jannh@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669736864; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kT8IZrjdmDIMAqdPXo5e6xGOYanI26oKv2vQ1WDKtQk=; b=qJOTFwIQacAhJRwi7vjeSeEynTJhwHF/OGkkxpTRRReyQqkFN5iKCbzjIc17tJJKg/0QRC LaQrolwr14TsU6h5DD+uq/Ts9LB0zMIjzFK4Bcs1jKygx/SpIHKf70+Ol3A+bVu5chgP5d 7/+f/jR/JfMf7dXWakrDm58axNWZ7oM= X-Stat-Signature: 384ejag3no3txqk6hi4fkawjwpfigouq X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 7968D14000B Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=BL0l4dA+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of jannh@google.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=jannh@google.com X-Rspam-User: X-HE-Tag: 1669736864-270608 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The khugepaged paths that remove page tables have to be careful to synchronize against the lockless_pages_from_mm() path, which traverses page tables while only being protected by disabled IRQs. lockless_pages_from_mm() must not: 1. interpret the contents of freed memory as page tables (and once a page table has been deposited, it can be freed) 2. interpret the contents of deposited page tables as PTEs, since some architectures will store non-PTE data inside deposited page tables (see radix__pgtable_trans_huge_deposit()) 3. create new page references from PTEs after the containing page table has been detached and: 3a. __collapse_huge_page_isolate() has checked the page refcount 3b. the page table has been reused at another virtual address and populated with new PTEs ("new page references" here refer to stable references returned to the caller; speculative references that are dropped on an error path are fine) commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse") addressed issue 3 by making the lockless_pages_from_mm() fastpath recheck the pmd_t to ensure that the page table was not removed by khugepaged in between (under the assumption that the page table is not repeatedly moving back and forth between two addresses, with one PTE repeatedly being populated with the same value). But to address issues 1 and 2, we need to send IPIs before freeing/reusing page tables. By doing that, issue 3 is also automatically addressed, so the fix from commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse") becomes redundant. We can ensure that the necessary IPI is sent by calling tlb_remove_table_sync_one() because, as noted in mm/gup.c, under configurations that define CONFIG_HAVE_FAST_GUP, there are two possible cases: 1. CONFIG_MMU_GATHER_RCU_TABLE_FREE is set, causing tlb_remove_table_sync_one() to send an IPI to synchronize with lockless_pages_from_mm(). 2. CONFIG_MMU_GATHER_RCU_TABLE_FREE is unset, indicating that all TLB flushes are already guaranteed to send IPIs. tlb_remove_table_sync_one() will do nothing, but we've already run pmdp_collapse_flush(), which did a TLB flush, which must have involved IPIs. Cc: stable@kernel.org Fixes: ba76149f47d8 ("thp: khugepaged") Reviewed-by: Yang Shi Acked-by: David Hildenbrand Signed-off-by: Jann Horn --- Notes: v4: - added ack from David Hildenbrand - made commit message more verbose v5: - added reviewed-by from Yang Shi - rewrote commit message based on feedback from Yang Shi include/asm-generic/tlb.h | 4 ++++ mm/khugepaged.c | 2 ++ mm/mmu_gather.c | 4 +--- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 492dce43236ea..cab7cfebf40bd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,12 +222,16 @@ extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #define tlb_needs_table_invalidate() (true) #endif +void tlb_remove_table_sync_one(void); + #else #ifdef tlb_needs_table_invalidate #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE #endif +static inline void tlb_remove_table_sync_one(void) { } + #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 674b111a24fa7..c3d3ce596bff7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1057,6 +1057,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); spin_lock(pte_ptl); result = __collapse_huge_page_isolate(vma, address, pte, cc, @@ -1415,6 +1416,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v lockdep_assert_held_write(&vma->anon_vma->root->rwsem); pmd = pmdp_collapse_flush(vma, addr, pmdp); + tlb_remove_table_sync_one(); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index add4244e5790d..3a2c3f8cad2fe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -153,7 +153,7 @@ static void tlb_remove_table_smp_sync(void *arg) /* Simply deliver the interrupt */ } -static void tlb_remove_table_sync_one(void) +void tlb_remove_table_sync_one(void) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -177,8 +177,6 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ -static void tlb_remove_table_sync_one(void) { } - static void tlb_remove_table_free(struct mmu_table_batch *batch) { __tlb_remove_table_free(batch);