From patchwork Fri Nov 18 01:10:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE47C43217 for ; Fri, 18 Nov 2022 01:10:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36C218E0003; Thu, 17 Nov 2022 20:10:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F70A6B0073; Thu, 17 Nov 2022 20:10:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16E7D8E0003; Thu, 17 Nov 2022 20:10:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0069E6B0071 for ; Thu, 17 Nov 2022 20:10:37 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D51091604E1 for ; Fri, 18 Nov 2022 01:10:37 +0000 (UTC) X-FDA: 80144782914.14.80733BA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf02.hostedemail.com (Postfix) with ESMTP id 7E4718000B for ; Fri, 18 Nov 2022 01:10:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bpOw2twfKJ5Aty2Xl+QxZU1Og2jk2hUFVZDT66cF6Tk=; b=cJcCUpxeOeHM0GOyo3NtGXGU8EfEIhRfxSeRZJHL6Po//9LALlhZ2/jUJOx+e7HlHkq6Ug G3SguVdXch3LyEf4f19uzBRfh5m2S1FFvPR3uK+DSC/Wr+gbsDCEZTe6iCHzlgruG4t+Ca FeFHQFNbIQkV49eTQ4gTFOzcmHlxUGc= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-45-Z6ggdCvGMjSOTlEas9S1Bw-1; Thu, 17 Nov 2022 20:10:35 -0500 X-MC-Unique: Z6ggdCvGMjSOTlEas9S1Bw-1 Received: by mail-qk1-f199.google.com with SMTP id h13-20020a05620a244d00b006fb713618b8so4459958qkn.0 for ; Thu, 17 Nov 2022 17:10:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bpOw2twfKJ5Aty2Xl+QxZU1Og2jk2hUFVZDT66cF6Tk=; b=t9r4uGWiZw4QmyH+3ixb7/9sQQgFloAmJUIRizycVXl3I85zZClwgRIKBSoltfOUUl q6Ek06udx9vyvTewF6sXHMoh75YydlosiZQnutF3eiPlxuYiupCEjxDDkqvK8CovsqJp oSXfIRTgfSMUn7orPcDYIbex6xYTQn0wfNdcPI/c8ZPWTMBiybV69HzULHg+c7icegwR dOwevBwdJCLfzi9sty/H/cxTighU6+8zfXXazQos73FQ1PbQ9Hj7gwncBH4E+QFKQu25 bSU8ohRa5/upOaWB6qwvaG/ECQCj+o5IdFBTRQtLSKL1WWCOb799/79cphue9+TjASpg IAdw== X-Gm-Message-State: ANoB5pkRGDtzzQhlrt0gq71CnCuJuLDXmge651Q8J57y/DbraIwSb1/W P2qIeBYInXwFSllg6KntJ3l/HIIsT3QNo4ccnqWY4Q7RoUwrYwxJFrm64wPgDegP6HdlIOYV/L7 JS+NyaIYUPD0PhY3Lj2ZGOi6decWa9b7aDNAbKRhkZrT08EJsLnsZytcGuijs X-Received: by 2002:ac8:4e0c:0:b0:3a5:279d:dd34 with SMTP id c12-20020ac84e0c000000b003a5279ddd34mr4655000qtw.531.1668733835078; Thu, 17 Nov 2022 17:10:35 -0800 (PST) X-Google-Smtp-Source: AA0mqf7L/DTG7iZGWFpqDk5hsUhnfTTrsJsKX9hBQZCwDrkhBBpJLcBKb29jKauqxKgKWLHLgAsufg== X-Received: by 2002:ac8:4e0c:0:b0:3a5:279d:dd34 with SMTP id c12-20020ac84e0c000000b003a5279ddd34mr4654969qtw.531.1668733834702; Thu, 17 Nov 2022 17:10:34 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:34 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 04/12] mm/hugetlb: Add pgtable walker lock Date: Thu, 17 Nov 2022 20:10:17 -0500 Message-Id: <20221118011025.2178986-5-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221118011025.2178986-1-peterx@redhat.com> References: <20221118011025.2178986-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733837; a=rsa-sha256; cv=none; b=hBoveURbz+4W2VuFKfIBpRf4B7PndpfvI8eS3E27Bz/vMR/yMd82P4QN4IrVOx6ZzOh6sC F4r1iJ7s5s9ZLp75RjjZ/JWdarEvCDJYCfqVgb5xsB7f5ZS5WOxqf3sZRiUIXCkWl4cjHu 1z7E+ZKTR9Zq85mGDp/C7v+1u9e46oU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cJcCUpxe; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bpOw2twfKJ5Aty2Xl+QxZU1Og2jk2hUFVZDT66cF6Tk=; b=cG1ASuD13IvGIQH6PvcewZkrEZFU/5IpjFew4Tg6AX5A7bgLeJrb5ytek8y8kVC+0eZoxj wwlCADR1Rk2qVx6tIbaB09IW9EkqcicqQwZMyDDQOGFo9558WlrudlLPjJQA3iq5Tm8PwV n5LCj/xb5JlxLoNKiFMzeMMMSELDRhM= X-Stat-Signature: dtkdgnhuygcay69jue3643n5gwwx5bxr X-Rspamd-Queue-Id: 7E4718000B Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cJcCUpxe; spf=pass (imf02.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1668733837-972528 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is potentially a pgtable walker, looking up pte_t* for a hugetlb address. Normally, it's always safe to walk a generic pgtable as long as we're with the mmap lock held for either read or write, because that guarantees the pgtable pages will always be valid during the process. But it's not true for hugetlbfs, especially shared: hugetlbfs can have its pgtable freed by pmd unsharing, it means that even with mmap lock held for current mm, the PMD pgtable page can still go away from under us if pmd unsharing is possible during the walk. So we have three ways to make it safe: (1) If the mapping is private we're not prone to pmd sharing or unsharing, so it's okay. (2) If we're with the hugetlb vma lock held for either read/write, it's okay too because pmd unshare cannot happen at all. (3) Otherwise we may need the pgtable walker lock The pgtable walker is implemented in different ways depending on the config: - if !ARCH_WANT_HUGE_PMD_SHARE: it's no-op - else if MMU_GATHER_RCU_TABLE_FREE: it should be rcu_read_lock() - else: it should be local_irq_disable() A more efficient way to take the lock is only taking them with valid vmas that are possible to have pmd sharing (aka, want_pmd_share() returns true), but since the lock is mostly lightweighted (only riscv will use irq disable, x86 & arm will use rcu) just take them unconditionally for now. Document all these explicitly for huge_pte_offset() too, because it's not that obvious. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 99 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 551834cd5299..8f85ad0d5bdb 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -12,6 +12,8 @@ #include #include #include +#include +#include struct ctl_table; struct user_struct; @@ -24,6 +26,71 @@ typedef struct { unsigned long pd; } hugepd_t; #define __hugepd(x) ((hugepd_t) { (x) }) #endif +/** + * @hugetlb_walker_[un]lock(): protects software hugetlb pgtable walkers + * + * The hugetlb walker lock needs to be taken before huge_pte_offset() and + * needs to be released after finishing using the pte_t* returned. It's + * used to guarantee we won't access a freed pgtable page. Normally in + * this case we already have the mmap lock held for read. + * + * When holding the hugetlb walker lock, the thread cannot sleep, nor can + * it already in irq context (just to simplify unlock with no-save for + * !RCU_TABLE_TREE). Before the thread yields itself, it should release + * the pgtable lock. After the lock released, pte_t* can become invalid + * anytime so it cannot be accessed anymore. + * + * The only reason for the lock is to protect concurrent pmd unsharing + * followed by e.g. munmap() to drop the pgtable page. For no-pmd-sharing + * archs, it's no-op because it's always safe to access hugetlb pgtables + * just like generic pgtables. + */ +#if !defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) +static inline void hugetlb_walker_lock(void) +{ +} +static inline void hugetlb_walker_unlock(void) +{ +} +static inline bool __hugetlb_walker_locked(void) +{ + return true; +} +#elif defined(CONFIG_MMU_GATHER_RCU_TABLE_FREE) +static inline void hugetlb_walker_lock(void) +{ + rcu_read_lock(); +} +static inline void hugetlb_walker_unlock(void) +{ + rcu_read_unlock(); +} +static inline bool __hugetlb_walker_locked(void) +{ + return rcu_read_lock_held(); +} +#else +static inline void hugetlb_walker_lock(void) +{ + WARN_ON_ONCE(irqs_disabled()); + local_irq_disable(); +} +static inline void hugetlb_walker_unlock(void) +{ + local_irq_enable(); +} +static inline bool __hugetlb_walker_locked(void) +{ + return irqs_disabled(); +} +#endif + +#ifdef CONFIG_LOCKDEP +#define hugetlb_walker_locked() __hugetlb_walker_locked() +#else +#define hugetlb_walker_locked() true +#endif + #ifdef CONFIG_HUGETLB_PAGE #include @@ -192,6 +259,38 @@ extern struct list_head huge_boot_pages; pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); +/* + * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. + * Returns the pte_t* if found, or NULL if the address is not mapped. + * + * NOTE: since this function will walk all the pgtable pages (including not + * only high-level pgtable page, but also PUD entry that can be unshared + * concurrently for VM_SHARED), the caller of this function should be + * responsible of its thread safety. One can follow this rule: + * + * (1) For private mappings: pmd unsharing is not possible, so it'll + * always be safe if we're with the mmap sem for either read or write. + * This is normally always the case, IOW we don't need to do anything + * special. + * + * (2) For shared mappings: pmd unsharing is possible (so the PUD-ranged + * pgtable page can go away from under us! It can be done by a pmd + * unshare with a follow up munmap() on the other process), then we + * need either: + * + * (2.1) hugetlb vma lock read or write held, to make sure pmd unshare + * won't happen upon the range (it also makes sure the pte_t we + * read is the right and stable one), or, + * + * (2.2) pgtable walker lock, to make sure even pmd unsharing happened, + * the old shared PUD page won't get freed from under us, so even + * the pteval can be obsolete, at least it's still always safe to + * access the pgtable page (e.g., de-referencing pte_t* would not + * cause use-after-free). + * + * PS: from the regard of (2.2), it's the same logic of fast-gup being safe + * for generic mm when walking the pgtables even without mmap lock. + */ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); unsigned long hugetlb_mask_last_page(struct hstate *h);