From patchwork Sat Aug 28 04:23:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12463255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A86B9C432BE for ; Sat, 28 Aug 2021 04:23:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 237B960EBD for ; Sat, 28 Aug 2021 04:23:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 237B960EBD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C1DA16B0071; Sat, 28 Aug 2021 00:23:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCCF96B0072; Sat, 28 Aug 2021 00:23:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A94088D0001; Sat, 28 Aug 2021 00:23:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 8F53F6B0071 for ; Sat, 28 Aug 2021 00:23:23 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2381027774 for ; Sat, 28 Aug 2021 04:23:23 +0000 (UTC) X-FDA: 78523195086.08.A2A5D8B Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf30.hostedemail.com (Postfix) with ESMTP id D57A2E0016B0 for ; Sat, 28 Aug 2021 04:23:22 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id m17so5298944plc.6 for ; Fri, 27 Aug 2021 21:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DOQhtRJ283WN5i7pZFdCAMLJddI7+kc0mDNTD4e/NnI=; b=uOhcmas8K2ET6mieFgVELlVqw9VnQ8PHqGHiuSLRHQcRot4aWXhghVdYKMCrq7CnfY +GNSvXNDB5spvFzbTwdnCK6jnS55XNMCN7po5Efk4/vYhebjKdNOZSI+8KGeFiU/bOCc gIiqftF6P3pS6ivslF/iekYGgCH6UDNpbjCFkZ7ocBL7eYOVZBdb4wvbDOkVSO+LFAvt Z/mZGT1ll2qoKlmysbwqBZuPoRBgJoWnORkNwuJyIKc6uuhbL0y20QXDR55pyFsbAm4k 617MUknLTK/z+QKvW5TJq6JJl+KtfZ7APSTO0r+WelWrfRqDXDgBrp64eJeKzcqlWLqH n4+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DOQhtRJ283WN5i7pZFdCAMLJddI7+kc0mDNTD4e/NnI=; b=kYL/B0YB0NDhTKyvav0MKneCxcRTHmRgHmBVlJksqpYM9kIRZ1FjDyQWpI2nCa0d/6 eAgXQ8pKbaO8qK3W6Wr/JRWj21BxeO84kMvMzR3xt0pgx2nOdk9ePnzj0ajzweQeZJ0i fde698sB6+9Rfmb4ghwnLd9sEv7zcszjyVuvkng7zvnd+D2gI0i/oiwsChv8CLIBS3E9 xfUO+wDcfyQdLvjaKWexeHYo4tUpXPZst2MPNjpfcGz5isq4WFfbuNGYpheS0IoL33bK SlKmBcY+myZNopKVJszAgc68MvuygfXGseMjV2TanYi9r0yMEEsmALDSPKT4zB/t7+L7 ghoQ== X-Gm-Message-State: AOAM531vX7Sp/LNDTRjZaOw3vGvabPtKiGwDEhocw8C947x0qmx10nJQ mqTGYBvGhFUPAj9kThcBLKzBbA== X-Google-Smtp-Source: ABdhPJzrwOcapjsD5epxpp/1rc70UdjtOMNFf0EdX6dc9t5hZwCuXgxkI7KkjeyzSfsRXv/kMJ95sQ== X-Received: by 2002:a17:90a:8b12:: with SMTP id y18mr27121724pjn.72.1630124601867; Fri, 27 Aug 2021 21:23:21 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id u15sm4720383pfl.14.2021.08.27.21.23.17 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Aug 2021 21:23:21 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tglx@linutronix.de, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, david@redhat.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, songmuchun@bytedance.com, Qi Zheng Subject: [PATCH v1 1/2] mm: introduce pmd_install() helper Date: Sat, 28 Aug 2021 12:23:05 +0800 Message-Id: <20210828042306.42886-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20210828042306.42886-1-zhengqi.arch@bytedance.com> References: <20210828042306.42886-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=uOhcmas8; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D57A2E0016B0 X-Stat-Signature: iwjmzgy9ytwsaw74bzw1dc886ux38eny X-HE-Tag: 1630124602-417619 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently we have three times the same few lines repeated in the code. Deduplicate them by newly introduced pmd_install() helper. Signed-off-by: Qi Zheng Reviewed-by: David Hildenbrand Reviewed-by: Muchun Song --- include/linux/mm.h | 1 + mm/filemap.c | 11 ++--------- mm/memory.c | 34 ++++++++++++++++------------------ 3 files changed, 19 insertions(+), 27 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cc83d64564..0af420a7e382 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2463,6 +2463,7 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud) return ptl; } +extern void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); extern void __init pagecache_init(void); extern void __init free_area_init_memoryless_node(int nid); extern void free_initmem(void); diff --git a/mm/filemap.c b/mm/filemap.c index c90b6e4984c9..923cbba1bf37 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3209,15 +3209,8 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct page *page) } } - if (pmd_none(*vmf->pmd)) { - vmf->ptl = pmd_lock(mm, vmf->pmd); - if (likely(pmd_none(*vmf->pmd))) { - mm_inc_nr_ptes(mm); - pmd_populate(mm, vmf->pmd, vmf->prealloc_pte); - vmf->prealloc_pte = NULL; - } - spin_unlock(vmf->ptl); - } + if (pmd_none(*vmf->pmd)) + pmd_install(mm, vmf->pmd, &vmf->prealloc_pte); /* See comment in handle_pte_fault() */ if (pmd_devmap_trans_unstable(vmf->pmd)) { diff --git a/mm/memory.c b/mm/memory.c index 39e7a1495c3c..ef7b1762e996 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -433,9 +433,20 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, } } +void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte) +{ + spinlock_t *ptl = pmd_lock(mm, pmd); + + if (likely(pmd_none(*pmd))) { /* Has another populated it ? */ + mm_inc_nr_ptes(mm); + pmd_populate(mm, pmd, *pte); + *pte = NULL; + } + spin_unlock(ptl); +} + int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) { - spinlock_t *ptl; pgtable_t new = pte_alloc_one(mm); if (!new) return -ENOMEM; @@ -455,13 +466,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) */ smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */ - ptl = pmd_lock(mm, pmd); - if (likely(pmd_none(*pmd))) { /* Has another populated it ? */ - mm_inc_nr_ptes(mm); - pmd_populate(mm, pmd, new); - new = NULL; - } - spin_unlock(ptl); + pmd_install(mm, pmd, &new); if (new) pte_free(mm, new); return 0; @@ -4027,17 +4032,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return ret; } - if (vmf->prealloc_pte) { - vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); - if (likely(pmd_none(*vmf->pmd))) { - mm_inc_nr_ptes(vma->vm_mm); - pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte); - vmf->prealloc_pte = NULL; - } - spin_unlock(vmf->ptl); - } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) { + if (vmf->prealloc_pte) + pmd_install(vma->vm_mm, vmf->pmd, &vmf->prealloc_pte); + else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) return VM_FAULT_OOM; - } } /* See comment in handle_pte_fault() */ From patchwork Sat Aug 28 04:23:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12463257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A58DC432BE for ; Sat, 28 Aug 2021 04:23:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1982960F36 for ; Sat, 28 Aug 2021 04:23:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1982960F36 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id B7FF26B0072; Sat, 28 Aug 2021 00:23:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2F1D8D0002; Sat, 28 Aug 2021 00:23:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1E6F8D0001; Sat, 28 Aug 2021 00:23:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 867A96B0072 for ; Sat, 28 Aug 2021 00:23:28 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2B0EA27744 for ; Sat, 28 Aug 2021 04:23:28 +0000 (UTC) X-FDA: 78523195296.40.6A24893 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf14.hostedemail.com (Postfix) with ESMTP id D8ECC6001982 for ; Sat, 28 Aug 2021 04:23:27 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id oa17so5970154pjb.1 for ; Fri, 27 Aug 2021 21:23:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=r4sxzAuXhEsyJUy6AggBHx5gh/7nTNy4dMi+0q2BJs8=; b=yyQAqAZC9uRXK9FQF2k6SnE4k4hEQtHILW6KYrD5m3mjzKnYvpnS3FXOf5do3xMSgt S1Amg8BS8q6cA8sOC+m+1rPL2mKEajZAZOPDaycaLy0814RWS7A2WaN9XQ8jVi5IigHY ukKnRBQSdJi+NLLSUnWJBY5Weh/u+I2bkO7x9p4Rgv3wMMk4ZI+BQ4zuvrmia+VeCCFN 6WpKibnp4XbUyQyBGwX7ebghnN+on6oDp+OKEGTd45ncGP1HN7l7w/l+Nn675tASffWM YE6GPfo7EIk/D6/wxzVKQiAhkGdlXxFiYVaSTPWDUMXFPKcuXE7NrOrcd/0sGg4yxQB6 RuiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=r4sxzAuXhEsyJUy6AggBHx5gh/7nTNy4dMi+0q2BJs8=; b=AnpnndM4NeJ+j1crPWu9SpDDvao1bFkweK12RfdediEUSggaC8cvghe8hMR3YzPIb4 kwR2VDlupMCgvlrRoUTEoGfG0SYWTOqchgBS/cui90KthapdhvXVh28Qb6lm9KuOwLtW x/6TZNQUAB8UhvBe92yVj9BgMXrVA8nLQkdj2giUi935FtdQHbutmjGuAzDDBuz1sKdt hxsGz56b5EsQnJBon8txuvDmg8tukbtKwC+bw1hCKUOYPegP6qwiTV5E6A9hjaXhSBe9 GXa/k7nmXfGM87TiXwuRAKkZUes3HpPlzQMg7hb+Yn8MqGC6Crnl19mYu4ijyBWR2McJ CRjw== X-Gm-Message-State: AOAM532XF/Vsh9OeBzD8A2S4rWs7xohHgLokYvUNNFB03V7BVX3EPQOS S544kaRrweJUFC5/DYNOqysUfw== X-Google-Smtp-Source: ABdhPJxAKh0jHXzQ3XUZSiM3xkbtWfUh3Ooz0IH1FNKINVaZ7/23hF3EU1CRBJZqTecOEq4BAnb9lA== X-Received: by 2002:a17:90b:1952:: with SMTP id nk18mr27330276pjb.193.1630124607071; Fri, 27 Aug 2021 21:23:27 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.254]) by smtp.gmail.com with ESMTPSA id u15sm4720383pfl.14.2021.08.27.21.23.22 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Aug 2021 21:23:26 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tglx@linutronix.de, hannes@cmpxchg.org, mhocko@kernel.org, vdavydov.dev@gmail.com, kirill.shutemov@linux.intel.com, mika.penttila@nextfour.com, david@redhat.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, songmuchun@bytedance.com, Qi Zheng Subject: [PATCH v1 2/2] mm: remove redundant smp_wmb() Date: Sat, 28 Aug 2021 12:23:06 +0800 Message-Id: <20210828042306.42886-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20210828042306.42886-1-zhengqi.arch@bytedance.com> References: <20210828042306.42886-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=yyQAqAZC; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf14.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.54 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D8ECC6001982 X-Stat-Signature: 8mbjrmsouammyzqxpwqbo37kf5ktd7m9 X-HE-Tag: 1630124607-317027 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The smp_wmb() which is in the __pte_alloc() is used to ensure all ptes setup is visible before the pte is made visible to other CPUs by being put into page tables. We only need this when the pte is actually populated, so move it to pte_install(). __pte_alloc_kernel(), __p4d_alloc(), __pud_alloc() and __pmd_alloc() are similar to this case. We can also defer smp_wmb() to the place where the pmd entry is really populated by preallocated pte. There are two kinds of user of preallocated pte, one is filemap & finish_fault(), another is THP. The former does not need another smp_wmb() because the smp_wmb() has been done by pte_install(). Fortunately, the latter also does not need another smp_wmb() because there is already a smp_wmb() before populating the new pte when the THP uses a preallocated pte to split a huge pmd. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- mm/memory.c | 47 ++++++++++++++++++++--------------------------- mm/sparse-vmemmap.c | 2 +- 2 files changed, 21 insertions(+), 28 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index ef7b1762e996..9c7534187454 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -439,6 +439,20 @@ void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte) if (likely(pmd_none(*pmd))) { /* Has another populated it ? */ mm_inc_nr_ptes(mm); + /* + * Ensure all pte setup (eg. pte page lock and page clearing) are + * visible before the pte is made visible to other CPUs by being + * put into page tables. + * + * The other side of the story is the pointer chasing in the page + * table walking code (when walking the page table without locking; + * ie. most of the time). Fortunately, these data accesses consist + * of a chain of data-dependent loads, meaning most CPUs (alpha + * being the notable exception) will already guarantee loads are + * seen in-order. See the alpha page table accessors for the + * smp_rmb() barriers in page table walking code. + */ + smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */ pmd_populate(mm, pmd, *pte); *pte = NULL; } @@ -451,21 +465,6 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) if (!new) return -ENOMEM; - /* - * Ensure all pte setup (eg. pte page lock and page clearing) are - * visible before the pte is made visible to other CPUs by being - * put into page tables. - * - * The other side of the story is the pointer chasing in the page - * table walking code (when walking the page table without locking; - * ie. most of the time). Fortunately, these data accesses consist - * of a chain of data-dependent loads, meaning most CPUs (alpha - * being the notable exception) will already guarantee loads are - * seen in-order. See the alpha page table accessors for the - * smp_rmb() barriers in page table walking code. - */ - smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */ - pmd_install(mm, pmd, &new); if (new) pte_free(mm, new); @@ -478,10 +477,9 @@ int __pte_alloc_kernel(pmd_t *pmd) if (!new) return -ENOMEM; - smp_wmb(); /* See comment in __pte_alloc */ - spin_lock(&init_mm.page_table_lock); if (likely(pmd_none(*pmd))) { /* Has another populated it ? */ + smp_wmb(); /* See comment in pmd_install() */ pmd_populate_kernel(&init_mm, pmd, new); new = NULL; } @@ -3857,7 +3855,6 @@ static vm_fault_t __do_fault(struct vm_fault *vmf) vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); if (!vmf->prealloc_pte) return VM_FAULT_OOM; - smp_wmb(); /* See comment in __pte_alloc() */ } ret = vma->vm_ops->fault(vmf); @@ -3919,7 +3916,6 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); if (!vmf->prealloc_pte) return VM_FAULT_OOM; - smp_wmb(); /* See comment in __pte_alloc() */ } vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); @@ -4144,7 +4140,6 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm); if (!vmf->prealloc_pte) return VM_FAULT_OOM; - smp_wmb(); /* See comment in __pte_alloc() */ } return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); @@ -4819,13 +4814,13 @@ int __p4d_alloc(struct mm_struct *mm, pgd_t *pgd, unsigned long address) if (!new) return -ENOMEM; - smp_wmb(); /* See comment in __pte_alloc */ - spin_lock(&mm->page_table_lock); if (pgd_present(*pgd)) /* Another has populated it */ p4d_free(mm, new); - else + else { + smp_wmb(); /* See comment in pmd_install() */ pgd_populate(mm, pgd, new); + } spin_unlock(&mm->page_table_lock); return 0; } @@ -4842,11 +4837,10 @@ int __pud_alloc(struct mm_struct *mm, p4d_t *p4d, unsigned long address) if (!new) return -ENOMEM; - smp_wmb(); /* See comment in __pte_alloc */ - spin_lock(&mm->page_table_lock); if (!p4d_present(*p4d)) { mm_inc_nr_puds(mm); + smp_wmb(); /* See comment in pmd_install() */ p4d_populate(mm, p4d, new); } else /* Another has populated it */ pud_free(mm, new); @@ -4867,11 +4861,10 @@ int __pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long address) if (!new) return -ENOMEM; - smp_wmb(); /* See comment in __pte_alloc */ - ptl = pud_lock(mm, pud); if (!pud_present(*pud)) { mm_inc_nr_pmds(mm); + smp_wmb(); /* See comment in pmd_install() */ pud_populate(mm, pud, new); } else /* Another has populated it */ pmd_free(mm, new); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..db6df27c852a 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -76,7 +76,7 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, set_pte_at(&init_mm, addr, pte, entry); } - /* Make pte visible before pmd. See comment in __pte_alloc(). */ + /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable);