From patchwork Sun Feb 16 19:18:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rafael Aquini X-Patchwork-Id: 11384625 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D9662109A for ; Sun, 16 Feb 2020 19:18:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7871520857 for ; Sun, 16 Feb 2020 19:18:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="dgI6yoT5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7871520857 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 18D1C6B0003; Sun, 16 Feb 2020 14:18:16 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 13EA86B0005; Sun, 16 Feb 2020 14:18:16 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02BB36B0007; Sun, 16 Feb 2020 14:18:15 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D85856B0003 for ; Sun, 16 Feb 2020 14:18:15 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 6DCDD180AD80F for ; Sun, 16 Feb 2020 19:18:15 +0000 (UTC) X-FDA: 76496950950.10.lunch44_6517271149b47 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,aquini@redhat.com,::linux-kernel@vger.kernel.org:mgorman@techsingularity.net:akpm@linux-foundation.org:mhocko@suse.com:vbabka@suse.cz,RULES_HIT:30045:30051:30054:30056:30070:30075:30079,0,RBL:205.139.110.61:@redhat.com:.lbl8.mailshell.net-62.18.0.100 66.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: lunch44_6517271149b47 X-Filterd-Recvd-Size: 7173 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Sun, 16 Feb 2020 19:18:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1581880694; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=tuGE6PQn1KB7+DuvWdmcQTP1e0rV79mM3JVVfbfKivs=; b=dgI6yoT5scml1cSkLWRYsMXQGugXrd45SCBRp8eMgsBHRtyrNtbS+0O1QuqpEJh8GavBet 9pqcfCuB6f5FyXxp7hcqcNZg2jCfoRIzz1N/vWtioMjP7EAUiTk/gjoyJwF2kNxn6e6/bp YbS/MGLP+U/gYGEYeupB3wnMirdLalM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-24-WXW4iEjeON-G447_iBq-nw-1; Sun, 16 Feb 2020 14:18:06 -0500 X-MC-Unique: WXW4iEjeON-G447_iBq-nw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 56D21107ACC7; Sun, 16 Feb 2020 19:18:04 +0000 (UTC) Received: from t490s.redhat.com (ovpn-116-86.phx2.redhat.com [10.3.116.86]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0295319756; Sun, 16 Feb 2020 19:18:02 +0000 (UTC) From: Rafael Aquini To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mgorman@techsingularity.net, akpm@linux-foundation.org, mhocko@suse.com, vbabka@suse.cz Subject: [PATCH] mm, numa: fix bad pmd by atomically check for pmd_trans_huge when marking page tables prot_numa Date: Sun, 16 Feb 2020 14:18:00 -0500 Message-Id: <20200216191800.22423-1-aquini@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mel Gorman A user reported a bug against a distribution kernel while running a proprietary workload described as "memory intensive that is not swapping" that is expected to apply to mainline kernels. The workload is read/write/modifying ranges of memory and checking the contents. They reported that within a few hours that a bad PMD would be reported followed by a memory corruption where expected data was all zeros. A partial report of the bad PMD looked like [ 5195.338482] ../mm/pgtable-generic.c:33: bad pmd ffff8888157ba008(000002e0396009e2) [ 5195.341184] ------------[ cut here ]------------ [ 5195.356880] kernel BUG at ../mm/pgtable-generic.c:35! .... [ 5195.410033] Call Trace: [ 5195.410471] [] change_protection_range+0x7dd/0x930 [ 5195.410716] [] change_prot_numa+0x18/0x30 [ 5195.410918] [] task_numa_work+0x1fe/0x310 [ 5195.411200] [] task_work_run+0x72/0x90 [ 5195.411246] [] exit_to_usermode_loop+0x91/0xc2 [ 5195.411494] [] prepare_exit_to_usermode+0x31/0x40 [ 5195.411739] [] retint_user+0x8/0x10 Decoding revealed that the PMD was a valid prot_numa PMD and the bad PMD was a false detection. The bug does not trigger if automatic NUMA balancing or transparent huge pages is disabled. The bug is due a race in change_pmd_range between a pmd_trans_huge and pmd_nond_or_clear_bad check without any locks held. During the pmd_trans_huge check, a parallel protection update under lock can have cleared the PMD and filled it with a prot_numa entry between the transhuge check and the pmd_none_or_clear_bad check. While this could be fixed with heavy locking, it's only necessary to make a copy of the PMD on the stack during change_pmd_range and avoid races. A new helper is created for this as the check if quite subtle and the existing similar helpful is not suitable. This passed 154 hours of testing (usually triggers between 20 minutes and 24 hours) without detecting bad PMDs or corruption. A basic test of an autonuma-intensive workload showed no significant change in behaviour. Although Mel withdrew the patch on the face of LKML comment https://lkml.org/lkml/2017/4/10/922 the race window aforementioned is still open, and we have reports of Linpack test reporting bad residuals after the bad PMD warning is observed. In addition to that, bad rss-counter and non-zero pgtables assertions are triggered on mm teardown for the task hitting the bad PMD. host kernel: mm/pgtable-generic.c:40: bad pmd 00000000b3152f68(8000000d2d2008e7) .... host kernel: BUG: Bad rss-counter state mm:00000000b583043d idx:1 val:512 host kernel: BUG: non-zero pgtables_bytes on freeing mm: 4096 The issue is observed on a v4.18-based distribution kernel, but the race window is expected to be applicable to mainline kernels, as well. Signed-off-by: Mel Gorman Cc: stable@vger.kernel.org Signed-off-by: Rafael Aquini Signed-off-by: Qian Cai --- mm/mprotect.c | 38 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index 7a8e84f86831..9ea8cc0ab2fd 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -161,6 +161,31 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, return pages; } +/* + * Used when setting automatic NUMA hinting protection where it is + * critical that a numa hinting PMD is not confused with a bad PMD. + */ +static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) +{ + pmd_t pmdval = pmd_read_atomic(pmd); + + /* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + barrier(); +#endif + + if (pmd_none(pmdval)) + return 1; + if (pmd_trans_huge(pmdval)) + return 0; + if (unlikely(pmd_bad(pmdval))) { + pmd_clear_bad(pmd); + return 1; + } + + return 0; +} + static inline unsigned long change_pmd_range(struct vm_area_struct *vma, pud_t *pud, unsigned long addr, unsigned long end, pgprot_t newprot, int dirty_accountable, int prot_numa) @@ -178,8 +203,17 @@ static inline unsigned long change_pmd_range(struct vm_area_struct *vma, unsigned long this_pages; next = pmd_addr_end(addr, end); - if (!is_swap_pmd(*pmd) && !pmd_trans_huge(*pmd) && !pmd_devmap(*pmd) - && pmd_none_or_clear_bad(pmd)) + + /* + * Automatic NUMA balancing walks the tables with mmap_sem + * held for read. It's possible a parallel update to occur + * between pmd_trans_huge() and a pmd_none_or_clear_bad() + * check leading to a false positive and clearing. + * Hence, it's ecessary to atomically read the PMD value + * for all the checks. + */ + if (!is_swap_pmd(*pmd) && !pmd_devmap(*pmd) && + pmd_none_or_clear_bad_unless_trans_huge(pmd)) goto next; /* invoke the mmu notifier if the pmd is populated */