From patchwork Tue Nov 13 05:50:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sasha Levin X-Patchwork-Id: 10679593 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9950213BB for ; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 895922A264 for ; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7DD4D2A305; Tue, 13 Nov 2018 05:51:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF6FD2A372 for ; Tue, 13 Nov 2018 05:51:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32F746B027B; Tue, 13 Nov 2018 00:51:39 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2DF886B027C; Tue, 13 Nov 2018 00:51:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CFE36B027D; Tue, 13 Nov 2018 00:51:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id CE5EE6B027B for ; Tue, 13 Nov 2018 00:51:38 -0500 (EST) Received: by mail-pl1-f200.google.com with SMTP id 94-v6so8773785pla.5 for ; Mon, 12 Nov 2018 21:51:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=E7aKex6/K1LOcebDiqEWvP243L5GuhYc4GuDE6fnM9Y=; b=C5k23FRKoB01//Kt/kQRujdJYzAEzxD21e+md8zbKdvl+TnOOjVFvYn/XXysbNm297 kX6U+axxhVZenfWliLkjd/gqwqHZV3YnvOz6VdUKtcRAXR2kkBqxjAwufsMeNi0JXzM+ LLkd7czrwJoDH2556OtEKf4SWRCVHRZWU/j95scOLdV71Py4VlPP8lqt7+uHkKYH8LkD dWRXz/fP11gqUJw4V1adWqdWXdoO3Hs1c5d8m7d7Lg4Ua12Gd00T+v1c6Iw+8P4VaGCW 3M3ck9VC7UF2q6srUhQzskbVSZ7qXBLcLfjxgO5tbhG9n6NRHcNaQ9N1x5KahT7uEotG Q4Hg== X-Gm-Message-State: AGRZ1gJaG7Np2nHvsNBVHbFiNpEEr+9BKFiBLBXH1EaDVMmWKzV6+xRZ c5F+8nLUYgkh2J+9XCXCb81IdbeKvM0ZBXBPcuDN5q6Zvvk7LWMvvfYFoYca8wOrZ2DYy/mBHH5 v+XTs5d4oHp7DIbVN++kghwSMK1VLi4oeEJU31grsf3Yh6rsg642iCLKcBcouJUzi7g== X-Received: by 2002:a63:1258:: with SMTP id 24mr3490259pgs.114.1542088298473; Mon, 12 Nov 2018 21:51:38 -0800 (PST) X-Google-Smtp-Source: AJdET5eBKn5zXACZFJTgSbY3V3WlXbV5uWkfkS6Qnxa3o9/qiYqB2UeRRmgfMaXlr2h49a0Gdgfu X-Received: by 2002:a63:1258:: with SMTP id 24mr3490228pgs.114.1542088297456; Mon, 12 Nov 2018 21:51:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542088297; cv=none; d=google.com; s=arc-20160816; b=ctCj3MwxcVrQicRv2O1JACJME+OiQ0ztLZym2GvRv4H343vDr0qgDgHKwaS5Ffd8HN FmO5kYwLukJgAYjoQUrW8/PBQ9yM1aBdh/sWkFyuMhXzfehihxJjd9NoIzO0kZgS+HXW FcDQLgzmbDBMrWR255+a1JYqyWau0Nvr1J08x1UrttNdyu3Vgqkxigg6N2bCyXZHX1Cd mo7qSA5x2txYXfMzfgZGBeWytR3KwY3a7oEfpAFIinSqsTeVifgdxzzalPuCUfHHNmq4 eVO+nnh3rFF9F1fFfCx4yqDOQHuxVAdqUOpiJ3MKWAxz5UPAqG+4ngQff8MdGz4uB0ab r9TA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=E7aKex6/K1LOcebDiqEWvP243L5GuhYc4GuDE6fnM9Y=; b=Qdtt79clEdGgBRGBWj1xr+cPVNv/aVpXJHU00aVOgnOMA9TnNT8KCptkTklLMtykLX zulzJTbZwp/kvvqcWe1N1K+pCZWkImseoih20COl/cHtNHsuPcPfed/vtSywd9pX4lAw TQGtaaGxob1qMKc+mXdbDmiooDfAJDN1aJp4CabU0Gf3NWS76gFSIfHQr1pS1qZBxNA6 gsv9fjh2jlytNJa22qng7Ug82GDHRi9yVEoyx5kXtNtu3uHYwIHVRASDvDROp2NvDBSZ zFs2K+JUjX65ztjSl2iDDeXmuAPrgfByFrZxXmunWf4/sPPnD60N7sfnQEQgCiUM0dNG lwFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=skOW9FC4; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from mail.kernel.org (mail.kernel.org. [198.145.29.99]) by mx.google.com with ESMTPS id m14si18356640pgd.326.2018.11.12.21.51.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Nov 2018 21:51:37 -0800 (PST) Received-SPF: pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) client-ip=198.145.29.99; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=skOW9FC4; spf=pass (google.com: domain of sashal@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=sashal@kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from sasha-vm.mshome.net (unknown [64.114.255.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0820522510; Tue, 13 Nov 2018 05:51:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542088297; bh=N6As6j6xCoPMaQzs4R/SYfp9cNiF7iuDlMLJPddbhIE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=skOW9FC48TgHb3LX9LE2GNXi9bg+8QOZPZx3WWr6IThlhoDplE/3fTrOxztxe+zRd VcyHQL/q7oBVOZx/G8L5CqIwNNsrC+rFnHJPiRk/OIPWJPitraZfpcwB0QfZZqNroW 3kbdOQpIKyY4K4lIiGOJ/aVaomg8v4hN2WHggjVA= From: Sasha Levin To: stable@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Jerome Glisse , Andrew Morton , Linus Torvalds , Sasha Levin , linux-mm@kvack.org Subject: [PATCH AUTOSEL 4.18 33/39] mm: thp: fix MADV_DONTNEED vs migrate_misplaced_transhuge_page race condition Date: Tue, 13 Nov 2018 00:50:47 -0500 Message-Id: <20181113055053.78352-33-sashal@kernel.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181113055053.78352-1-sashal@kernel.org> References: <20181113055053.78352-1-sashal@kernel.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Andrea Arcangeli [ Upstream commit d7c3393413fe7e7dc54498ea200ea94742d61e18 ] Patch series "migrate_misplaced_transhuge_page race conditions". Aaron found a new instance of the THP MADV_DONTNEED race against pmdp_clear_flush* variants, that was apparently left unfixed. While looking into the race found by Aaron, I may have found two more issues in migrate_misplaced_transhuge_page. These race conditions would not cause kernel instability, but they'd corrupt userland data or leave data non zero after MADV_DONTNEED. I did only minor testing, and I don't expect to be able to reproduce this (especially the lack of ->invalidate_range before migrate_page_copy, requires the latest iommu hardware or infiniband to reproduce). The last patch is noop for x86 and it needs further review from maintainers of archs that implement flush_cache_range() (not in CC yet). To avoid confusion, it's not the first patch that introduces the bug fixed in the second patch, even before removing the pmdp_huge_clear_flush_notify, that _notify suffix was called after migrate_page_copy already run. This patch (of 3): This is a corollary of ced108037c2aa ("thp: fix MADV_DONTNEED vs. numa balancing race"), 58ceeb6bec8 ("thp: fix MADV_DONTNEED vs. MADV_FREE race") and 5b7abeae3af8c ("thp: fix MADV_DONTNEED vs clear soft dirty race). When the above three fixes where posted Dave asked https://lkml.kernel.org/r/929b3844-aec2-0111-fef7-8002f9d4e2b9@intel.com but apparently this was missed. The pmdp_clear_flush* in migrate_misplaced_transhuge_page() was introduced in a54a407fbf7 ("mm: Close races between THP migration and PMD numa clearing"). The important part of such commit is only the part where the page lock is not released until the first do_huge_pmd_numa_page() finished disarming the pagenuma/protnone. The addition of pmdp_clear_flush() wasn't beneficial to such commit and there's no commentary about such an addition either. I guess the pmdp_clear_flush() in such commit was added just in case for safety, but it ended up introducing the MADV_DONTNEED race condition found by Aaron. At that point in time nobody thought of such kind of MADV_DONTNEED race conditions yet (they were fixed later) so the code may have looked more robust by adding the pmdp_clear_flush(). This specific race condition won't destabilize the kernel, but it can confuse userland because after MADV_DONTNEED the memory won't be zeroed out. This also optimizes the code and removes a superfluous TLB flush. [akpm@linux-foundation.org: reflow comment to 80 cols, fix grammar and typo (beacuse)] Link: http://lkml.kernel.org/r/20181013002430.698-2-aarcange@redhat.com Signed-off-by: Andrea Arcangeli Reported-by: Aaron Tomlin Acked-by: Mel Gorman Acked-by: Kirill A. Shutemov Cc: Jerome Glisse Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Sasha Levin --- mm/migrate.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index f49eb9589d73..38ad6365ed10 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2086,15 +2086,26 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); /* - * Clear the old entry under pagetable lock and establish the new PTE. - * Any parallel GUP will either observe the old page blocking on the - * page lock, block on the page table lock or observe the new page. - * The SetPageUptodate on the new page and page_add_new_anon_rmap - * guarantee the copy is visible before the pagetable update. + * Overwrite the old entry under pagetable lock and establish + * the new PTE. Any parallel GUP will either observe the old + * page blocking on the page lock, block on the page table + * lock or observe the new page. The SetPageUptodate on the + * new page and page_add_new_anon_rmap guarantee the copy is + * visible before the pagetable update. */ flush_cache_range(vma, mmun_start, mmun_end); page_add_anon_rmap(new_page, vma, mmun_start, true); - pmdp_huge_clear_flush_notify(vma, mmun_start, pmd); + /* + * At this point the pmd is numa/protnone (i.e. non present) and the TLB + * has already been flushed globally. So no TLB can be currently + * caching this non present pmd mapping. There's no need to clear the + * pmd before doing set_pmd_at(), nor to flush the TLB after + * set_pmd_at(). Clearing the pmd here would introduce a race + * condition against MADV_DONTNEED, because MADV_DONTNEED only holds the + * mmap_sem for reading. If the pmd is set to NULL at any given time, + * MADV_DONTNEED won't wait on the pmd lock and it'll skip clearing this + * pmd. + */ set_pmd_at(mm, mmun_start, pmd, entry); update_mmu_cache_pmd(vma, address, &entry); @@ -2108,7 +2119,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm, * No need to double call mmu_notifier->invalidate_range() callback as * the above pmdp_huge_clear_flush_notify() did already call it. */ - mmu_notifier_invalidate_range_only_end(mm, mmun_start, mmun_end); + mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end); /* Take an "isolate" reference and put new page on the LRU. */ get_page(new_page);