From patchwork Tue Apr 16 13:44:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903147 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5036161F for ; Tue, 16 Apr 2019 13:47:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89E43286CF for ; Tue, 16 Apr 2019 13:47:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7B514288CF; Tue, 16 Apr 2019 13:47:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EEE10288CF for ; Tue, 16 Apr 2019 13:47:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF9C36B026B; Tue, 16 Apr 2019 09:46:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CA99F6B026C; Tue, 16 Apr 2019 09:46:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1FB66B026D; Tue, 16 Apr 2019 09:46:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 647EA6B026B for ; Tue, 16 Apr 2019 09:46:57 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id d2so10671535edo.23 for ; Tue, 16 Apr 2019 06:46:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=c4uoK04ICGZxbErKax+QXjYd+oc6OVnKOXPlRP5m6go=; b=bMAA0aIMqjS+5lCf+RqOpeVpYQroaOajG1SvIa8yBJjoegZmSnDLgueYt631/oUEN+ uuoy5K745vGHhxC/ynA4MbXBlC/4bQnqJt7Ec89kBvJ+iNYRvd1pGbCBIIH2jWGhKTKX VVMPezAgIOhe7eWhL4ac70OH4SY3H6y1mq/OTMCT6bXujuaXN9U0b37jQ3H2xwAn5tx2 VvOzHRcyp8WDq8Yp7VBHAStGcco9oglaF8EjyXfszzUezegdUy7iCDU7vxd2D0sw6O3s EAxbDK3maw6g+vyuWFbHXiQ39fDefY72WcyWf5pGg379mAiASrBfTjwrIDRFOi+olbl5 T3gA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAW2jX1uWCfUxMRRnuc/LWTBVraDl+XRxPhFkGaIP/s4VYyGT+mf ESGYPCwBtJh/2LxcwVj9h7e/aI9TrAAtER+xca0Q+J2ZImDsnZEsVqx1YHKY2kQ2oTpFeOUMA8H I7wu5n4i9Z1MFcz7QuuXsv9QlNs14DnAmDvwfYS4ay8YBzunCSxX823HvEabtm3Xa8A== X-Received: by 2002:a50:8719:: with SMTP id i25mr1986226edb.172.1555422416904; Tue, 16 Apr 2019 06:46:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqwr9/5umfSZkZmRrMBbtqsVtQVZG84W9UeuBg2p2Si5DHqz+Qh3fnIx/Y7Ig8WjvL6sMpWP X-Received: by 2002:a50:8719:: with SMTP id i25mr1985935edb.172.1555422412151; Tue, 16 Apr 2019 06:46:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422412; cv=none; d=google.com; s=arc-20160816; b=krLMIOqP6skgp20/2/78P5pRdoO1aM+G5B66PqRHnTh5GdUmtz4A4UZNvYhlfAcEd/ Oc4whdUGb7ov/5SwtDY/SLFWeEdWwKM+kksmWN+q2e+KowZEQaQzVXVXsCkVjkca8MWz 5vrFeJlfpgy8WX2Xovwx9hpiZay4qV9kB40Z0Ios+07H/BEQ/UHVCaOdIjSd6YvnnyQY CdmFBHlWBs5z3OdjpP4RkiOk2IWkwOhQaJ4rMFh4EKZwMgwxWPqTmQy+YvLGV+9Gmkzo rCj6Uuep/shPrrOHchZamFYDMSU56d3t4WzOxsHhsE2kO2JaA/YE9wm4RxpjX/LdnxHf MEDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=c4uoK04ICGZxbErKax+QXjYd+oc6OVnKOXPlRP5m6go=; b=PE0aT3rNZzEHMwUANHzL0w7f5er+IwjWzB3FsOSsKAlpecOJPfCRQ2oMfaN+ECAIup uaaWQva/cGoBgGFEv+3dezUfk6Icch+O4LEM2zkRcoYHGmIsc7DE4UgR/VjjlgwkP20w BQhCOyTyfE/m/lYI9aGBu+79auWtLw37HSG5BLw6AyuwKWsedqf2gwJTF6PRp7mnikJG otaEyVrL51pmWiCO5/PHmDM57UdYUsyeO1aWZjrLF86nCDyQXqTkH2pDHNxJZsjs75ji L1iEMX4tjNIkxRr+eueg1UB/8NJcV4P3PPGzrxA2T8zijLcjP1aIxV5qtmS8J2ro+A1q MjIw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id j9si996811ejn.10.2019.04.16.06.46.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkcPa175912 for ; Tue, 16 Apr 2019 09:46:50 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwfbatrua-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:44 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:39 +0100 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:29 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjSvi36962434 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:28 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5A08D4C044; Tue, 16 Apr 2019 13:45:28 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4AD3D4C058; Tue, 16 Apr 2019 13:45:26 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:26 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 01/31] mm: introduce CONFIG_SPECULATIVE_PAGE_FAULT Date: Tue, 16 Apr 2019 15:44:52 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0008-0000-0000-000002DA6FA8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0009-0000-0000-00002246A82F Message-Id: <20190416134522.17540-2-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This configuration variable will be used to build the code needed to handle speculative page fault. By default it is turned off, and activated depending on architecture support, ARCH_HAS_PTE_SPECIAL, SMP and MMU. The architecture support is needed since the speculative page fault handler is called from the architecture's page faulting code, and some code has to be added there to handle the speculative handler. The dependency on ARCH_HAS_PTE_SPECIAL is required because vm_normal_page() does processing that is not compatible with the speculative handling in the case ARCH_HAS_PTE_SPECIAL is not set. Suggested-by: Thomas Gleixner Suggested-by: David Rientjes Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- mm/Kconfig | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 0eada3f818fa..ff278ac9978a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -761,4 +761,26 @@ config GUP_BENCHMARK config ARCH_HAS_PTE_SPECIAL bool +config ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT + def_bool n + +config SPECULATIVE_PAGE_FAULT + bool "Speculative page faults" + default y + depends on ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT + depends on ARCH_HAS_PTE_SPECIAL && MMU && SMP + help + Try to handle user space page faults without holding the mmap_sem. + + This should allow better concurrency for massively threaded processes + since the page fault handler will not wait for other thread's memory + layout change to be done, assuming that this change is done in + another part of the process's memory space. This type of page fault + is named speculative page fault. + + If the speculative page fault fails because a concurrent modification + is detected or because underlying PMD or PTE tables are not yet + allocated, the speculative page fault fails and a classic page fault + is then tried. + endmenu From patchwork Tue Apr 16 13:44:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903173 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4E83217E1 for ; Tue, 16 Apr 2019 13:48:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2DE82286CF for ; Tue, 16 Apr 2019 13:48:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 204102843C; Tue, 16 Apr 2019 13:48:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A3CED2843C for ; Tue, 16 Apr 2019 13:48:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1F7E6B028C; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ACF166B028D; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BDD06B028F; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 4A96B6B028D for ; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id w27so10987473edb.13 for ; Tue, 16 Apr 2019 06:47:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=b+MVdc1f/Z29Fjq9C3qXW5Prfsae3+JEITZmBn3UMyY=; b=lxtrwsGivizGbof3vYVYvoWQy+6r5chFvx3IfT8+yspazl8yyLeswYwBhJ8MgCrCJk tXUJSGO21bJ3Vg7ta45QqFsJeMO6jk5kjgsP8hSP1COasILPWorAS9HaKvaLgnhxu+BZ i0s9x1nYTaQDjTFJnIRd1T98TcOORUpiHk0tr4E96WdcbP2jF20vvn4XODGxDxEQgfrZ KwEfrhuIncKhkSYTlqwkdKvqg2pAuCNdRPbjR4AFx7iBSGjjysxb32/gfUI/bwACwsLJ iEoe9sqKNxNCJrjNKLR4mP+Bm0eXMLuW4Tn1JkpKbZtJutz7yqYXRdE+L6JZbyMTH6EM easQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAW/5wfOAumurDAb22tSfaocEEnAJpIwrsKjhP2eKDep8JzhL6Qi 1ncna9aPh0R5YR981c4L2tAAgOX5VuwepPmCrnVnoFhKlPeEpqxsHQmFhH9YtVWpths5pQ/4pCc 68QDoeKW+V/5kCIDqcbS7Kwjs/j6e/DI2ST2sCZLHKbKTf3+re+euNGdL7DAm6yk08Q== X-Received: by 2002:aa7:c750:: with SMTP id c16mr5722333eds.106.1555422448810; Tue, 16 Apr 2019 06:47:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqyQqL+FdnvJRQGrO4sIc6z+rZnt+rffvxR9VR85ij0Gi2VUe1ScbgNhhOfmTzFSLmVOHmZn X-Received: by 2002:aa7:c750:: with SMTP id c16mr5722277eds.106.1555422447736; Tue, 16 Apr 2019 06:47:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422447; cv=none; d=google.com; s=arc-20160816; b=RJb8Lyln6vQSPFFoc3acZnhgTVZfrYD6pqE1mLmqeIcW9GJmtv/UH8FCIumSf8QnUl K9X2UW1sv+A7XlOI7gAC5xexYZntO8i14x4z3MsZplfEoomNWNV3Z6FJtll3eXI9Q5a/ KkPAZNcArZvk0bRSAmOZXKjK/XosWQjwnf5cLLYOPedoN2GuJd8HY6kRV4kfByD+jdAF x/qUQkAfvKBuCm8zuq0ZwYen91P1ucXQZnASMAaXBdITI27hk0aRjqa519sKryCxrl4G z2/BLq/+8EBuaXbRaTIZqNmPbeDEA7xH/Htd7G6ioMOsNFTACqv61ONjfFLrz78hL+Ed 2dHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=b+MVdc1f/Z29Fjq9C3qXW5Prfsae3+JEITZmBn3UMyY=; b=hnyaGZnzSguCzLG5KMPAp0SS+H5/g5CBSX+QCWMg4+ubd7J5uD6/0k4Wh9bdtN/hcA XZ2SLYoskg3k/pL8ZuXCGaw3GU8qP8hsqatGqoeosV2Z9Z8xsNIXdR5TF32dhj1KCYLr t61IqcErQ6zdHT6mH5fHDnclRuNWDtqe/Y7NkxrwRvn1FrEmm9MQDe3J6CbW3FFrlUaT kCXwYKDxsx3oBROqoh6jTK+Wxx8LoSAaxcvNd4Oqq7DWvRS6xuGWkWPz3LvtP4qzsj0l JBbxFej7YnbrCSisc4YOjBbUFUXnXuXIHQSyTkXBH2tqS6oBoiAP2X8PKY3E3uSQOjjY lw5A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id k33si3352280edb.13.2019.04.16.06.47.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:27 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlHw7010813 for ; Tue, 16 Apr 2019 09:47:26 -0400 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe56nwxt-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:15 -0400 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:41 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:32 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjUpB46924002 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:31 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D80A44C044; Tue, 16 Apr 2019 13:45:30 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 693384C052; Tue, 16 Apr 2019 13:45:29 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:29 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 02/31] x86/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Tue, 16 Apr 2019 15:44:53 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0012-0000-0000-0000030F6EEF X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0013-0000-0000-00002147A84B Message-Id: <20190416134522.17540-3-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=877 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT which turns on the Speculative Page Fault handler when building for 64bit. Cc: Thomas Gleixner Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 0f2ab09da060..8bd575184d0b 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -30,6 +30,7 @@ config X86_64 select SWIOTLB select X86_DEV_DMA_OPS select ARCH_HAS_SYSCALL_WRAPPER + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT # # Arch settings From patchwork Tue Apr 16 13:44:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903145 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05256161F for ; Tue, 16 Apr 2019 13:47:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DE8972874E for ; Tue, 16 Apr 2019 13:47:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D29F2286CF; Tue, 16 Apr 2019 13:47:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 47C032874E for ; Tue, 16 Apr 2019 13:47:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF33F6B026A; Tue, 16 Apr 2019 09:46:56 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CA72C6B026B; Tue, 16 Apr 2019 09:46:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6E416B026C; Tue, 16 Apr 2019 09:46:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id 9351F6B026A for ; Tue, 16 Apr 2019 09:46:56 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id h125so15749529ybh.4 for ; Tue, 16 Apr 2019 06:46:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=X8WMS1D0bTB8Pmp4y6yQGdPb3gyYC2VmqSmN2VwFJ50=; b=e2V7eTuDZJqTnjwnqgIeGLqz6RQEi4wgb+yzV5PJ1LwVNYfe6IvSC9G6RfKByftVuC sVUR0Q1WcVAOpUePTnOQAvBe7Mx62brTG4B3pPj67owjD/yRFtgR5+I0YSv48Q4Lz21w D1ey9zLEoVz+Wgh2foqvaI+vlGZvPWj1XRtTHfjs/FDEZyo4N9nq6wwUYb0Tx8ggiThr 5rKk09fOGseca+32xcJTrcKdrfIn3hjElQEOU2xolY4JK/Rgnk42/ECiY/SC3adK06nE Aeruc5Srw4rjdS5unoKLz9hgGnglybN3vl4/t94B/QY7bZS1jS89NeVq2tN+kUzonGf+ GFFQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAW4Td2hFjY7oAdWXLDyLh8xlmYuYFfHpCPhLy8/oxwxogJmuXd9 obZ8yfv6MEBK370s0fRMS1XojpPjY6ha1F+apQYDwCFXyF3+Gb7WK/5Afy3urpxfvPlWNdq78DZ fOM0XV290uaSx2VllYu3sGq/QH/FPjBjT1mhzX1HLBecHvqTsxOUIvXwY/ynmD+J4iw== X-Received: by 2002:a0d:dbd7:: with SMTP id d206mr61785324ywe.332.1555422416303; Tue, 16 Apr 2019 06:46:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqxHG6dFy9KzmMVMdsW34HoFrAi/rVh8fUcOATYZwttj2D/mgcU9tq2kPeNptejXEJtTku02 X-Received: by 2002:a0d:dbd7:: with SMTP id d206mr61785217ywe.332.1555422415085; Tue, 16 Apr 2019 06:46:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422415; cv=none; d=google.com; s=arc-20160816; b=RNSjwCIA8AFriLWYwQNOkpqJAH7F86HbyrZt2sqNC6zNQTqzrSxXy6tvFTiJTdQae4 E+irn3+uYv5/WPD7jSW7Lpgf6odFQqAh3oxwg5Kn1fdefo4iTAcA664+04Q1uBJUaxqY HfU4Axrfk1+Mpe1PKjByXKlu/x7ABdQD0T82jSDAA9RUgPSLud3j7RKy3Emvp7JAVuNY r/XytJIM2wMi7b9fxDiYlMdWOriRwqydA7gZgBec2fQ3tizeSfFuhU6Vc/e95gd7ZCCb pp0BsG0ZHSLBFCfeap1HVmp3eXxqx3lhaLlYiP1JpZ1JqOQnTnrTvDiDGT6rC90xAyv2 uO5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=X8WMS1D0bTB8Pmp4y6yQGdPb3gyYC2VmqSmN2VwFJ50=; b=Rl/wmdF6ftBcbJ8xLbAvSnm/An+lLWBlNuWZKJHJHuC9UfaRicCIT7gzDItw7BCpI2 lJ6NNcVQqi+AHEqCBYRZbBIi7XQlK5O2yzcCstPHTo2kfavhLnqApOJN9vwPfuM3vZiq lylj/iJLNWJvfyf1nqKQvxug9V+8ZHV9YZdbGkwxRBj6i3OszIGiy47j3raaY1U80JYN T8PnyM4PNysWk5zkW1q9NcEgx+oWPe38+Z3krzoobV9Z/e9cJUGuT8ofNGB7dJVR5xaD /EOZB8EWmVOUZrx+zLQspix5UvWUcYNf7kqKzYPB8VQ+LsQ9NHYNIKCPD2qrMufgGZsf CO9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id e128si20158833ybe.171.2019.04.16.06.46.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:55 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkSZl132311 for ; Tue, 16 Apr 2019 09:46:54 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwdd2qv5h-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:42 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:46 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:36 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjYou50724972 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:34 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 445334C05A; Tue, 16 Apr 2019 13:45:34 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E6D874C058; Tue, 16 Apr 2019 13:45:31 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:31 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 03/31] powerpc/mm: set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Tue, 16 Apr 2019 15:44:54 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0028-0000-0000-0000036170BA X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0029-0000-0000-00002420A853 Message-Id: <20190416134522.17540-4-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=983 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for BOOK3S_64. This enables the Speculative Page Fault handler. Support is only provide for BOOK3S_64 currently because: - require CONFIG_PPC_STD_MMU because checks done in set_access_flags_filter() - require BOOK3S because we can't support for book3e_hugetlb_preload() called by update_mmu_cache() Cc: Michael Ellerman Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 2d0be82c3061..a29887ea5383 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -238,6 +238,7 @@ config PPC select PCI_SYSCALL if PCI select RTC_LIB select SPARSE_IRQ + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT if PPC_BOOK3S_64 select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK select VIRT_TO_BUS if !PPC64 From patchwork Tue Apr 16 13:44:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903127 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39B79161F for ; Tue, 16 Apr 2019 13:46:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1158D28650 for ; Tue, 16 Apr 2019 13:46:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 04CDF289FC; Tue, 16 Apr 2019 13:46:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E96D28650 for ; Tue, 16 Apr 2019 13:46:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 331546B0007; Tue, 16 Apr 2019 09:46:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E10D6B0008; Tue, 16 Apr 2019 09:46:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CED66B000A; Tue, 16 Apr 2019 09:46:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id C2E616B0007 for ; Tue, 16 Apr 2019 09:46:32 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id e22so9341060edd.9 for ; Tue, 16 Apr 2019 06:46:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=EOJUH5UxCRZTR6sReLNjhW1Ydu3e6MQFNMVn2SqWD7M=; b=mpInnqEXsjHZWofj9FDh2ihKL0/qLtWaYykhr1PVioVT9lUDWAfcy8C2XJq6R6JWYu s+kpQ+MMvQrGUqMx+ceDTs/dXGSVA3LsmgGraRFCg/CUKgBZrSEptBwOgy16mdCOPg8J p1jdr9Zt+pU/sohDuMWAVu9psvOCkWGQ2mo3QZIXPFzwojC7Tl9cebdbDy/prgBr9tZf rwI1S7Rc9n89G21J2Yo/yd4LgRTscEKH0Srof3xMxjmjoVnwiuhh2ApUJDSpVUZ70tFu clgyzgsOmlIZxhhXUZQuMxHjFDmRgFgIhEV1i6wzop4/wW51dyfWTQd2NvQoWW/57pP1 6VwQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAXCRZpAQxwreExtQZAP4ZkDQmAgfvF3ULpdvPyzeypAMsVCGdax 7KayPo/ybY/jHadu+ANrlOEwfDkptJRcX5eDFoicio//niL4WEFNULrx0CPONF9qqzR2OFxjzRX pn3Er13dF3OkFwDeuEB6lPpBtmr3b3yk2NNR7eK1b8hg2ptghAURow8joztoXqZapiw== X-Received: by 2002:a05:6402:1557:: with SMTP id p23mr10693765edx.27.1555422392151; Tue, 16 Apr 2019 06:46:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqzVfbviiGvj6JiBEGjkbFATy88UmtU9v6LaWDVnfWUHhVETFAhX9l+VqLYmB1SLB6OgfT5D X-Received: by 2002:a05:6402:1557:: with SMTP id p23mr10693693edx.27.1555422390982; Tue, 16 Apr 2019 06:46:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422390; cv=none; d=google.com; s=arc-20160816; b=mJ6vGEY3OlnEgtAOjwFnGF249viCVYYMJiZefhfv98JWHqbWAqeRCOnbv+pfJHTBOT AtchbJnkRQiQ7FRiw5vmZQQu+nSt5SKnS+AbWbWxAZ7YAoCOJBwkr9fMMl8ClNHJeL9h +cMKfHT/uuaWYT8G72isBo3UBkRd2irNPvo5xoeVHs9imlHfp/lNvleVGZHPENYpxKHX R2I+bvmEaMXg3HdrzOeixVHyPqwgddV/Bq4Y7PjB0CD6ltDM4WGAiCu7Rl2i6iksr+BK HaApkyXuVR0iEKS7rRyttGkd1/kdt0mCfetFKDm7eiWOFEr1vf+Jvirlyk63rnt0Zyf7 Ce1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=EOJUH5UxCRZTR6sReLNjhW1Ydu3e6MQFNMVn2SqWD7M=; b=IgJSSNPZI/nl/RVOFdu054DEyQkoqLV6Nr7F6KB0+AArP0PW6tLFBF6GkJmnMwVIbl rhh9evgXjdx/NUsdKrIHcLbBhq/rG1MLsGFyVl+NbYeW2MYfegQFEMQ3/vD3YX2BSfQ0 WVCqtV/KvPJT/Ks81wMBjuxInCbdc8e6vAr+p/VwevIEWrgebNxa9hEO9s6ctC9KTsWi oHibOfbE0Sl/2RXMtScxp9XnmitCYpysp8Km/suaC7N1wDaVFY2BC37UvEh3NAkZwyUF RHwRDcJIeOJUySlivhnu6gmIlsU1GT3ktyOlDEuy/xIvidKPomgJtJC+67L85NRCAS3h CZvg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id d9si10677102edq.270.2019.04.16.06.46.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:30 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkHts129524 for ; Tue, 16 Apr 2019 09:46:28 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe6cnnkx-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:26 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:48 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:38 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjaDU61276172 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:36 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C18A04C052; Tue, 16 Apr 2019 13:45:36 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 56A644C040; Tue, 16 Apr 2019 13:45:35 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:35 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 04/31] arm64/mm: define ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT Date: Tue, 16 Apr 2019 15:44:55 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0008-0000-0000-000002DA6FAB X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0009-0000-0000-00002246A833 Message-Id: <20190416134522.17540-5-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=972 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Mahendran Ganesh Set ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT for arm64. This enables Speculative Page Fault handler. Signed-off-by: Ganesh Mahendran Reviewed-by: Jérôme Glisse --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 870ef86a64ed..8e86934d598b 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -174,6 +174,7 @@ config ARM64 select SWIOTLB select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK + select ARCH_SUPPORTS_SPECULATIVE_PAGE_FAULT help ARM 64-bit (AArch64) Linux support. From patchwork Tue Apr 16 13:44:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903129 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA3FD17E1 for ; Tue, 16 Apr 2019 13:46:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEF8E286FF for ; Tue, 16 Apr 2019 13:46:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B220B28947; Tue, 16 Apr 2019 13:46:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3D872874E for ; Tue, 16 Apr 2019 13:46:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2DB16B0008; Tue, 16 Apr 2019 09:46:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DDCEA6B000A; Tue, 16 Apr 2019 09:46:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAF196B000C; Tue, 16 Apr 2019 09:46:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 7763A6B0008 for ; Tue, 16 Apr 2019 09:46:43 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id j3so10977001edb.14 for ; Tue, 16 Apr 2019 06:46:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=9jZTtITuVwuX3A0pUuEay1zff/oOC++asy4NappICdo=; b=m/4fMsH2rHXVH95ncpPhbxGochB9R306CnBIw+3RXHjbBI4GINu6WIBrWPJiwqaTU/ wVfDBC2nEJtRcsE5wlRKGCCE0/l4DPjcgQLors4VrQ2rqOn/spd2CvIzEjjF++bgc8KU yiXhHzJrZsuJCh8c+YXK8E5v48amHhOYeQdmLlwZU7aByL8+jX3o/cXIsp2prrah3vil xsbR8eLZo/QmQJx5QyINUtXG13wrEsLdyObJkzZ5spnUA64D2LyvFO9Pfoax78RNuxCs MfTUJMTCpJp53IoUNifx6YJv23VdNuTuY6swBpMInmApJpXdNUAxymFXvhvAlLzwl7Tt 8wYA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVp8ljD5nbX0NdiKAyUO+KlYRAMVPpa+MRdZpOXna43ldyfb5jO DPPR0bNqOHLhq5Oybwxc0JG8b82uloHu2Sj9AfGk7h9rhJTrbW7rQQ7BG+UmbFBEhhICmhXunNq 1plkxFoB3P07F6MPOELq8iyQSLG7+cUzC4DcsSIvY2SVxrh6/rkNJyk+0cw68Ddg5aA== X-Received: by 2002:a17:906:4c4e:: with SMTP id d14mr35794765ejw.127.1555422402612; Tue, 16 Apr 2019 06:46:42 -0700 (PDT) X-Google-Smtp-Source: APXvYqzjpPyvahpXWSfwo8OAK+mWHzC8mXTh/UzTebasX1s2ZUP0WjcC0O02IuWiNOPE6BaC3duf X-Received: by 2002:a17:906:4c4e:: with SMTP id d14mr35794688ejw.127.1555422400770; Tue, 16 Apr 2019 06:46:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422400; cv=none; d=google.com; s=arc-20160816; b=HVOXx7BoST8T8XkEJQQrX360wRxB9+ErqGT/1aEu1S7iN2MzcIgusWfhWVxg4ImxDT I1nyllnOYhWC6BDq0U/j9KDlLAPO+BabjAiHVOlRG+YFBS154eZ6IMPJ/XIjvXBHdlza XfSaazzT3Ws4tA0YUMJwUH6zsxIJN2fdzP/K/z+E7JFYXmwSgrwWiTJt2Fx9qG+uI+nI qALr9v0dQIi/LkZLcf1H79srbxakxM+vzVjvxd/ahBh1WqW5uYTqzJYHvKcMISbyLwDb EFaJR8wTJV8cK91iBdnDhfowkYrjhMx/Z75dhFRl/Bsc5LpW4TUYZRzkTMsA14NYXoqY hJLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=9jZTtITuVwuX3A0pUuEay1zff/oOC++asy4NappICdo=; b=nTZresSQ1CqW7oJIrWT12VROmd5ithdyKAP2OY2mwUHNWYRSwgRWlEzvYy4wyfi3JY Ktc+//qofOOneE5S4F+4wHhwbkUKIyJv7w82oyzZSAuNTZiT6xyhSvdrwJq+ZMgwHviT lzRUTptUV54VZeGFbczYin4xYzm06yku22tvvT6vW6u1fa7ZW58GZ7NznfgadAzQp8Tm w+8FeC4+K61TOzQO7onU73pXX7I1Uh5wg+a1G1m7GwkaDOtkI89a6zLTWWHBJsQmJwgI sVAZzG/bbHtaciXht2Nt7vtqlSd+T+RsqciY8nHcPlZlNnocw/dqC6t2cAC38QRP0inN JwYg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id l3si2135818edn.275.2019.04.16.06.46.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:40 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkVcp113139 for ; Tue, 16 Apr 2019 09:46:38 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwentmtj3-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:36 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:50 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:41 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjdqX46661836 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:39 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4BA094C052; Tue, 16 Apr 2019 13:45:39 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D1B344C046; Tue, 16 Apr 2019 13:45:37 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:37 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 05/31] mm: prepare for FAULT_FLAG_SPECULATIVE Date: Tue, 16 Apr 2019 15:44:56 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-4275-0000-0000-00000328764E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-4276-0000-0000-00003837A74F Message-Id: <20190416134522.17540-6-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Peter Zijlstra When speculating faults (without holding mmap_sem) we need to validate that the vma against which we loaded pages is still valid when we're ready to install the new PTE. Therefore, replace the pte_offset_map_lock() calls that (re)take the PTL with pte_map_lock() which can fail in case we find the VMA changed since we started the fault. Signed-off-by: Peter Zijlstra (Intel) [Port to 4.12 kernel] [Remove the comment about the fault_env structure which has been implemented as the vm_fault structure in the kernel] [move pte_map_lock()'s definition upper in the file] [move the define of FAULT_FLAG_SPECULATIVE later in the series] [review error path in do_swap_page(), do_anonymous_page() and wp_page_copy()] Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- mm/memory.c | 87 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 58 insertions(+), 29 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c6ddadd9d2b7..fc3698d13cb5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2073,6 +2073,13 @@ int apply_to_page_range(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(apply_to_page_range); +static inline bool pte_map_lock(struct vm_fault *vmf) +{ + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + return true; +} + /* * handle_pte_fault chooses page fault handler according to an entry which was * read non-atomically. Before making any commitment, on those architectures @@ -2261,25 +2268,26 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) int page_copied = 0; struct mem_cgroup *memcg; struct mmu_notifier_range range; + int ret = VM_FAULT_OOM; if (unlikely(anon_vma_prepare(vma))) - goto oom; + goto out; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { new_page = alloc_zeroed_user_highpage_movable(vma, vmf->address); if (!new_page) - goto oom; + goto out; } else { new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address); if (!new_page) - goto oom; + goto out; cow_user_page(new_page, old_page, vmf->address, vma); } if (mem_cgroup_try_charge_delay(new_page, mm, GFP_KERNEL, &memcg, false)) - goto oom_free_new; + goto out_free_new; __SetPageUptodate(new_page); @@ -2291,7 +2299,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) /* * Re-check the pte - we dropped the lock */ - vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + goto out_uncharge; + } if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { if (old_page) { if (!PageAnon(old_page)) { @@ -2378,12 +2389,14 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) put_page(old_page); } return page_copied ? VM_FAULT_WRITE : 0; -oom_free_new: +out_uncharge: + mem_cgroup_cancel_charge(new_page, memcg, false); +out_free_new: put_page(new_page); -oom: +out: if (old_page) put_page(old_page); - return VM_FAULT_OOM; + return ret; } /** @@ -2405,8 +2418,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) { WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED)); - vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) + return VM_FAULT_RETRY; /* * We might have raced with another page fault while we released the * pte_offset_map_lock. @@ -2527,8 +2540,11 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) get_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); lock_page(vmf->page); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) { + unlock_page(vmf->page); + put_page(vmf->page); + return VM_FAULT_RETRY; + } if (!pte_same(*vmf->pte, vmf->orig_pte)) { unlock_page(vmf->page); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2744,11 +2760,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!page) { /* - * Back out if somebody else faulted in this pte - * while we released the pte lock. + * Back out if the VMA has changed in our back during + * a speculative page fault or if somebody else + * faulted in this pte while we released the pte lock. */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) { + delayacct_clear_flag(DELAYACCT_PF_SWAPIN); + ret = VM_FAULT_RETRY; + goto out; + } if (likely(pte_same(*vmf->pte, vmf->orig_pte))) ret = VM_FAULT_OOM; delayacct_clear_flag(DELAYACCT_PF_SWAPIN); @@ -2801,10 +2821,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* - * Back out if somebody else already faulted in this pte. + * Back out if the VMA has changed in our back during a speculative + * page fault or if somebody else already faulted in this pte. */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; + goto out_cancel_cgroup; + } if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; @@ -2882,8 +2905,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) out: return ret; out_nomap: - mem_cgroup_cancel_charge(page, memcg, false); pte_unmap_unlock(vmf->pte, vmf->ptl); +out_cancel_cgroup: + mem_cgroup_cancel_charge(page, memcg, false); out_page: unlock_page(page); out_release: @@ -2934,8 +2958,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) !mm_forbids_zeropage(vma->vm_mm)) { entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address), vma->vm_page_prot)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + if (!pte_map_lock(vmf)) + return VM_FAULT_RETRY; if (!pte_none(*vmf->pte)) goto unlock; ret = check_stable_address_space(vma->vm_mm); @@ -2971,14 +2995,16 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); - if (!pte_none(*vmf->pte)) + if (!pte_map_lock(vmf)) { + ret = VM_FAULT_RETRY; goto release; + } + if (!pte_none(*vmf->pte)) + goto unlock_and_release; ret = check_stable_address_space(vma->vm_mm); if (ret) - goto release; + goto unlock_and_release; /* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { @@ -3000,10 +3026,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; +unlock_and_release: + pte_unmap_unlock(vmf->pte, vmf->ptl); release: mem_cgroup_cancel_charge(page, memcg, false); put_page(page); - goto unlock; + return ret; oom_free_page: put_page(page); oom: @@ -3118,8 +3146,9 @@ static vm_fault_t pte_alloc_one_map(struct vm_fault *vmf) * pte_none() under vmf->ptl protection when we return to * alloc_set_pte(). */ - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + if (!pte_map_lock(vmf)) + return VM_FAULT_RETRY; + return 0; } From patchwork Tue Apr 16 13:44:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903183 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7B918161F for ; Tue, 16 Apr 2019 13:48:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5CC45286CF for ; Tue, 16 Apr 2019 13:48:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4FA23286FF; Tue, 16 Apr 2019 13:48:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D011B286CF for ; Tue, 16 Apr 2019 13:48:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CC066B0296; Tue, 16 Apr 2019 09:47:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 380376B0298; Tue, 16 Apr 2019 09:47:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26C136B0299; Tue, 16 Apr 2019 09:47:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id 0697E6B0296 for ; Tue, 16 Apr 2019 09:47:52 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id g186so15674158ybg.16 for ; Tue, 16 Apr 2019 06:47:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=h+dSz3dIugqGQM4BNY4sysZNpxbYjrXmKmAZ5NCn5zQ=; b=QyP5WRPSHJAqyKLfyKhADlXtBO2aXIFZBi0LmloNa8c3mHyMF9XUoqJKrzPOrKMmQz YBTRl6ZZHGT7pm1BmVJ0nhLF99WdU8qAy8o/GoAvwgVtppn/REaGQ0vAuJYStqtxQANb ipGsg1YtqUv6689tim1mTg0vAsOF6no+4ACEvL05Kxsx6Huq0H2spEdzZsC00apasr7W 0lHqHp4kc3cVSnUrn4E/htcKXopthj38Eg2D8v96n7fTVTSdo2rH10rlP9CmP0aSjBJL GSv9rV/OqDtVAA5p58FQYfOzivEJIDeHkOX2qRRIZ6Fqb48/BYG/8Oj/jGdi43ib3hxf 0MYA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVsBJv7SvYxr2b+5QOOc01bQzI54nmLV/F+6n7sbAfZMxuq/Y17 TCUrr4XT4ViOFCNXRXwW264JgKNBXyam/z8vN/GS0AqA8YILUSgVRA+tQhunZn9uhtTWQm12lp8 uBcyZK9OMrEea9lluXORMdN5t4wB4KhjE3HFm22ZS1N0W4OL0MksBSeuKBtT/o2eO4A== X-Received: by 2002:a25:c8c7:: with SMTP id y190mr15979914ybf.285.1555422471763; Tue, 16 Apr 2019 06:47:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqyQchnBrhhdSDL7Yz6aZZYHC/sMs+WJPhfCvMh+VmTFKElfBus83ypuJmvlxtgMXpXJo+jV X-Received: by 2002:a25:c8c7:: with SMTP id y190mr15979820ybf.285.1555422470624; Tue, 16 Apr 2019 06:47:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422470; cv=none; d=google.com; s=arc-20160816; b=Q5puU3BDxlWUj/iMSi1ozBSEKVkHsGphwA5YmmMtikVepjZ6F4QXOeTpDJ+U9gyQ1g qloBEhxGjBniNhDXL7lZ2cigSxYPN8y4y6opczwyK2RWxFKvxGUWtS+Rzi00n1jZzjX8 mf+5H3ur7nO6ye6c7O4o+BhGQ8RwVNLE0pN9OTrlVhy+JzliBw4n1QtMktgP5bxinE8F Bu054hCSNUocGCkKslJ/7UIBdIL6W9HM+TnzDYN7TxmXFFLnDEqGCEkOsS14aV6AQHhr to7p3d4GSOY/OQuLxZQR1r6n88fFn1gdAYe8iyJe3+mozZsueF258LwriXOhJky0jhb9 9WfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=h+dSz3dIugqGQM4BNY4sysZNpxbYjrXmKmAZ5NCn5zQ=; b=Q3RM39NP6IYYdI0jGv0WiXYTJfh8hD4gaCWVfs+iS6ZEXi3hdczmYO2JGrC5Dmjnza uWNuvoN+A7fR94UP/o9zdK2o/wIqPn6+Hhx3MkC6olKFhg6dUyWd0YWlOamvFJwoxrCF wMZopx8bRpiwbEK/+Ow4ppMnjSttczWfyBC0K/REd3rilYlOo+PFZwMLff6rHTMsz8KW pJJyZ3P2rAaazSOfZkU7LOB4Uw3NVaaP44u/f+aPJrRi+w3m9dS0d86Lnz3m5cEe/yxg lJH35pKpNrG7wY/BoDzncDbRYaV7JWREENqHvZ6qegar1MP8aPlR4XFIGgj6OYY2Sgf/ Whsg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id x17si10550696ybj.198.2019.04.16.06.47.50 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:50 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlT6E093889 for ; Tue, 16 Apr 2019 09:47:49 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe1t6d9k-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:41 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:53 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:43 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjfqs27262988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:41 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C7A184C062; Tue, 16 Apr 2019 13:45:41 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5A8EA4C058; Tue, 16 Apr 2019 13:45:40 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:40 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 06/31] mm: introduce pte_spinlock for FAULT_FLAG_SPECULATIVE Date: Tue, 16 Apr 2019 15:44:57 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7267 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD77 Message-Id: <20190416134522.17540-7-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When handling page fault without holding the mmap_sem the fetch of the pte lock pointer and the locking will have to be done while ensuring that the VMA is not touched in our back. So move the fetch and locking operations in a dedicated function. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- mm/memory.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index fc3698d13cb5..221ccdf34991 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2073,6 +2073,13 @@ int apply_to_page_range(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(apply_to_page_range); +static inline bool pte_spinlock(struct vm_fault *vmf) +{ + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + spin_lock(vmf->ptl); + return true; +} + static inline bool pte_map_lock(struct vm_fault *vmf) { vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, @@ -3656,8 +3663,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * validation through pte_unmap_same(). It's of NUMA type but * the pfn may be screwed if the read is non atomic. */ - vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd); - spin_lock(vmf->ptl); + if (!pte_spinlock(vmf)) + return VM_FAULT_RETRY; if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; @@ -3850,8 +3857,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); - vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); - spin_lock(vmf->ptl); + if (!pte_spinlock(vmf)) + return VM_FAULT_RETRY; entry = vmf->orig_pte; if (unlikely(!pte_same(*vmf->pte, entry))) goto unlock; From patchwork Tue Apr 16 13:44:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903131 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA30517E1 for ; Tue, 16 Apr 2019 13:46:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88DBF286CF for ; Tue, 16 Apr 2019 13:46:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7C69128A08; Tue, 16 Apr 2019 13:46:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41366286CF for ; Tue, 16 Apr 2019 13:46:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5DC36B000A; Tue, 16 Apr 2019 09:46:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A33356B000C; Tue, 16 Apr 2019 09:46:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 921BE6B000D; Tue, 16 Apr 2019 09:46:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 3EA246B000A for ; Tue, 16 Apr 2019 09:46:48 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id h10so2910308edn.22 for ; Tue, 16 Apr 2019 06:46:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=VpgllxzPc37sNU/qdZLcQd5VeTMUV+mWthl8cxM+qN0=; b=bWvd6+NhapisvdyBw7oRux20ShGt7wymfBGHvSThZvPeOz/e4sxQaSmGkp98kTz69S o87JqhotmSenwtAI834l7OWBngvGDza+8oxtzm52sp3Dz0ATxm4KSyyAkm1URechz61v jdApBZk9B16hp02/sqsGl5tKYRbo92lhqr/xslYFriI03HswdsFvmzFAlGKjqc8jnBtg i+wAnNEsqqC+kIShJW5egHPmYQX/J1qq1RZ3N2NMr1TAghl9Ku6Nv7BSibjNwzR7sUPa YRBg5ffkrkcSETaobXQf3yLdxSznkhp63vSObjIP8LcBJWixDq9Gu2kup1ei00i093Yf +kcQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAU4TcwPyOyyKHUSWeBhClSBaeS3MrJ2HPfo37lP9xiZJVHAAjd5 ISN2eDsqClkz+IpMhdIIJsigCNzpcDNfiSXOqo78cayT3uHXFlkT7zCUzJYtR0lMZ49ePAQXUFM cPEiw4XEYbZAQu34di1J8KoqatymqQykTqiIkot8MatCrd86Uxs+d8+T4ZlAa5gUqsA== X-Received: by 2002:aa7:c38c:: with SMTP id k12mr6079446edq.33.1555422407718; Tue, 16 Apr 2019 06:46:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqwwyRlBeKA/f7fGRXUEQGuKi1QuzZLqer1ProhCj9xyHk3+Zuk7JxqQEsfLh6uSP4XKZmpK X-Received: by 2002:aa7:c38c:: with SMTP id k12mr6079370edq.33.1555422406406; Tue, 16 Apr 2019 06:46:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422406; cv=none; d=google.com; s=arc-20160816; b=rpZp3+4lzHot1jyTb7lf4dzYiTskbNbOLLUehX8eLj3n8GxVxKmDQgLgqWL/vg5Bas IfgpujExMcG58In+Q9WESlo/Kge/gpdLn6ysKxAQ2gzdFPrPQlEnvJXMf681C3PMOJhz 3LI4Ws015MWnS5hj+LgOiDp21p60DT5NQJo9sMJ+D7FcwAcIqeK+iQWemjW/7XJ0wpKh 5nY/cy5pKHJq5tWOx5jGFVGQigecyxx31K61pvhiuG63nd2ZXzBOwoIK3xO67I7xkpHj V2rjdaOGpOMDy37lVqb6+mu8em4mLwOtZZwi8kNVnAFUm+c60m3HKPmXmoSBrw/LXXI/ /ENA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=VpgllxzPc37sNU/qdZLcQd5VeTMUV+mWthl8cxM+qN0=; b=B4LZlp8UHYy+iKgS6xc5Nb8rWaWGH5JpwwTHpSEFWxblwt1FQkiNWC3WiG8LM+amoG jXbuyGXzcG75NV5QZxQ8xEmnT4Q7ZF3ErX3iKf0sXwjW9cm4hjHyD3HTlO6zYejuPxXs VQNX0tt0af1gkhBMRb0gf8WozyM3ki7xC4W9HLZ0OuJwshJQz9/rPuwDlKHR1+/CVrMH gdvxSLtMOArT00ZlRB6XNMpH2ZjWV5jvBmVEgTYrJ/obvwMLt7GgqCRvFtast20svktH CYXHK+l4930a+xVxwk/6UymLMzetvh7xAprA/gZ4X2XiNlDgbn9xtgATAuJ12O9qcnyD T8+A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id jp19si248245ejb.116.2019.04.16.06.46.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:46 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkd9T178780 for ; Tue, 16 Apr 2019 09:46:45 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwfpusewg-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:42 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:56 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:46 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjiab16646206 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:44 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5066E4C052; Tue, 16 Apr 2019 13:45:44 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D71B34C062; Tue, 16 Apr 2019 13:45:42 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:42 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 07/31] mm: make pte_unmap_same compatible with SPF Date: Tue, 16 Apr 2019 15:44:58 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7268 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD7B Message-Id: <20190416134522.17540-8-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP pte_unmap_same() is making the assumption that the page table are still around because the mmap_sem is held. This is no more the case when running a speculative page fault and additional check must be made to ensure that the final page table are still there. This is now done by calling pte_spinlock() to check for the VMA's consistency while locking for the page tables. This is requiring passing a vm_fault structure to pte_unmap_same() which is containing all the needed parameters. As pte_spinlock() may fail in the case of a speculative page fault, if the VMA has been touched in our back, pte_unmap_same() should now return 3 cases : 1. pte are the same (0) 2. pte are different (VM_FAULT_PTNOTSAME) 3. a VMA's changes has been detected (VM_FAULT_RETRY) The case 2 is handled by the introduction of a new VM_FAULT flag named VM_FAULT_PTNOTSAME which is then trapped in cow_user_page(). If VM_FAULT_RETRY is returned, it is passed up to the callers to retry the page fault while holding the mmap_sem. Acked-by: David Rientjes Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm_types.h | 6 +++++- mm/memory.c | 37 +++++++++++++++++++++++++++---------- 2 files changed, 32 insertions(+), 11 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8ec38b11b361..fd7d38ee2e33 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -652,6 +652,8 @@ typedef __bitwise unsigned int vm_fault_t; * @VM_FAULT_NEEDDSYNC: ->fault did not modify page tables and needs * fsync() to complete (for synchronous page faults * in DAX) + * @VM_FAULT_PTNOTSAME Page table entries have changed during a + * speculative page fault handling. * @VM_FAULT_HINDEX_MASK: mask HINDEX value * */ @@ -669,6 +671,7 @@ enum vm_fault_reason { VM_FAULT_FALLBACK = (__force vm_fault_t)0x000800, VM_FAULT_DONE_COW = (__force vm_fault_t)0x001000, VM_FAULT_NEEDDSYNC = (__force vm_fault_t)0x002000, + VM_FAULT_PTNOTSAME = (__force vm_fault_t)0x004000, VM_FAULT_HINDEX_MASK = (__force vm_fault_t)0x0f0000, }; @@ -693,7 +696,8 @@ enum vm_fault_reason { { VM_FAULT_RETRY, "RETRY" }, \ { VM_FAULT_FALLBACK, "FALLBACK" }, \ { VM_FAULT_DONE_COW, "DONE_COW" }, \ - { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" } + { VM_FAULT_NEEDDSYNC, "NEEDDSYNC" }, \ + { VM_FAULT_PTNOTSAME, "PTNOTSAME" } struct vm_special_mapping { const char *name; /* The name, e.g. "[vdso]". */ diff --git a/mm/memory.c b/mm/memory.c index 221ccdf34991..d5bebca47d98 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2094,21 +2094,29 @@ static inline bool pte_map_lock(struct vm_fault *vmf) * parts, do_swap_page must check under lock before unmapping the pte and * proceeding (but do_wp_page is only called after already making such a check; * and do_anonymous_page can safely check later on). + * + * pte_unmap_same() returns: + * 0 if the PTE are the same + * VM_FAULT_PTNOTSAME if the PTE are different + * VM_FAULT_RETRY if the VMA has changed in our back during + * a speculative page fault handling. */ -static inline int pte_unmap_same(struct mm_struct *mm, pmd_t *pmd, - pte_t *page_table, pte_t orig_pte) +static inline vm_fault_t pte_unmap_same(struct vm_fault *vmf) { - int same = 1; + int ret = 0; + #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT) if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(mm, pmd); - spin_lock(ptl); - same = pte_same(*page_table, orig_pte); - spin_unlock(ptl); + if (pte_spinlock(vmf)) { + if (!pte_same(*vmf->pte, vmf->orig_pte)) + ret = VM_FAULT_PTNOTSAME; + spin_unlock(vmf->ptl); + } else + ret = VM_FAULT_RETRY; } #endif - pte_unmap(page_table); - return same; + pte_unmap(vmf->pte); + return ret; } static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma) @@ -2714,8 +2722,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) int exclusive = 0; vm_fault_t ret = 0; - if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte)) + ret = pte_unmap_same(vmf); + if (ret) { + /* + * If pte != orig_pte, this means another thread did the + * swap operation in our back. + * So nothing else to do. + */ + if (ret == VM_FAULT_PTNOTSAME) + ret = 0; goto out; + } entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { From patchwork Tue Apr 16 13:44:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903167 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 106ED17E1 for ; Tue, 16 Apr 2019 13:48:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E5BCE2843C for ; Tue, 16 Apr 2019 13:48:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9513286CF; Tue, 16 Apr 2019 13:48:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55DC62843C for ; Tue, 16 Apr 2019 13:48:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 048F06B0276; Tue, 16 Apr 2019 09:47:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F3D7F6B0277; Tue, 16 Apr 2019 09:47:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDD3E6B0278; Tue, 16 Apr 2019 09:47:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id B408E6B0276 for ; Tue, 16 Apr 2019 09:47:19 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id h125so15750165ybh.4 for ; Tue, 16 Apr 2019 06:47:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=3+OIxY87hN1e4nLQTbE+bCfUvLBlUDF54UIVplG5FKA=; b=GrwvOfGhFYPdjUK0/KlaY7F2rKZDmSCjNnPxZkIg22fCgXR/5fKUA7eDRpV8IISoLN qfJi8dqvBGE9i0uts5T5TlPZK95Eldink4x+wzCAN1iuBsqTbD3mboujSdZaazKyeghu N6fpbebSjS0uyUiu/UixWYZzqFnGo1j2krPWGVhzLWPdQqAqfDkNfFdqTenBBVXszIR6 1Ghu+Rs9bPp1Qf5qISopXoolZxtmhSvLYsfMQ07FrweYDyS6Ny/nTd3byHo5szxmteJd M8cAA7LFVT2WsNjhh5yTdKpanwgQXpzTkDyH/jd5Pw2Kn9gqsu5ExzKDtrT7T2tgn7IN gG0Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAV9No21z8+nt90qY8wEcIJSKJSd/OdKBeeRx+hr5YtU9ZhDUdgJ 6uMMqMU+uUhbmzfVIc7ghhIW5M0y76P7ED7nu6XfSESoqxbEyoAMXcGI1NcMcw+NkCAiik6sYL+ Q+qPN/VlYaW0IzQNNNjkFddlgIW0LcuTXddt1YJVOsGip8By6Q3IkKzPVAW3W2ENOSg== X-Received: by 2002:a81:29ce:: with SMTP id p197mr63968443ywp.350.1555422439448; Tue, 16 Apr 2019 06:47:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqzQAfd//yll6GRHTUUV5o3dDhUUiHW/u/EOLDntuIHFutDF58tWSM+Huj1fwI8PJrcr9A5v X-Received: by 2002:a81:29ce:: with SMTP id p197mr63968337ywp.350.1555422438142; Tue, 16 Apr 2019 06:47:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422438; cv=none; d=google.com; s=arc-20160816; b=fcVn0RjjyoM86YPmCr1/b5vQGtadBEyGE4Zp7y+wWaLEoGNcbfvUk3pxCtCf1P/FrO JrMKyxu+omW+rwRg58ZoKwmmsWMoI0kXEAXDevx3NEqSactSIGR8PdUjHBALchAQS3um 6JFL1Ba4DwSO8ehDRNsZK5UNRhk7WBG0ME1x5pmV0lz777f0Pf3DjfsTQ2r7X6EKYRVr gS/4BAPfBLDdv1CfAXW3cWdm842SgjF5BUPy1jjSW2+l7bBDyE01KCuDvfyzglOl+hwG PBgnAPqP7OvoHAK58/5lBm1b57QTAFS7MHu9i1rqIVT8MCzDSZ2A+OIfqaYNaMoM1naB TAaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=3+OIxY87hN1e4nLQTbE+bCfUvLBlUDF54UIVplG5FKA=; b=UKMvLykpdBCmJhVNS4tG3gBFg/uJKOO1pROU2UQg+mT6tU+w32nxvLODwfachH0sKh Q5HA7mr05UGXzd50q1L8sUSXSY3tIrNcgKCNkIHngX1KaUuwn9u4wiRoJvSl6bYTmgl/ YZ1M98e7pbLCLvPFv8XkeQynhHLgR9/1uuEkMgtZ3Dna9UiC4fq22XUpBHBR9EQMclsQ 07dDsk9oD/d6FdmUx1gB5PfzWTAAoREdNCiTQX27jhwEq6u0Vq5TZ5CKrj6aHddxah64 OYisq8WTmzRC6gF23IoovsTD2pSaEAWxOVS6DSKRM9V0POKA5bIcJIuIU7BWep/F1/yE 9gWw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id q127si17114847ybq.70.2019.04.16.06.47.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkuNd060182 for ; Tue, 16 Apr 2019 09:47:17 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe36nxw9-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:11 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:45:59 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:48 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjkYU44237050 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:47 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C29A84C050; Tue, 16 Apr 2019 13:45:46 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 601804C062; Tue, 16 Apr 2019 13:45:45 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:45 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 08/31] mm: introduce INIT_VMA() Date: Tue, 16 Apr 2019 15:44:59 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F726C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD7E Message-Id: <20190416134522.17540-9-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Some VMA struct fields need to be initialized once the VMA structure is allocated. Currently this only concerns anon_vma_chain field but some other will be added to support the speculative page fault. Instead of spreading the initialization calls all over the code, let's introduce a dedicated inline function. Signed-off-by: Laurent Dufour --- fs/exec.c | 1 + include/linux/mm.h | 5 +++++ kernel/fork.c | 2 +- mm/mmap.c | 3 +++ mm/nommu.c | 1 + 5 files changed, 11 insertions(+), 1 deletion(-) diff --git a/fs/exec.c b/fs/exec.c index 2e0033348d8e..9762e060295c 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -266,6 +266,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) vma->vm_start = vma->vm_end - PAGE_SIZE; vma->vm_flags = VM_SOFTDIRTY | VM_STACK_FLAGS | VM_STACK_INCOMPLETE_SETUP; vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); + INIT_VMA(vma); err = insert_vm_struct(mm, vma); if (err) diff --git a/include/linux/mm.h b/include/linux/mm.h index 4ba2f53f9d60..2ceb1d2869a6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1407,6 +1407,11 @@ struct zap_details { pgoff_t last_index; /* Highest page->index to unmap */ }; +static inline void INIT_VMA(struct vm_area_struct *vma) +{ + INIT_LIST_HEAD(&vma->anon_vma_chain); +} + struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte, bool with_public_device); #define vm_normal_page(vma, addr, pte) _vm_normal_page(vma, addr, pte, false) diff --git a/kernel/fork.c b/kernel/fork.c index 915be4918a2b..f8dae021c2e5 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -341,7 +341,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) if (new) { *new = *orig; - INIT_LIST_HEAD(&new->anon_vma_chain); + INIT_VMA(new); } return new; } diff --git a/mm/mmap.c b/mm/mmap.c index bd7b9f293b39..5ad3a3228d76 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1765,6 +1765,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_flags = vm_flags; vma->vm_page_prot = vm_get_page_prot(vm_flags); vma->vm_pgoff = pgoff; + INIT_VMA(vma); if (file) { if (vm_flags & VM_DENYWRITE) { @@ -3037,6 +3038,7 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla } vma_set_anonymous(vma); + INIT_VMA(vma); vma->vm_start = addr; vma->vm_end = addr + len; vma->vm_pgoff = pgoff; @@ -3395,6 +3397,7 @@ static struct vm_area_struct *__install_special_mapping( if (unlikely(vma == NULL)) return ERR_PTR(-ENOMEM); + INIT_VMA(vma); vma->vm_start = addr; vma->vm_end = addr + len; diff --git a/mm/nommu.c b/mm/nommu.c index 749276beb109..acf7ca72ca90 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1210,6 +1210,7 @@ unsigned long do_mmap(struct file *file, region->vm_flags = vm_flags; region->vm_pgoff = pgoff; + INIT_VMA(vma); vma->vm_flags = vm_flags; vma->vm_pgoff = pgoff; From patchwork Tue Apr 16 13:45:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903175 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 11077161F for ; Tue, 16 Apr 2019 13:48:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E9AC82867F for ; Tue, 16 Apr 2019 13:48:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DB6EA286F3; Tue, 16 Apr 2019 13:48:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BE24B286CF for ; Tue, 16 Apr 2019 13:48:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06F926B028D; Tue, 16 Apr 2019 09:47:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 01F836B0290; Tue, 16 Apr 2019 09:47:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E07A36B0291; Tue, 16 Apr 2019 09:47:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id B78DA6B028D for ; Tue, 16 Apr 2019 09:47:34 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id a75so15577030ywh.8 for ; Tue, 16 Apr 2019 06:47:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=kUz3G0QDyjTuW2R2fjyO6alqd0rMD6GtLnwwAlb/tIs=; b=rXVu32TTsqtghqfKi+HEZFHwaIz8akQJr9FyFwW+LNhyJCCj0IM43tF65s5R4MD9Uc 0Mf8iasf+41eg+a8DiZhWjJPyuhI/a8nxVN6VA1acGJLLvLitr8/ST0QhirhbPuDSnxN EdujeX73uCtFxVXyABiAqpJzQQc58YqlIjeU2RwR5UqORIqi4W4ckdSHDmdjhAWJKcbs BRnrZKAcr3VJoih8Ty4OSGstC5EIlaYZUgP8GF2+Ho/RzstpVe5h0CXzmJADlribIHFO 3q9u6Rjp1KBjVnfByMsVLUeTLWI8EmcU500J89i9s4B0YZhD6vJhLTfyViNk7fXi/Z8G 7Trw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAUQNF0IfawYhWEdEwvnsagUl65ikEgcg0yhWADJiMiMBbJiokp3 UKwpvHwQYszZRT6n9SJXE1WMym1RzKxrHSOUr1JhxbSv/Qe8xCUTGcG60Xm9LUs25+IWDOrAOV1 Gsu812AwUo1WY7VBNZqvb6Dklv+CpRx/ZDnCulVnLZ7p/5XH0x8AV5gm92y5aN5He1A== X-Received: by 2002:a5b:4c8:: with SMTP id u8mr1406905ybp.291.1555422454404; Tue, 16 Apr 2019 06:47:34 -0700 (PDT) X-Google-Smtp-Source: APXvYqxyoZWyXd895k3WhFq3drx2t1ASveKg6+iRG2uYCLCbwhMJHDCHZ6fjX378FBHSEQNEaCBa X-Received: by 2002:a5b:4c8:: with SMTP id u8mr1406830ybp.291.1555422453473; Tue, 16 Apr 2019 06:47:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422453; cv=none; d=google.com; s=arc-20160816; b=x41J7rbugwIt5ZI3e2uXrUM3TYowEXbpW1uBfQJcYO/92kLnGCaqdIywp4GaTUTneF 5gbE2VVR8eNSuhVadum1xVQXDsIoG41nUiDVnVxmIO5mByNxqjoYJqW6BR2vk3DTWrUF fwRUqSQZvtW7iQsMhLW7gHBWB1myb8UVASde89YY761DvFH5NAHAzRpzk2EckVoUCxoL AtzQwFMxtUI8BljYgPXyv+oeoYlzMR9jh11v2jNyeyvL+49AB3QENkJbsHchAiQHHY66 mcgzPhO/QVPqvKTSw+J8VYiK5Rj4yWLx63mqhmvP9xa0OPO9ErvE1zApGt1IKxo6wB6D hXYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=kUz3G0QDyjTuW2R2fjyO6alqd0rMD6GtLnwwAlb/tIs=; b=xwRLtvtLvHsaOv/kBJajmL5IdQQ66tQK8UT4+5f4VKUlZwh3eEyBKGmOxHFEwCARHb tWoWqzhnDEPLWclzCzAaSDCMxNcHhs0Jx8NzTsUkIHtKSG3WudKQHVgtCQu0cnFhXkYM GT52Wi6JW35Hm4DBi+eGVr+BsW4ynOLPNDooXUX/cLHFiDF6smiPa//ROR+wM8OeuddX O8aBmDa0LDRIklY1DirpBOcX/qantt2bopJdT/5Bbl/+GB5rOxwe+SzoliMgBKiB6tKA +G46ygRuhxDX+WYQTv65zejv24eazxIp3XDANP+Ls/kurG8QZ8WvMtCLCh7SjS+O7Kt2 KXQw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id v1si20354733ywf.2.2019.04.16.06.47.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:33 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlJw8091703 for ; Tue, 16 Apr 2019 09:47:32 -0400 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwfsus81v-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:23 -0400 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:01 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:51 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjneJ48103450 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:49 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5AF6A4C05A; Tue, 16 Apr 2019 13:45:49 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DE2934C059; Tue, 16 Apr 2019 13:45:47 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:47 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 09/31] mm: VMA sequence count Date: Tue, 16 Apr 2019 15:45:00 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0012-0000-0000-0000030F6EFB X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0013-0000-0000-00002147A858 Message-Id: <20190416134522.17540-10-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Peter Zijlstra Wrap the VMA modifications (vma_adjust/unmap_page_range) with sequence counts such that we can easily test if a VMA is changed. The calls to vm_write_begin/end() in unmap_page_range() are used to detect when a VMA is being unmap and thus that new page fault should not be satisfied for this VMA. If the seqcount hasn't changed when the page table are locked, this means we are safe to satisfy the page fault. The flip side is that we cannot distinguish between a vma_adjust() and the unmap_page_range() -- where with the former we could have re-checked the vma bounds against the address. The VMA's sequence counter is also used to detect change to various VMA's fields used during the page fault handling, such as: - vm_start, vm_end - vm_pgoff - vm_flags, vm_page_prot - anon_vma - vm_policy Signed-off-by: Peter Zijlstra (Intel) [Port to 4.12 kernel] [Build depends on CONFIG_SPECULATIVE_PAGE_FAULT] [Introduce vm_write_* inline function depending on CONFIG_SPECULATIVE_PAGE_FAULT] [Fix lock dependency between mapping->i_mmap_rwsem and vma->vm_sequence by using vm_raw_write* functions] [Fix a lock dependency warning in mmap_region() when entering the error path] [move sequence initialisation INIT_VMA()] [Review the patch description about unmap_page_range()] Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm.h | 44 ++++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 3 +++ mm/memory.c | 2 ++ mm/mmap.c | 30 +++++++++++++++++++++++++++ 4 files changed, 79 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2ceb1d2869a6..906b9e06f18e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1410,6 +1410,9 @@ struct zap_details { static inline void INIT_VMA(struct vm_area_struct *vma) { INIT_LIST_HEAD(&vma->anon_vma_chain); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + seqcount_init(&vma->vm_sequence); +#endif } struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, @@ -1534,6 +1537,47 @@ static inline void unmap_shared_mapping_range(struct address_space *mapping, unmap_mapping_range(mapping, holebegin, holelen, 0); } +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static inline void vm_write_begin(struct vm_area_struct *vma) +{ + write_seqcount_begin(&vma->vm_sequence); +} +static inline void vm_write_begin_nested(struct vm_area_struct *vma, + int subclass) +{ + write_seqcount_begin_nested(&vma->vm_sequence, subclass); +} +static inline void vm_write_end(struct vm_area_struct *vma) +{ + write_seqcount_end(&vma->vm_sequence); +} +static inline void vm_raw_write_begin(struct vm_area_struct *vma) +{ + raw_write_seqcount_begin(&vma->vm_sequence); +} +static inline void vm_raw_write_end(struct vm_area_struct *vma) +{ + raw_write_seqcount_end(&vma->vm_sequence); +} +#else +static inline void vm_write_begin(struct vm_area_struct *vma) +{ +} +static inline void vm_write_begin_nested(struct vm_area_struct *vma, + int subclass) +{ +} +static inline void vm_write_end(struct vm_area_struct *vma) +{ +} +static inline void vm_raw_write_begin(struct vm_area_struct *vma) +{ +} +static inline void vm_raw_write_end(struct vm_area_struct *vma) +{ +} +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + extern int access_process_vm(struct task_struct *tsk, unsigned long addr, void *buf, int len, unsigned int gup_flags); extern int access_remote_vm(struct mm_struct *mm, unsigned long addr, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fd7d38ee2e33..e78f72eb2576 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -337,6 +337,9 @@ struct vm_area_struct { struct mempolicy *vm_policy; /* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + seqcount_t vm_sequence; +#endif } __randomize_layout; struct core_thread { diff --git a/mm/memory.c b/mm/memory.c index d5bebca47d98..423fa8ea0569 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1256,6 +1256,7 @@ void unmap_page_range(struct mmu_gather *tlb, unsigned long next; BUG_ON(addr >= end); + vm_write_begin(vma); tlb_start_vma(tlb, vma); pgd = pgd_offset(vma->vm_mm, addr); do { @@ -1265,6 +1266,7 @@ void unmap_page_range(struct mmu_gather *tlb, next = zap_p4d_range(tlb, vma, pgd, addr, next, details); } while (pgd++, addr = next, addr != end); tlb_end_vma(tlb, vma); + vm_write_end(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index 5ad3a3228d76..a4e4d52a5148 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -726,6 +726,30 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, long adjust_next = 0; int remove_next = 0; + /* + * Why using vm_raw_write*() functions here to avoid lockdep's warning ? + * + * Locked is complaining about a theoretical lock dependency, involving + * 3 locks: + * mapping->i_mmap_rwsem --> vma->vm_sequence --> fs_reclaim + * + * Here are the major path leading to this dependency : + * 1. __vma_adjust() mmap_sem -> vm_sequence -> i_mmap_rwsem + * 2. move_vmap() mmap_sem -> vm_sequence -> fs_reclaim + * 3. __alloc_pages_nodemask() fs_reclaim -> i_mmap_rwsem + * 4. unmap_mapping_range() i_mmap_rwsem -> vm_sequence + * + * So there is no way to solve this easily, especially because in + * unmap_mapping_range() the i_mmap_rwsem is grab while the impacted + * VMAs are not yet known. + * However, the way the vm_seq is used is guarantying that we will + * never block on it since we just check for its value and never wait + * for it to move, see vma_has_changed() and handle_speculative_fault(). + */ + vm_raw_write_begin(vma); + if (next) + vm_raw_write_begin(next); + if (next && !insert) { struct vm_area_struct *exporter = NULL, *importer = NULL; @@ -950,6 +974,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * "vma->vm_next" gap must be updated. */ next = vma->vm_next; + if (next) + vm_raw_write_begin(next); } else { /* * For the scope of the comment "next" and @@ -996,6 +1022,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (insert && file) uprobe_mmap(insert); + if (next && next != vma) + vm_raw_write_end(next); + vm_raw_write_end(vma); + validate_mm(mm); return 0; From patchwork Tue Apr 16 13:45:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903189 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E1050161F for ; Tue, 16 Apr 2019 13:48:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C2D402843C for ; Tue, 16 Apr 2019 13:48:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B6037286CF; Tue, 16 Apr 2019 13:48:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9964E2843C for ; Tue, 16 Apr 2019 13:48:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C1B86B029B; Tue, 16 Apr 2019 09:48:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7A07A6B029E; Tue, 16 Apr 2019 09:48:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59F946B029F; Tue, 16 Apr 2019 09:48:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 034826B029B for ; Tue, 16 Apr 2019 09:48:14 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id h27so10684942eda.8 for ; Tue, 16 Apr 2019 06:48:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=Rzqn1HhdYspdyMrbZaPvWkCFg9mUFKqEPfg7Lcjtn5I=; b=glr4qmqFMQg/5a5tF2Wp8o/5qT/3Q6ZjKIvG6Qe35vlJw0z37CMoLluEVL0AXOeBSy IzadzR6AcY1CSYoFKsDMkMWonq8I9PmqjlldFRcDU4AdLNe9ej1ZYaulzoCzxpNzOAD4 UEh3aAAabvs1UF53kqumcGj7kNSdoFXpl5I4oZQmeDIej7uWmIH/KoUzciQhShLIJ4RX KrlPGnhKy9sw+NwYTDChaeabPsNwbR2dxyYRDbApukuXkZN1UNzltOKkfHtoD711MFzd Lr3fXfeaF1qQT+hKbVax/TwNtVBxePB4qFZmfwSMgN2parjIfknEw4aipe8kTx78Vgb+ 7o9w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVC1EzQ6LHln0g33lvdWlLCJHEsjlcbKleXdbmknuC1RVgODOGN RWoqMow8SVW6kqnhFDkvkpVQHDaNzc10nxNTtVJ6FLWOUQiZNAeq0fButZzD02JTk/rmBDxAUN4 Utti7pgjON/Zd2DCaIg3VOusnVPBD2RtrV+Ez5nZuzrKu/mYzSkCt5YvHjygzvc49kQ== X-Received: by 2002:a17:906:251b:: with SMTP id i27mr43830546ejb.146.1555422493423; Tue, 16 Apr 2019 06:48:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqwouZ8YCWI8SnBB1QlOsGUcX3XKuBOa0FeV8Kh6YyBELcUSO3LJb2nAlY+e6xeVCD43+AY6 X-Received: by 2002:a17:906:251b:: with SMTP id i27mr43830457ejb.146.1555422491241; Tue, 16 Apr 2019 06:48:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422491; cv=none; d=google.com; s=arc-20160816; b=fpYWnNZ11AuJ6cg0BdEqMQNLzV3EQVNEMZEbDahElJODE/o2nG68cZb70casAMKNSt UK3CJzan5NK7fgSYWLnLi8gOjXfp3ZJH7TK+Ta7/fZk23Z+P6GuS6oOjQkkUfLTj0sXO KqPxcy8W7/AmWm5z3xfRDlJAkGD4GYrzmVIvLs5g9KjKwwzUGwUsV4soeP2MToliwQRu PNT4UxUrZSWk3lV8xd91pL56vKu4uhDQMk2wADkLfWkXKDYVwuHGuWUbtcVyJrFyFsOp cHeRsW2Mmy6odWCPP8l6FRV5d8xYAgWVwrtAERkLgAQNLrJ0CQPruZmvlBjLKYCgHick PPAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=Rzqn1HhdYspdyMrbZaPvWkCFg9mUFKqEPfg7Lcjtn5I=; b=ycJPJPbrEWANZb4+cpWoVVH+XsFMVclt/REM72kOpVIoBw/8xkgTqk0yvvcnZduRFt g8ND8dL3ipT4kXk/o23IkCt0UgCn13ViWZVSyOIg74UOUSZO2ttOTGndgO6Fa+IJf82w QPUCoH62nuICjURrnpXH0mIuwZq61gTj59s/M1jjYOP4zte+T6ACWtedWB90j+2vlwsg L2PJH7MxIUXIiDYbgCm+wIO8gNgbbG9LeQRayfJVa8Q1WM/ZXH+ksbRMgf+TJm4sD0BW j2ytewjWg25pFWYTgz6isB2k45hFBxMULEYqZaY6zB6dWawA6hZkkIMjn4p1V2qknf0o ZpzA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id z8si3309052edd.352.2019.04.16.06.48.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:48:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlHfR010806 for ; Tue, 16 Apr 2019 09:48:09 -0400 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe56nxcm-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:43 -0400 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:01 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:52 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjpCb52232298 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:51 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0E2AF4C046; Tue, 16 Apr 2019 13:45:51 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 90FDA4C058; Tue, 16 Apr 2019 13:45:49 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:49 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 10/31] mm: protect VMA modifications using VMA sequence count Date: Tue, 16 Apr 2019 15:45:01 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0020-0000-0000-000003307050 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0021-0000-0000-00002182B501 Message-Id: <20190416134522.17540-11-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The VMA sequence count has been introduced to allow fast detection of VMA modification when running a page fault handler without holding the mmap_sem. This patch provides protection against the VMA modification done in : - madvise() - mpol_rebind_policy() - vma_replace_policy() - change_prot_numa() - mlock(), munlock() - mprotect() - mmap_region() - collapse_huge_page() - userfaultd registering services In addition, VMA fields which will be read during the speculative fault path needs to be written using WRITE_ONCE to prevent write to be split and intermediate values to be pushed to other CPUs. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- fs/proc/task_mmu.c | 5 ++++- fs/userfaultfd.c | 17 ++++++++++++---- mm/khugepaged.c | 3 +++ mm/madvise.c | 6 +++++- mm/mempolicy.c | 51 ++++++++++++++++++++++++++++++---------------- mm/mlock.c | 13 +++++++----- mm/mmap.c | 28 ++++++++++++++++--------- mm/mprotect.c | 4 +++- mm/swap_state.c | 10 ++++++--- 9 files changed, 95 insertions(+), 42 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 01d4eb0e6bd1..0864c050b2de 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1162,8 +1162,11 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, goto out_mm; } for (vma = mm->mmap; vma; vma = vma->vm_next) { - vma->vm_flags &= ~VM_SOFTDIRTY; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, + vma->vm_flags & ~VM_SOFTDIRTY); vma_set_page_prot(vma); + vm_write_end(vma); } downgrade_write(&mm->mmap_sem); break; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 3b30301c90ec..2e0f98cadd81 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -667,8 +667,11 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vm_write_begin(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~(VM_UFFD_WP | VM_UFFD_MISSING); + WRITE_ONCE(vma->vm_flags, + vma->vm_flags & ~(VM_UFFD_WP | VM_UFFD_MISSING)); + vm_write_end(vma); return 0; } @@ -908,8 +911,10 @@ static int userfaultfd_release(struct inode *inode, struct file *file) vma = prev; else prev = vma; - vma->vm_flags = new_flags; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; + vm_write_end(vma); } skip_mm: up_write(&mm->mmap_sem); @@ -1474,8 +1479,10 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ - vma->vm_flags = new_flags; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx; + vm_write_end(vma); skip: prev = vma; @@ -1636,8 +1643,10 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ - vma->vm_flags = new_flags; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; + vm_write_end(vma); skip: prev = vma; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a335f7c1fac4..6a0cbca3885e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,6 +1011,7 @@ static void collapse_huge_page(struct mm_struct *mm, if (mm_find_pmd(mm, address) != pmd) goto out; + vm_write_begin(vma); anon_vma_lock_write(vma->anon_vma); pte = pte_offset_map(pmd, address); @@ -1046,6 +1047,7 @@ static void collapse_huge_page(struct mm_struct *mm, pmd_populate(mm, pmd, pmd_pgtable(_pmd)); spin_unlock(pmd_ptl); anon_vma_unlock_write(vma->anon_vma); + vm_write_end(vma); result = SCAN_FAIL; goto out; } @@ -1081,6 +1083,7 @@ static void collapse_huge_page(struct mm_struct *mm, set_pmd_at(mm, address, pmd, _pmd); update_mmu_cache_pmd(vma, address, pmd); spin_unlock(pmd_ptl); + vm_write_end(vma); *hpage = NULL; diff --git a/mm/madvise.c b/mm/madvise.c index a692d2a893b5..6cf07dc546fc 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -184,7 +184,9 @@ static long madvise_behavior(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_sem held in write mode. */ - vma->vm_flags = new_flags; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, new_flags); + vm_write_end(vma); out: return error; } @@ -450,9 +452,11 @@ static void madvise_free_page_range(struct mmu_gather *tlb, .private = tlb, }; + vm_write_begin(vma); tlb_start_vma(tlb, vma); walk_page_range(addr, end, &free_walk); tlb_end_vma(tlb, vma); + vm_write_end(vma); } static int madvise_free_single_vma(struct vm_area_struct *vma, diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 2219e747df49..94c103c5034a 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -380,8 +380,11 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new) struct vm_area_struct *vma; down_write(&mm->mmap_sem); - for (vma = mm->mmap; vma; vma = vma->vm_next) + for (vma = mm->mmap; vma; vma = vma->vm_next) { + vm_write_begin(vma); mpol_rebind_policy(vma->vm_policy, new); + vm_write_end(vma); + } up_write(&mm->mmap_sem); } @@ -575,9 +578,11 @@ unsigned long change_prot_numa(struct vm_area_struct *vma, { int nr_updated; + vm_write_begin(vma); nr_updated = change_protection(vma, addr, end, PAGE_NONE, 0, 1); if (nr_updated) count_vm_numa_events(NUMA_PTE_UPDATES, nr_updated); + vm_write_end(vma); return nr_updated; } @@ -683,6 +688,7 @@ static int vma_replace_policy(struct vm_area_struct *vma, if (IS_ERR(new)) return PTR_ERR(new); + vm_write_begin(vma); if (vma->vm_ops && vma->vm_ops->set_policy) { err = vma->vm_ops->set_policy(vma, new); if (err) @@ -690,11 +696,17 @@ static int vma_replace_policy(struct vm_area_struct *vma, } old = vma->vm_policy; - vma->vm_policy = new; /* protected by mmap_sem */ + /* + * The speculative page fault handler accesses this field without + * hodling the mmap_sem. + */ + WRITE_ONCE(vma->vm_policy, new); + vm_write_end(vma); mpol_put(old); return 0; err_out: + vm_write_end(vma); mpol_put(new); return err; } @@ -1654,23 +1666,28 @@ COMPAT_SYSCALL_DEFINE4(migrate_pages, compat_pid_t, pid, struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, unsigned long addr) { - struct mempolicy *pol = NULL; + struct mempolicy *pol; - if (vma) { - if (vma->vm_ops && vma->vm_ops->get_policy) { - pol = vma->vm_ops->get_policy(vma, addr); - } else if (vma->vm_policy) { - pol = vma->vm_policy; + if (!vma) + return NULL; - /* - * shmem_alloc_page() passes MPOL_F_SHARED policy with - * a pseudo vma whose vma->vm_ops=NULL. Take a reference - * count on these policies which will be dropped by - * mpol_cond_put() later - */ - if (mpol_needs_cond_ref(pol)) - mpol_get(pol); - } + if (vma->vm_ops && vma->vm_ops->get_policy) + return vma->vm_ops->get_policy(vma, addr); + + /* + * This could be called without holding the mmap_sem in the + * speculative page fault handler's path. + */ + pol = READ_ONCE(vma->vm_policy); + if (pol) { + /* + * shmem_alloc_page() passes MPOL_F_SHARED policy with + * a pseudo vma whose vma->vm_ops=NULL. Take a reference + * count on these policies which will be dropped by + * mpol_cond_put() later + */ + if (mpol_needs_cond_ref(pol)) + mpol_get(pol); } return pol; diff --git a/mm/mlock.c b/mm/mlock.c index 080f3b36415b..f390903d9bbb 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -445,7 +445,9 @@ static unsigned long __munlock_pagevec_fill(struct pagevec *pvec, void munlock_vma_pages_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, vma->vm_flags & VM_LOCKED_CLEAR_MASK); + vm_write_end(vma); while (start < end) { struct page *page; @@ -569,10 +571,11 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, * It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ - - if (lock) - vma->vm_flags = newflags; - else + if (lock) { + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, newflags); + vm_write_end(vma); + } else munlock_vma_pages_range(vma, start, end); out: diff --git a/mm/mmap.c b/mm/mmap.c index a4e4d52a5148..b77ec0149249 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -877,17 +877,18 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, } if (start != vma->vm_start) { - vma->vm_start = start; + WRITE_ONCE(vma->vm_start, start); start_changed = true; } if (end != vma->vm_end) { - vma->vm_end = end; + WRITE_ONCE(vma->vm_end, end); end_changed = true; } - vma->vm_pgoff = pgoff; + WRITE_ONCE(vma->vm_pgoff, pgoff); if (adjust_next) { - next->vm_start += adjust_next << PAGE_SHIFT; - next->vm_pgoff += adjust_next; + WRITE_ONCE(next->vm_start, + next->vm_start + (adjust_next << PAGE_SHIFT)); + WRITE_ONCE(next->vm_pgoff, next->vm_pgoff + adjust_next); } if (root) { @@ -1850,12 +1851,14 @@ unsigned long mmap_region(struct file *file, unsigned long addr, out: perf_event_mmap(vma); + vm_write_begin(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); if (vm_flags & VM_LOCKED) { if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || is_vm_hugetlb_page(vma) || vma == get_gate_vma(current->mm)) - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + WRITE_ONCE(vma->vm_flags, + vma->vm_flags &= VM_LOCKED_CLEAR_MASK); else mm->locked_vm += (len >> PAGE_SHIFT); } @@ -1870,9 +1873,10 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * then new mapped in-place (which must be aimed as * a completely new data area). */ - vma->vm_flags |= VM_SOFTDIRTY; + WRITE_ONCE(vma->vm_flags, vma->vm_flags | VM_SOFTDIRTY); vma_set_page_prot(vma); + vm_write_end(vma); return addr; @@ -2430,7 +2434,9 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); anon_vma_interval_tree_pre_update_vma(vma); - vma->vm_end = address; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_end, address); + vm_write_end(vma); anon_vma_interval_tree_post_update_vma(vma); if (vma->vm_next) vma_gap_update(vma->vm_next); @@ -2510,8 +2516,10 @@ int expand_downwards(struct vm_area_struct *vma, mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); anon_vma_interval_tree_pre_update_vma(vma); - vma->vm_start = address; - vma->vm_pgoff -= grow; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_start, address); + WRITE_ONCE(vma->vm_pgoff, vma->vm_pgoff - grow); + vm_write_end(vma); anon_vma_interval_tree_post_update_vma(vma); vma_gap_update(vma); spin_unlock(&mm->page_table_lock); diff --git a/mm/mprotect.c b/mm/mprotect.c index 65242f1e4457..78fce873ca3a 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -427,12 +427,14 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev, * vm_flags and vm_page_prot are protected by the mmap_sem * held in write mode. */ - vma->vm_flags = newflags; + vm_write_begin(vma); + WRITE_ONCE(vma->vm_flags, newflags); dirty_accountable = vma_wants_writenotify(vma, vma->vm_page_prot); vma_set_page_prot(vma); change_protection(vma, start, end, vma->vm_page_prot, dirty_accountable, 0); + vm_write_end(vma); /* * Private VM_LOCKED VMA becoming writable: trigger COW to avoid major diff --git a/mm/swap_state.c b/mm/swap_state.c index eb714165afd2..c45f9122b457 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -523,7 +523,11 @@ static unsigned long swapin_nr_pages(unsigned long offset) * This has been extended to use the NUMA policies from the mm triggering * the readahead. * - * Caller must hold read mmap_sem if vmf->vma is not NULL. + * Caller must hold down_read on the vma->vm_mm if vmf->vma is not NULL. + * This is needed to ensure the VMA will not be freed in our back. In the case + * of the speculative page fault handler, this cannot happen, even if we don't + * hold the mmap_sem. Callees are assumed to take care of reading VMA's fields + * using READ_ONCE() to read consistent values. */ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, struct vm_fault *vmf) @@ -624,9 +628,9 @@ static inline void swap_ra_clamp_pfn(struct vm_area_struct *vma, unsigned long *start, unsigned long *end) { - *start = max3(lpfn, PFN_DOWN(vma->vm_start), + *start = max3(lpfn, PFN_DOWN(READ_ONCE(vma->vm_start)), PFN_DOWN(faddr & PMD_MASK)); - *end = min3(rpfn, PFN_DOWN(vma->vm_end), + *end = min3(rpfn, PFN_DOWN(READ_ONCE(vma->vm_end)), PFN_DOWN((faddr & PMD_MASK) + PMD_SIZE)); } From patchwork Tue Apr 16 13:45:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903177 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A497161F for ; Tue, 16 Apr 2019 13:48:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B1372843C for ; Tue, 16 Apr 2019 13:48:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DB63286CF; Tue, 16 Apr 2019 13:48:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2602B2843C for ; Tue, 16 Apr 2019 13:48:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 307C26B0290; Tue, 16 Apr 2019 09:47:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E1BA6B0292; Tue, 16 Apr 2019 09:47:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CEBD6B0293; Tue, 16 Apr 2019 09:47:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id BEEFB6B0290 for ; Tue, 16 Apr 2019 09:47:39 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id d2so10672540edo.23 for ; Tue, 16 Apr 2019 06:47:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=coDoVYn8E+wKnWAWcNhAWW2x0vFovvAISraEBhD2A3Q=; b=o11QesUGS4zI8wzpY18zbm8W4FcBNJ+D8h3qYUbzJy3JOYWRocqOERBQOTciLRbjGW 29+shFAjP8RLdK1yMbtMbsvYpPFn47OWXmPbu7t/xI3Tdxf/VP5uM2gGjAhU1v+FvmI2 d7gSRfSFVwruH4TlBAxOEM2WCuboxGQxKsqNd3HikwICBkdg5soTJIMwd80ttggmJ3YM V4ai79PQ6406ERm5WLLGHEU+kDFuGlW1ekPmgRfAIz35Qp/oz5CGQ8htnYlnrD6AruQZ ssCk/1eDM2vKAjiCWECLoeLIwvXm782wtIA4qMPvyFwcCdLlVJ7xSI6dOzSrxU2HSfTJ jL5A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAUlmYtVZ3aFp4Wjow1JlBbtffibmpIZcjjy2S9BwqfFfI1oVkd3 hqk05bpNQix7BcfbxL8GDV1ClUjNcZzrJtS3DbLF66alxkH7pQfxZmUMdpFwmagdslMzBtW86OP u8wyANYUC/7jMmVb9JGtD0Z+C7o/BI8DwtBcYrI0Dt9VTiOqt2OSLP0tOrpZdwOXs7w== X-Received: by 2002:aa7:d3ca:: with SMTP id o10mr9146948edr.160.1555422459159; Tue, 16 Apr 2019 06:47:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqyp/4RZzdmjJ1x6dc5mA8jtpZB5Vtw5ea3hprCvupNOJTBAVMsGSJkcTqMFMUO86XKpUEqc X-Received: by 2002:aa7:d3ca:: with SMTP id o10mr9146880edr.160.1555422457904; Tue, 16 Apr 2019 06:47:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422457; cv=none; d=google.com; s=arc-20160816; b=nhM6OdrBhp7DkiigOkQXVzAJAzEM2m++CMrYyFP3OHO8AYOyp/btLm+8ltlHbM/oV2 ZWWcYAmWdiUWMu2qcAovaIipfOXB5XRmNn6Mrcm3+BNTI1/Y5WmXuyvP6cgskbOCF1Uu vTiqnTgFjlv3UHHJ3uVz4/9I/UUhIRf91qo4C1ZkpPLmQSKCldnbbXI8IVAGOsiTKu1e /8Mk29+oko43D2Iln/Xk9cmo74uhQoYKzIcfrkNFAST/lycCTOEDml8l5nl+BsitLdum Tm5nxaumhZVdALf9pyYIcs+hrGWLqVPDA4Nm/6stdzkTE3SZzpX85IWwdv9hvQL4+qHz J99Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=coDoVYn8E+wKnWAWcNhAWW2x0vFovvAISraEBhD2A3Q=; b=sF+LlVNaga0SCJ/GGTeoCeeYKfiyjPL0udTT8yu7CES5Uw0j/UZytSsjR32RdZrYMA JlXg6Ju9tvheKBpu+ZHPzfZiMQ0pzcNOjeSLi4SwI633Dl7EuCEn5rzx0cyZ1Qy4rcMD szl5j/91YFGUQ0awx1QAIpoQyYzK5t+RqX8OAhinylZq3mi4fE1uRFTDVC9QCLTbrRVD PR+NdLAKqbjD1c4dIx5e4ws1vuorgwebl9+7MQKBqpqovyzOvFCw1II07E5z76TYLlec 5VcyDveHpdblg9m9LCpk7lrw4h9E+QahckVrp6cNmE9yhLbEI5ZlnysmYgPfGMbtqCGY uQHA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id b17si5592031edd.415.2019.04.16.06.47.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:37 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlDLZ060551 for ; Tue, 16 Apr 2019 09:47:36 -0400 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwe3k5utk-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:25 -0400 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:05 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:56 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjsYg35651690 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:54 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0E6794C046; Tue, 16 Apr 2019 13:45:54 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2BBEB4C05A; Tue, 16 Apr 2019 13:45:52 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:52 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 11/31] mm: protect mremap() against SPF hanlder Date: Tue, 16 Apr 2019 15:45:02 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0012-0000-0000-0000030F6EFE X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0013-0000-0000-00002147A85A Message-Id: <20190416134522.17540-12-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP If a thread is remapping an area while another one is faulting on the destination area, the SPF handler may fetch the vma from the RB tree before the pte has been moved by the other thread. This means that the moved ptes will overwrite those create by the page fault handler leading to page leaked. CPU 1 CPU2 enter mremap() unmap the dest area copy_vma() Enter speculative page fault handler >> at this time the dest area is present in the RB tree fetch the vma matching dest area create a pte as the VMA matched Exit the SPF handler move_ptes() > it is assumed that the dest area is empty, > the move ptes overwrite the page mapped by the CPU2. To prevent that, when the VMA matching the dest area is extended or created by copy_vma(), it should be marked as non available to the SPF handler. The usual way to so is to rely on vm_write_begin()/end(). This is already in __vma_adjust() called by copy_vma() (through vma_merge()). But __vma_adjust() is calling vm_write_end() before returning which create a window for another thread. This patch adds a new parameter to vma_merge() which is passed down to vma_adjust(). The assumption is that copy_vma() is returning a vma which should be released by calling vm_raw_write_end() by the callee once the ptes have been moved. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm.h | 24 ++++++++++++++++----- mm/mmap.c | 53 +++++++++++++++++++++++++++++++++++----------- mm/mremap.c | 13 ++++++++++++ 3 files changed, 73 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 906b9e06f18e..5d45b7d8718d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2343,18 +2343,32 @@ void anon_vma_interval_tree_verify(struct anon_vma_chain *node); /* mmap.c */ extern int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin); + extern int __vma_adjust(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgoff_t pgoff, struct vm_area_struct *insert, - struct vm_area_struct *expand); + struct vm_area_struct *expand, bool keep_locked); + static inline int vma_adjust(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgoff_t pgoff, struct vm_area_struct *insert) { - return __vma_adjust(vma, start, end, pgoff, insert, NULL); + return __vma_adjust(vma, start, end, pgoff, insert, NULL, false); } -extern struct vm_area_struct *vma_merge(struct mm_struct *, + +extern struct vm_area_struct *__vma_merge(struct mm_struct *mm, + struct vm_area_struct *prev, unsigned long addr, unsigned long end, + unsigned long vm_flags, struct anon_vma *anon, struct file *file, + pgoff_t pgoff, struct mempolicy *mpol, + struct vm_userfaultfd_ctx uff, bool keep_locked); + +static inline struct vm_area_struct *vma_merge(struct mm_struct *mm, struct vm_area_struct *prev, unsigned long addr, unsigned long end, - unsigned long vm_flags, struct anon_vma *, struct file *, pgoff_t, - struct mempolicy *, struct vm_userfaultfd_ctx); + unsigned long vm_flags, struct anon_vma *anon, struct file *file, + pgoff_t off, struct mempolicy *pol, struct vm_userfaultfd_ctx uff) +{ + return __vma_merge(mm, prev, addr, end, vm_flags, anon, file, off, + pol, uff, false); +} + extern struct anon_vma *find_mergeable_anon_vma(struct vm_area_struct *); extern int __split_vma(struct mm_struct *, struct vm_area_struct *, unsigned long addr, int new_below); diff --git a/mm/mmap.c b/mm/mmap.c index b77ec0149249..13460b38b0fb 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -714,7 +714,7 @@ static inline void __vma_unlink_prev(struct mm_struct *mm, */ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgoff_t pgoff, struct vm_area_struct *insert, - struct vm_area_struct *expand) + struct vm_area_struct *expand, bool keep_locked) { struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *next = vma->vm_next, *orig_vma = vma; @@ -830,8 +830,12 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, importer->anon_vma = exporter->anon_vma; error = anon_vma_clone(importer, exporter); - if (error) + if (error) { + if (next && next != vma) + vm_raw_write_end(next); + vm_raw_write_end(vma); return error; + } } } again: @@ -1025,7 +1029,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (next && next != vma) vm_raw_write_end(next); - vm_raw_write_end(vma); + if (!keep_locked) + vm_raw_write_end(vma); validate_mm(mm); @@ -1161,12 +1166,13 @@ can_vma_merge_after(struct vm_area_struct *vma, unsigned long vm_flags, * parameter) may establish ptes with the wrong permissions of NNNN * instead of the right permissions of XXXX. */ -struct vm_area_struct *vma_merge(struct mm_struct *mm, +struct vm_area_struct *__vma_merge(struct mm_struct *mm, struct vm_area_struct *prev, unsigned long addr, unsigned long end, unsigned long vm_flags, struct anon_vma *anon_vma, struct file *file, pgoff_t pgoff, struct mempolicy *policy, - struct vm_userfaultfd_ctx vm_userfaultfd_ctx) + struct vm_userfaultfd_ctx vm_userfaultfd_ctx, + bool keep_locked) { pgoff_t pglen = (end - addr) >> PAGE_SHIFT; struct vm_area_struct *area, *next; @@ -1214,10 +1220,11 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, /* cases 1, 6 */ err = __vma_adjust(prev, prev->vm_start, next->vm_end, prev->vm_pgoff, NULL, - prev); + prev, keep_locked); } else /* cases 2, 5, 7 */ err = __vma_adjust(prev, prev->vm_start, - end, prev->vm_pgoff, NULL, prev); + end, prev->vm_pgoff, NULL, prev, + keep_locked); if (err) return NULL; khugepaged_enter_vma_merge(prev, vm_flags); @@ -1234,10 +1241,12 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, vm_userfaultfd_ctx)) { if (prev && addr < prev->vm_end) /* case 4 */ err = __vma_adjust(prev, prev->vm_start, - addr, prev->vm_pgoff, NULL, next); + addr, prev->vm_pgoff, NULL, next, + keep_locked); else { /* cases 3, 8 */ err = __vma_adjust(area, addr, next->vm_end, - next->vm_pgoff - pglen, NULL, next); + next->vm_pgoff - pglen, NULL, next, + keep_locked); /* * In case 3 area is already equal to next and * this is a noop, but in case 8 "area" has @@ -3259,9 +3268,20 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, if (find_vma_links(mm, addr, addr + len, &prev, &rb_link, &rb_parent)) return NULL; /* should never get here */ - new_vma = vma_merge(mm, prev, addr, addr + len, vma->vm_flags, - vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma), - vma->vm_userfaultfd_ctx); + + /* There is 3 cases to manage here in + * AAAA AAAA AAAA AAAA + * PPPP.... PPPP......NNNN PPPP....NNNN PP........NN + * PPPPPPPP(A) PPPP..NNNNNNNN(B) PPPPPPPPPPPP(1) NULL + * PPPPPPPPNNNN(2) + * PPPPNNNNNNNN(3) + * + * new_vma == prev in case A,1,2 + * new_vma == next in case B,3 + */ + new_vma = __vma_merge(mm, prev, addr, addr + len, vma->vm_flags, + vma->anon_vma, vma->vm_file, pgoff, + vma_policy(vma), vma->vm_userfaultfd_ctx, true); if (new_vma) { /* * Source vma may have been merged into new_vma @@ -3299,6 +3319,15 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + /* + * As the VMA is linked right now, it may be hit by the + * speculative page fault handler. But we don't want it to + * to start mapping page in this area until the caller has + * potentially move the pte from the moved VMA. To prevent + * that we protect it right now, and let the caller unprotect + * it once the move is done. + */ + vm_raw_write_begin(new_vma); vma_link(mm, new_vma, prev, rb_link, rb_parent); *need_rmap_locks = false; } diff --git a/mm/mremap.c b/mm/mremap.c index fc241d23cd97..ae5c3379586e 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -357,6 +357,14 @@ static unsigned long move_vma(struct vm_area_struct *vma, if (!new_vma) return -ENOMEM; + /* new_vma is returned protected by copy_vma, to prevent speculative + * page fault to be done in the destination area before we move the pte. + * Now, we must also protect the source VMA since we don't want pages + * to be mapped in our back while we are copying the PTEs. + */ + if (vma != new_vma) + vm_raw_write_begin(vma); + moved_len = move_page_tables(vma, old_addr, new_vma, new_addr, old_len, need_rmap_locks); if (moved_len < old_len) { @@ -373,6 +381,8 @@ static unsigned long move_vma(struct vm_area_struct *vma, */ move_page_tables(new_vma, new_addr, vma, old_addr, moved_len, true); + if (vma != new_vma) + vm_raw_write_end(vma); vma = new_vma; old_len = new_len; old_addr = new_addr; @@ -381,7 +391,10 @@ static unsigned long move_vma(struct vm_area_struct *vma, mremap_userfaultfd_prep(new_vma, uf); arch_remap(mm, old_addr, old_addr + old_len, new_addr, new_addr + new_len); + if (vma != new_vma) + vm_raw_write_end(vma); } + vm_raw_write_end(new_vma); /* Conceal VM_ACCOUNT so old reservation is not undone */ if (vm_flags & VM_ACCOUNT) { From patchwork Tue Apr 16 13:45:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903135 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 59275161F for ; Tue, 16 Apr 2019 13:46:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38FCE286CF for ; Tue, 16 Apr 2019 13:46:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2CD8E289F5; Tue, 16 Apr 2019 13:46:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAAA1286CF for ; Tue, 16 Apr 2019 13:46:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FECC6B000D; Tue, 16 Apr 2019 09:46:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 85EE26B000E; Tue, 16 Apr 2019 09:46:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 700FA6B0010; Tue, 16 Apr 2019 09:46:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 221E46B000D for ; Tue, 16 Apr 2019 09:46:52 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id f7so11004963edi.3 for ; Tue, 16 Apr 2019 06:46:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=9ekLajwcLy0Sow96Zz9b/mB4kzMw9l9rEQcwQPWbNlo=; b=PK5lddnZyzbkvtSBVLui4tikJ4Uej8nq8TCAFpJhzzJSGY1sOBfwBaMneoEmB4gHDv 6VZxBRtbmCLtZCuLTWcMTSeuuhCMo4KjXiHgrsdI4JaAcH3XGYGfY8qWbpXFW8nBL/Yk YDCT5bPJAGIu3oMAsQWvoj0D169tNi2qLmsRkOQEMQ61f9S9UJWvyu20ej/javATzP24 1asI0lgGwC9KT5fmlzv2jMlJ7DIaCKi3gxFDgE1Yg2X8lAAgEnwURLQtZR5hG7c7srNt ILv/ECz0s9NWlQ4Fmxlm64gsMxdaYvgHXjWNi/ikE8RDN2dEE3Ld14we9edVXKERDKe/ m6hw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAWT5Hy1SLuQOzk/Uz6cMH3ETJSAUqM5xnkTQMWLeJF0XUipSzNQ 9dl7Y1NNq3UOhT2pJq73Wt5NdLMxfb+R6IBj+6zoGvzxON+FePmF7EpyOh86NSOHUooVLPCMLyg l/Ugwbgf+YC+4t26w63v1kR4+IAVPrZbFIkl4amKlue7Wjx6o/mmzcBb4Y0rj5L7hjw== X-Received: by 2002:a50:a4e4:: with SMTP id x33mr51420589edb.61.1555422411632; Tue, 16 Apr 2019 06:46:51 -0700 (PDT) X-Google-Smtp-Source: APXvYqxIRFjejtvJPDXMg4xFT+409iDZOwHtn60SE/kg29Ok5Pg5gWt4KpZxTVEPT1NJft7acIdz X-Received: by 2002:a50:a4e4:: with SMTP id x33mr51420288edb.61.1555422406863; Tue, 16 Apr 2019 06:46:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422406; cv=none; d=google.com; s=arc-20160816; b=MMQIy43jhkul99LPmEX6Gwn7z6H2+KeCoD0PACKe/V4Bw8SINPTkT/anyP8RzegQTD 3ZtILySEQsHS6e7acYkaht9KrKwTaNu/aEQ3FTd2c+bwQ+3BsiHwloyaXhqUY8/Rvi7K 7WPmdMxNET8juygMlrQhZWpv1cdC4j/PVFxAiO1EttLR0Hxns6xjBlMJB7Hfb2TnEi1a oxVosKWMEtya6uNfaO6lNsmYYyw8HJxXKQUCF7TQXOR+NQEpYP/Shuy6cvH+OBvOgPlx qJltJDwyAmR09soqWQKT8sLFBJot9C2xF4Duox9lckPuKbp67s9kmOC/3r4GWU4EO7N6 P5rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=9ekLajwcLy0Sow96Zz9b/mB4kzMw9l9rEQcwQPWbNlo=; b=kRFuJZQ6t2zQh3gmSjjPwy9PuFMa1V49RYETSDiEEh5x3e+AyLVf9lXaUMJMY9QMrI sTgWeQooEqKaqG8hygOWwHkxGsV2lJhbtTWQ5uWwVy/al0MXjpSvzzRGM01hYVSRVnbR v1rRAsQraPVyizFBM08hsGcI0lf3uyhduyDP52BXO+aTroWS9nq7mJkNOX246tjkqIwu ZuWi6Xmg0KCOx8XlNVwuHW5YATRqIq8bDP/gedsdYtnBtDcHV0lQaVsi/jyJTZ8LWMUb fvRVtEzAbj3HdbVPHRdS02t2hmpt//NgYatgAVdAGe+Sp4Uc7fv+Z/9cETLSNAy109VX c80g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id n10si8759006ejh.79.2019.04.16.06.46.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:46 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkW02070715 for ; Tue, 16 Apr 2019 09:46:45 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwftpha5x-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:41 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:09 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:45:59 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDjvHK45482116 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:45:58 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D07D04C058; Tue, 16 Apr 2019 13:45:57 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 27A384C05A; Tue, 16 Apr 2019 13:45:55 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:55 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 12/31] mm: protect SPF handler against anon_vma changes Date: Tue, 16 Apr 2019 15:45:03 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7271 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD87 Message-Id: <20190416134522.17540-13-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The speculative page fault handler must be protected against anon_vma changes. This is because page_add_new_anon_rmap() is called during the speculative path. In addition, don't try speculative page fault if the VMA don't have an anon_vma structure allocated because its allocation should be protected by the mmap_sem. In __vma_adjust() when importer->anon_vma is set, there is no need to protect against speculative page faults since speculative page fault is aborted if the vma->anon_vma is not set. When calling page_add_new_anon_rmap() vma->anon_vma is necessarily valid since we checked for it when locking the pte and the anon_vma is removed once the pte is unlocked. So even if the speculative page fault handler is running concurrently with do_unmap(), as the pte is locked in unmap_region() - through unmap_vmas() - and the anon_vma unlinked later, because we check for the vma sequence counter which is updated in unmap_page_range() before locking the pte, and then in free_pgtables() so when locking the pte the change will be detected. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 423fa8ea0569..2cf7b6185daa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -377,7 +377,9 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + vm_write_begin(vma); unlink_anon_vmas(vma); + vm_write_end(vma); unlink_file_vma(vma); if (is_vm_hugetlb_page(vma)) { @@ -391,7 +393,9 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, && !is_vm_hugetlb_page(next)) { vma = next; next = vma->vm_next; + vm_write_begin(vma); unlink_anon_vmas(vma); + vm_write_end(vma); unlink_file_vma(vma); } free_pgd_range(tlb, addr, vma->vm_end, From patchwork Tue Apr 16 13:45:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903153 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DACE17E1 for ; Tue, 16 Apr 2019 13:47:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B2C62843C for ; Tue, 16 Apr 2019 13:47:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5EA1A289F9; Tue, 16 Apr 2019 13:47:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19B812843C for ; Tue, 16 Apr 2019 13:47:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 618DB6B026E; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 57B666B0270; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F3506B0271; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id 13E206B026E for ; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id g186so15672956ybg.16 for ; Tue, 16 Apr 2019 06:47:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=PTSOch4L+fXVWlK6LWprFn0fIBj8utDcUEXogBml36Q=; b=swIeYwEue5FQleR3Zz+o6BjdXosRhmTl0GnbayrxO/7YVDR+OFJlAClJPg8jXoIiOk 2BkkHzrqvQuQC/ahkX65aSqOnrVI9ZJVteeASPf+AZr20iVTYyYgHp9aYXLNmm7jV0qq 8hTOgaTu3tFwacuBzLmEG000kToDDohfXhyFQp1lJB5WM+YQaf3IUhM3dHFI4N6Nc8i2 zsFUPBGeexdej4RzkaEwN5s4QUiMC6gB20c+8bS5OnRTxtNOEx8GMpyIMU/qx1PzF5dn yhPNFI8HKf8Z25K3z7R4/1G9PYTeVAUwjuJxbKyzwsDAY7wXJi0E3cTfvmusdTC5Yj7c LSCQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAV5/RPUyNSSFWeD1bQILUYEjP0VkK5MTuCY7pqDKBRq85tFe2mv 07Mrvck8f6j3OOyMwkbdWfpaU7wJFXIXHNQaWaLeYHMejjPDdadTANnJgfNL4Ur/TiBhv7zDXrF tpFIara4GqpQa/ulDy2MNwPyTfuYKYnapUfFAxmFuwHN9NethqCVKRWdYeVrqqOn9NQ== X-Received: by 2002:a81:a350:: with SMTP id a77mr64692112ywh.66.1555422422748; Tue, 16 Apr 2019 06:47:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqxon0nS2Wxsnju92mYca9QbMpnPzwgJMyKy5pNEXReCJws0/3zVIJvata003udAI0qRKmsn X-Received: by 2002:a81:a350:: with SMTP id a77mr64691925ywh.66.1555422420794; Tue, 16 Apr 2019 06:47:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422420; cv=none; d=google.com; s=arc-20160816; b=UoUtfMnNUYQOpI23h8NaZyo4mlObLT8IwKqQyFjiq/2yV7rlzkCSP4e3RxIjQafz4p +FT1imIJPl5lSkTvxc5V0ZTHSxrpzQcd205Bluui9gk/kUoXS6xun1ej606/dYv1jBCK ADK3+HoJUTVAFOpPaqcgm1sbBsscoAtXft9AQ6sK+hKUO+FOfxrkrfO63z7DhYCan6Ir 9mNSszFEdw9/C/cTZIWxGiNZ56LFhvTkOkau6xxVR+KdHfjErTzzJGn30M4kH0sRqvyC vHLPnCU7R7oyA3qMKpNt0gygUCFcvwtmTyYfiA/2FROnuq0+2xYJG92571NyvZjhEiwY r8VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=PTSOch4L+fXVWlK6LWprFn0fIBj8utDcUEXogBml36Q=; b=m7EZLEqUyvc8AfuGNilIMCjVETAENDYcvJAtqD6TAG3SUlLHnC/JP3EzTsVjQq2s4E JlJ/75jWWh0UcIL+ZUbSR6h0GzWgwVLY8DiJCH8gO95Ps/qBeCQ6i3dRsNxJIsNALp0w 7Xw9jbgRoqHIHbUoSEDOj8tewstXBz+iQtshUrri8Lst1KoFgpgnYrzlps8pWo+mT7JN vsqATUlv+xMeLcNyyUrhfKiObhRFW6FfXy8J762E4ZZkE4UF5f7TyM3tvlKezXWYdsr6 U5AuWDDG6dezN7kZPAQsHu72IbzWhuAOPDQ5hOU9PiWORif8zlljg+hAePJqb3l8lK1s 59GA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id 14si31170852yww.219.2019.04.16.06.47.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:00 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkpMe059581 for ; Tue, 16 Apr 2019 09:46:59 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe36ny53-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:56 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:12 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:02 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDk0sc55836732 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:00 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 76C1C4C040; Tue, 16 Apr 2019 13:46:00 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 003A94C059; Tue, 16 Apr 2019 13:45:59 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:45:58 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 13/31] mm: cache some VMA fields in the vm_fault structure Date: Tue, 16 Apr 2019 15:45:04 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0008-0000-0000-000002DA6FB2 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0009-0000-0000-00002246A83C Message-Id: <20190416134522.17540-14-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When handling speculative page fault, the vma->vm_flags and vma->vm_page_prot fields are read once the page table lock is released. So there is no more guarantee that these fields would not change in our back. They will be saved in the vm_fault structure before the VMA is checked for changes. In the detail, when we deal with a speculative page fault, the mmap_sem is not taken, so parallel VMA's changes can occurred. When a VMA change is done which will impact the page fault processing, we assumed that the VMA sequence counter will be changed. In the page fault processing, at the time the PTE is locked, we checked the VMA sequence counter to detect changes done in our back. If no change is detected we can continue further. But this doesn't prevent the VMA to not be changed in our back while the PTE is locked. So VMA's fields which are used while the PTE is locked must be saved to ensure that we are using *static* values. This is important since the PTE changes will be made on regards to these VMA fields and they need to be consistent. This concerns the vma->vm_flags and vma->vm_page_prot VMA fields. This patch also set the fields in hugetlb_no_page() and __collapse_huge_page_swapin even if it is not need for the callee. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm.h | 10 +++++++-- mm/huge_memory.c | 6 +++--- mm/hugetlb.c | 2 ++ mm/khugepaged.c | 2 ++ mm/memory.c | 53 ++++++++++++++++++++++++---------------------- mm/migrate.c | 2 +- 6 files changed, 44 insertions(+), 31 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5d45b7d8718d..f465bb2b049e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -439,6 +439,12 @@ struct vm_fault { * page table to avoid allocation from * atomic context. */ + /* + * These entries are required when handling speculative page fault. + * This way the page handling is done using consistent field values. + */ + unsigned long vma_flags; + pgprot_t vma_page_prot; }; /* page entry size for vm->huge_fault() */ @@ -781,9 +787,9 @@ void free_compound_page(struct page *page); * pte_mkwrite. But get_user_pages can cause write faults for mappings * that do not have writing enabled, when used by access_process_vm. */ -static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) +static inline pte_t maybe_mkwrite(pte_t pte, unsigned long vma_flags) { - if (likely(vma->vm_flags & VM_WRITE)) + if (likely(vma_flags & VM_WRITE)) pte = pte_mkwrite(pte); return pte; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 823688414d27..865886a689ee 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1244,8 +1244,8 @@ static vm_fault_t do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { pte_t entry; - entry = mk_pte(pages[i], vma->vm_page_prot); - entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = mk_pte(pages[i], vmf->vma_page_prot); + entry = maybe_mkwrite(pte_mkdirty(entry), vmf->vma_flags); memcg = (void *)page_private(pages[i]); set_page_private(pages[i], 0); page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false); @@ -2228,7 +2228,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_swp_mksoft_dirty(entry); } else { entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); - entry = maybe_mkwrite(entry, vma); + entry = maybe_mkwrite(entry, vma->vm_flags); if (!write) entry = pte_wrprotect(entry); if (!young) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 109f5de82910..13246da4bc50 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3812,6 +3812,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, .vma = vma, .address = haddr, .flags = flags, + .vma_flags = vma->vm_flags, + .vma_page_prot = vma->vm_page_prot, /* * Hard to debug if it ends up being * used by a callee that assumes diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6a0cbca3885e..42469037240a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -888,6 +888,8 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, .flags = FAULT_FLAG_ALLOW_RETRY, .pmd = pmd, .pgoff = linear_page_index(vma, address), + .vma_flags = vma->vm_flags, + .vma_page_prot = vma->vm_page_prot, }; /* we only decide to swapin, if there is enough young ptes */ diff --git a/mm/memory.c b/mm/memory.c index 2cf7b6185daa..d0de58464479 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1560,7 +1560,8 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, goto out_unlock; } entry = pte_mkyoung(*pte); - entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = maybe_mkwrite(pte_mkdirty(entry), + vma->vm_flags); if (ptep_set_access_flags(vma, addr, pte, entry, 1)) update_mmu_cache(vma, addr, pte); } @@ -1575,7 +1576,7 @@ static vm_fault_t insert_pfn(struct vm_area_struct *vma, unsigned long addr, if (mkwrite) { entry = pte_mkyoung(entry); - entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = maybe_mkwrite(pte_mkdirty(entry), vma->vm_flags); } set_pte_at(mm, addr, pte, entry); @@ -2257,7 +2258,7 @@ static inline void wp_page_reuse(struct vm_fault *vmf) flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); entry = pte_mkyoung(vmf->orig_pte); - entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = maybe_mkwrite(pte_mkdirty(entry), vmf->vma_flags); if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1)) update_mmu_cache(vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -2335,8 +2336,8 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) inc_mm_counter_fast(mm, MM_ANONPAGES); } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); - entry = mk_pte(new_page, vma->vm_page_prot); - entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = mk_pte(new_page, vmf->vma_page_prot); + entry = maybe_mkwrite(pte_mkdirty(entry), vmf->vma_flags); /* * Clear the pte entry and flush it first, before updating the * pte with the new entry. This will avoid a race condition @@ -2401,7 +2402,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * Don't let another task, with possibly unlocked vma, * keep the mlocked page. */ - if (page_copied && (vma->vm_flags & VM_LOCKED)) { + if (page_copied && (vmf->vma_flags & VM_LOCKED)) { lock_page(old_page); /* LRU manipulation */ if (PageMlocked(old_page)) munlock_vma_page(old_page); @@ -2438,7 +2439,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) */ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) { - WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED)); + WARN_ON_ONCE(!(vmf->vma_flags & VM_SHARED)); if (!pte_map_lock(vmf)) return VM_FAULT_RETRY; /* @@ -2540,7 +2541,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) * We should not cow pages in a shared writeable mapping. * Just mark the pages writable and/or call ops->pfn_mkwrite. */ - if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) == + if ((vmf->vma_flags & (VM_WRITE|VM_SHARED)) == (VM_WRITE|VM_SHARED)) return wp_pfn_shared(vmf); @@ -2599,7 +2600,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) return VM_FAULT_WRITE; } unlock_page(vmf->page); - } else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) == + } else if (unlikely((vmf->vma_flags & (VM_WRITE|VM_SHARED)) == (VM_WRITE|VM_SHARED))) { return wp_page_shared(vmf); } @@ -2878,9 +2879,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); - pte = mk_pte(page, vma->vm_page_prot); + pte = mk_pte(page, vmf->vma_page_prot); if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) { - pte = maybe_mkwrite(pte_mkdirty(pte), vma); + pte = maybe_mkwrite(pte_mkdirty(pte), vmf->vma_flags); vmf->flags &= ~FAULT_FLAG_WRITE; ret |= VM_FAULT_WRITE; exclusive = RMAP_EXCLUSIVE; @@ -2905,7 +2906,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swap_free(entry); if (mem_cgroup_swap_full(page) || - (vma->vm_flags & VM_LOCKED) || PageMlocked(page)) + (vmf->vma_flags & VM_LOCKED) || PageMlocked(page)) try_to_free_swap(page); unlock_page(page); if (page != swapcache && swapcache) { @@ -2963,7 +2964,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_t entry; /* File mapping without ->vm_ops ? */ - if (vma->vm_flags & VM_SHARED) + if (vmf->vma_flags & VM_SHARED) return VM_FAULT_SIGBUS; /* @@ -2987,7 +2988,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm)) { entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address), - vma->vm_page_prot)); + vmf->vma_page_prot)); if (!pte_map_lock(vmf)) return VM_FAULT_RETRY; if (!pte_none(*vmf->pte)) @@ -3021,8 +3022,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) */ __SetPageUptodate(page); - entry = mk_pte(page, vma->vm_page_prot); - if (vma->vm_flags & VM_WRITE) + entry = mk_pte(page, vmf->vma_page_prot); + if (vmf->vma_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); if (!pte_map_lock(vmf)) { @@ -3242,7 +3243,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) for (i = 0; i < HPAGE_PMD_NR; i++) flush_icache_page(vma, page + i); - entry = mk_huge_pmd(page, vma->vm_page_prot); + entry = mk_huge_pmd(page, vmf->vma_page_prot); if (write) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); @@ -3318,11 +3319,11 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, return VM_FAULT_NOPAGE; flush_icache_page(vma, page); - entry = mk_pte(page, vma->vm_page_prot); + entry = mk_pte(page, vmf->vma_page_prot); if (write) - entry = maybe_mkwrite(pte_mkdirty(entry), vma); + entry = maybe_mkwrite(pte_mkdirty(entry), vmf->vma_flags); /* copy-on-write page */ - if (write && !(vma->vm_flags & VM_SHARED)) { + if (write && !(vmf->vma_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); @@ -3362,7 +3363,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf) /* Did we COW the page? */ if ((vmf->flags & FAULT_FLAG_WRITE) && - !(vmf->vma->vm_flags & VM_SHARED)) + !(vmf->vma_flags & VM_SHARED)) page = vmf->cow_page; else page = vmf->page; @@ -3641,7 +3642,7 @@ static vm_fault_t do_fault(struct vm_fault *vmf) } } else if (!(vmf->flags & FAULT_FLAG_WRITE)) ret = do_read_fault(vmf); - else if (!(vma->vm_flags & VM_SHARED)) + else if (!(vmf->vma_flags & VM_SHARED)) ret = do_cow_fault(vmf); else ret = do_shared_fault(vmf); @@ -3698,7 +3699,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * accessible ptes, some can allow access by kernel mode. */ old_pte = ptep_modify_prot_start(vma, vmf->address, vmf->pte); - pte = pte_modify(old_pte, vma->vm_page_prot); + pte = pte_modify(old_pte, vmf->vma_page_prot); pte = pte_mkyoung(pte); if (was_writable) pte = pte_mkwrite(pte); @@ -3732,7 +3733,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * Flag if the page is shared between multiple address spaces. This * is later used when determining whether to group tasks together */ - if (page_mapcount(page) > 1 && (vma->vm_flags & VM_SHARED)) + if (page_mapcount(page) > 1 && (vmf->vma_flags & VM_SHARED)) flags |= TNF_SHARED; last_cpupid = page_cpupid_last(page); @@ -3777,7 +3778,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd) return vmf->vma->vm_ops->huge_fault(vmf, PE_SIZE_PMD); /* COW handled on pte level: split pmd */ - VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma); + VM_BUG_ON_VMA(vmf->vma_flags & VM_SHARED, vmf->vma); __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL); return VM_FAULT_FALLBACK; @@ -3924,6 +3925,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, .flags = flags, .pgoff = linear_page_index(vma, address), .gfp_mask = __get_fault_gfp_mask(vma), + .vma_flags = vma->vm_flags, + .vma_page_prot = vma->vm_page_prot, }; unsigned int dirty = flags & FAULT_FLAG_WRITE; struct mm_struct *mm = vma->vm_mm; diff --git a/mm/migrate.c b/mm/migrate.c index f2ecc2855a12..a9138093a8e2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -240,7 +240,7 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma, */ entry = pte_to_swp_entry(*pvmw.pte); if (is_write_migration_entry(entry)) - pte = maybe_mkwrite(pte, vma); + pte = maybe_mkwrite(pte, vma->vm_flags); if (unlikely(is_zone_device_page(new))) { if (is_device_private_page(new)) { From patchwork Tue Apr 16 13:45:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903181 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4F54B17E1 for ; Tue, 16 Apr 2019 13:48:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E1AB2843C for ; Tue, 16 Apr 2019 13:48:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E306286CF; Tue, 16 Apr 2019 13:48:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 91DEB2843C for ; Tue, 16 Apr 2019 13:48:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8155E6B0294; Tue, 16 Apr 2019 09:47:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 79C746B0296; Tue, 16 Apr 2019 09:47:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68C716B0297; Tue, 16 Apr 2019 09:47:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 4874B6B0294 for ; Tue, 16 Apr 2019 09:47:44 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id s11so15575940ywa.18 for ; Tue, 16 Apr 2019 06:47:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=Xp+UT7lx7SKMZtWDQJKM8VxsYgdJIamGRljD+mKc5XE=; b=ZrDl4OqRnnsCrrZWQnma48FWkAr4CRBM8fHHI618/XUUQ8zwYH3X6denqIlbnEJ10v 3iGqidHyTU33M3mGE3npzna8SAF7rN8w3/r7CJzcOseNanXWkT7RFpQDkti6E2u6jBT8 O/JHKsNz7/w8CFyKTxbkFCmCOwM36uMJjwBndSXW/99eJlPfqV6g4PUSzNNHiFrkkaY1 pC2QMarHyBesLmIurYb0Z0Lv9N6tiG904ka7K66MolC6XxaqVSjAslFGCPs8I4ZhHteS 44PznEpQFLRSMiEamxAm5V1Ve7VI/7X7YrXPbyjMV2BG819Br4EqsSkWWZ/yB/oIgbvk MrNg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAXLiUXFWAGLEyi4nK5de+WkL9gHDmZ+U27zBe4gaKci06l4epR6 FoQidkwzNSbch2f1D9Tw1/K4ZupK/AuY7ifjEpT/7QACGrDtjRY7YBebHbxH0Koms5AAI0kfRI2 KYIR1bcBxkPkw3B9MQK+m8/L2mb8E05TXUq+cZTL0H5PD5Pqdk/X2pRbb8LdCLxVneg== X-Received: by 2002:a81:9ad0:: with SMTP id r199mr64350296ywg.310.1555422463952; Tue, 16 Apr 2019 06:47:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqw6rknmXsC9+fRRNst14wRcT0aQxPNHrKLZQAo7DGENDzFtENsgcBQjAHCBzMHOT7NJpi5t X-Received: by 2002:a81:9ad0:: with SMTP id r199mr64350211ywg.310.1555422462976; Tue, 16 Apr 2019 06:47:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422462; cv=none; d=google.com; s=arc-20160816; b=Nsd0d4teBOQnbmHMhlRxpp1lW8SN5apYyNRn2T3uSDkTEoVKYUUZ58P3cz5CCl+z6N pIpJRKl3r6ZmW2Qq/8xmWgmEjqiEhYkJh6ov24813Xp58y1Z7ofc5b3uaxV3BIzRMotf UdbuAefzBPnJE8q2o6BC1ohHkUA8hz6du1aFf4P3T9ZakFHISUVcd+bY0N0SeA9ZWDef O8ZvYGGbcO0E1g3sAP4SHxpowsK8i4sJZ8jStxttl/eh2WtVueS7iYrp+ya4P8mgpFSL Kw5TeuLzoUHw0/j0pcuOK/ERFgY3pAWhRwqLJeazPYInoV48BbdPzMzbDAPVyWYhx0h6 II0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=Xp+UT7lx7SKMZtWDQJKM8VxsYgdJIamGRljD+mKc5XE=; b=UUBFeGTVU5I9wN82UBugHoS5gWj5QfDQdRpydOWHgFOlYG8fzx4Vv1fkBiZXwbYpgY YuFOhLmkgkj+eYxQMBFqMzwVHgxx7AYlAZgZg7D3DsEkL6y26oD+5Yp9SzkXGi8y5SCl /ZdKvYwejVzJYK1jAH1k4N+cogVX41nI6w9/9DYvQMaGb2EiIdTI62dWtFkB/Kb7gIQq vRE/FXpaHi3/FzERfKMk/MzEgO8Jd8wLgLN8R1Zy0Laup27rdzBpLbjQTyuck90ysMM2 VtiL5YsxxpPmcYh+G+A6wTEf5IxrhcygvyCa58dSTUMzxoPOf0BNHXMLy7tZwr92aRFQ JYCA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id u31si20813894ywh.269.2019.04.16.06.47.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:42 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlKYP091712 for ; Tue, 16 Apr 2019 09:47:42 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwfsus89w-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:30 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:17 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:07 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDk5E642074152 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:05 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 735CD4C050; Tue, 16 Apr 2019 13:46:05 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8A1104C040; Tue, 16 Apr 2019 13:46:01 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:01 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 14/31] mm/migrate: Pass vm_fault pointer to migrate_misplaced_page() Date: Tue, 16 Apr 2019 15:45:05 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7273 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD8A Message-Id: <20190416134522.17540-15-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP migrate_misplaced_page() is only called during the page fault handling so it's better to pass the pointer to the struct vm_fault instead of the vma. This way during the speculative page fault path the saved vma->vm_flags could be used. Acked-by: David Rientjes Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/migrate.h | 4 ++-- mm/memory.c | 2 +- mm/migrate.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index e13d9bf2f9a5..0197e40325f8 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -125,14 +125,14 @@ static inline void __ClearPageMovable(struct page *page) #ifdef CONFIG_NUMA_BALANCING extern bool pmd_trans_migrating(pmd_t pmd); extern int migrate_misplaced_page(struct page *page, - struct vm_area_struct *vma, int node); + struct vm_fault *vmf, int node); #else static inline bool pmd_trans_migrating(pmd_t pmd) { return false; } static inline int migrate_misplaced_page(struct page *page, - struct vm_area_struct *vma, int node) + struct vm_fault *vmf, int node) { return -EAGAIN; /* can't migrate now */ } diff --git a/mm/memory.c b/mm/memory.c index d0de58464479..56802850e72c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3747,7 +3747,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) } /* Migrate to the requested node */ - migrated = migrate_misplaced_page(page, vma, target_nid); + migrated = migrate_misplaced_page(page, vmf, target_nid); if (migrated) { page_nid = target_nid; flags |= TNF_MIGRATED; diff --git a/mm/migrate.c b/mm/migrate.c index a9138093a8e2..633bd9abac54 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1938,7 +1938,7 @@ bool pmd_trans_migrating(pmd_t pmd) * node. Caller is expected to have an elevated reference count on * the page that will be dropped by this function before returning. */ -int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, +int migrate_misplaced_page(struct page *page, struct vm_fault *vmf, int node) { pg_data_t *pgdat = NODE_DATA(node); @@ -1951,7 +1951,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma, * with execute permissions as they are probably shared libraries. */ if (page_mapcount(page) != 1 && page_is_file_cache(page) && - (vma->vm_flags & VM_EXEC)) + (vmf->vma_flags & VM_EXEC)) goto out; /* From patchwork Tue Apr 16 13:45:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903185 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50D3C17E1 for ; Tue, 16 Apr 2019 13:48:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37B612843C for ; Tue, 16 Apr 2019 13:48:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A853286CF; Tue, 16 Apr 2019 13:48:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 834702843C for ; Tue, 16 Apr 2019 13:48:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4927F6B0298; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 418356B029A; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E43F6B029C; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f71.google.com (mail-yw1-f71.google.com [209.85.161.71]) by kanga.kvack.org (Postfix) with ESMTP id 007DA6B0298 for ; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) Received: by mail-yw1-f71.google.com with SMTP id v123so15583162ywf.16 for ; Tue, 16 Apr 2019 06:48:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=3IPoxrf+6+tcIsdMy7SAxB3smSVYv9hSJsTrPSmFiK0=; b=iF/K2QaFd7uNBKkCrRXJd1IzQBm5j481ldDS9Kmm7Q64Ou0aqOi0w86QKmSU6iobru VZCl5G2mYspqEof2PXctprDd0OcNNRTE0ijBoqOPOqJQ74QVsKMUeW3S1yxvDSp+w4Do w7fIew3CzKMhQUJkiIgXoYZEwrRXyDZMotFMIJ+e4naGYxJXowG6QEwNb4akFVXBkFuR 1yrIyHGIxb4ZdS2pGzB5l7xbzMErrRNKBLJU9TdoaVZer9V2P+pL115Zh7Q4c+rgwX63 pq9xh3Hcs8YkDkFZQ39OZyQLbd5+0GwZtTSi5eofd16KDncRVFIxZgErcMoQ85d7wHQo 2dVQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAWgotX5BZ10wJgN+WRMcAsfQ8DjQuWJnnelQrcZOqQypxGC09LP u1uZmeY6faMCWQLzYG7S+BO9ob8uWdtjFIG8kQr01TeE+gxA36FasciKBpvYXr+ddGu+xPJFJlU SycVhiSKboGx6x7/S5n/9b1xSAWEPY9tZViJA7GpWJDaq/9NExytTDLrAUUr4AXW12A== X-Received: by 2002:a81:2849:: with SMTP id o70mr65690896ywo.180.1555422482684; Tue, 16 Apr 2019 06:48:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqw00QYIBluyT61NpLFJhGzZttDdEPvqvAVvCkFS8ESKNlrby4kM3MrCIZhXYEgV6Ez8wGWl X-Received: by 2002:a81:2849:: with SMTP id o70mr65690688ywo.180.1555422480392; Tue, 16 Apr 2019 06:48:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422480; cv=none; d=google.com; s=arc-20160816; b=Lu6niWPC7BBnvp1js7r8BZgSpCWFelhQ1wZ+EIZJYdezfs7NFbLhcvMY8uQonal7xO nOKqfh+zyP5mJY/35llJPgJodXI8ZPKCMfj0wc1/hwnWxFOyPCMssi6mXnugsZ252I8U qpIxrnWe/Qbu28eo085XmlmnbGq9fxSO74Z78wnfRBWsQcrMr/N/xlojgEobLZ16XVwA xxiQJiGeS+gjwb/oiFl2iIyvZX2tBeapSu6CXMHeo2AMRPIalnSBlrnR+wj6kweEq2wN MX/Taggy4YUIX2ugaYEXyTOTJ8PcNDwVk2lGf6GzJsxzLG7NRVe3VFwyHxpRixxQwJV3 IMMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=3IPoxrf+6+tcIsdMy7SAxB3smSVYv9hSJsTrPSmFiK0=; b=MlRWzpxdVj5A2odgjxwwp3uPClRjv8CAaG7d5dOZ3svDB3+0HZ+8gzngjrpzldEe1I vRszwoj37RvEghe9mWTJDKRCJcrdCGLtngJnKHpGhNJCvLpafecO/fSzYN9q9fD5P5Vl 2S/FqPQTW/xOjgPgHSu/ofZJbiaWUIn1SjQByjYX7ghrFGfaZ5vam9J5z4u2rwf7xAU3 4xi+OO4da30GVFeSmnOCYWyC2GAS07zgGp+v3NLtw23L6Izm5UAJHaKYqaa40zx5KfR5 QITuNKkjO7QgXnsZRpZ+VDvD4v9jZmY9Td/V4RzwKxniQ12HPryXQKZPve0iWWASN4oi HuEw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id j185si30165534ywf.80.2019.04.16.06.48.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:48:00 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlTue093894 for ; Tue, 16 Apr 2019 09:47:59 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe1t6dqp-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:45 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:20 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:11 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDk9qK37617726 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:09 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 978DD4C040; Tue, 16 Apr 2019 13:46:09 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9DF8D4C04A; Tue, 16 Apr 2019 13:46:06 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:06 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 15/31] mm: introduce __lru_cache_add_active_or_unevictable Date: Tue, 16 Apr 2019 15:45:06 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7275 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD8C Message-Id: <20190416134522.17540-16-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The speculative page fault handler which is run without holding the mmap_sem is calling lru_cache_add_active_or_unevictable() but the vm_flags is not guaranteed to remain constant. Introducing __lru_cache_add_active_or_unevictable() which has the vma flags value parameter instead of the vma pointer. Acked-by: David Rientjes Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/swap.h | 10 ++++++++-- mm/memory.c | 8 ++++---- mm/swap.c | 6 +++--- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 4bfb5c4ac108..d33b94eb3c69 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -343,8 +343,14 @@ extern void deactivate_file_page(struct page *page); extern void mark_page_lazyfree(struct page *page); extern void swap_setup(void); -extern void lru_cache_add_active_or_unevictable(struct page *page, - struct vm_area_struct *vma); +extern void __lru_cache_add_active_or_unevictable(struct page *page, + unsigned long vma_flags); + +static inline void lru_cache_add_active_or_unevictable(struct page *page, + struct vm_area_struct *vma) +{ + return __lru_cache_add_active_or_unevictable(page, vma->vm_flags); +} /* linux/mm/vmscan.c */ extern unsigned long zone_reclaimable_pages(struct zone *zone); diff --git a/mm/memory.c b/mm/memory.c index 56802850e72c..85ec5ce5c0a8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2347,7 +2347,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) ptep_clear_flush_notify(vma, vmf->address, vmf->pte); page_add_new_anon_rmap(new_page, vma, vmf->address, false); mem_cgroup_commit_charge(new_page, memcg, false, false); - lru_cache_add_active_or_unevictable(new_page, vma); + __lru_cache_add_active_or_unevictable(new_page, vmf->vma_flags); /* * We call the notify macro here because, when using secondary * mmu page tables (such as kvm shadow page tables), we want the @@ -2896,7 +2896,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(page != swapcache && swapcache)) { page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); - lru_cache_add_active_or_unevictable(page, vma); + __lru_cache_add_active_or_unevictable(page, vmf->vma_flags); } else { do_page_add_anon_rmap(page, vma, vmf->address, exclusive); mem_cgroup_commit_charge(page, memcg, true, false); @@ -3048,7 +3048,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); - lru_cache_add_active_or_unevictable(page, vma); + __lru_cache_add_active_or_unevictable(page, vmf->vma_flags); setpte: set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); @@ -3327,7 +3327,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); - lru_cache_add_active_or_unevictable(page, vma); + __lru_cache_add_active_or_unevictable(page, vmf->vma_flags); } else { inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page)); page_add_file_rmap(page, false); diff --git a/mm/swap.c b/mm/swap.c index 3a75722e68a9..a55f0505b563 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -450,12 +450,12 @@ void lru_cache_add(struct page *page) * directly back onto it's zone's unevictable list, it does NOT use a * per cpu pagevec. */ -void lru_cache_add_active_or_unevictable(struct page *page, - struct vm_area_struct *vma) +void __lru_cache_add_active_or_unevictable(struct page *page, + unsigned long vma_flags) { VM_BUG_ON_PAGE(PageLRU(page), page); - if (likely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) != VM_LOCKED)) + if (likely((vma_flags & (VM_LOCKED | VM_SPECIAL)) != VM_LOCKED)) SetPageActive(page); else if (!TestSetPageMlocked(page)) { /* From patchwork Tue Apr 16 13:45:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED7C517E1 for ; Tue, 16 Apr 2019 13:47:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB7BC2843C for ; Tue, 16 Apr 2019 13:47:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF91228A0B; Tue, 16 Apr 2019 13:47:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1993A289FC for ; Tue, 16 Apr 2019 13:47:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D42426B000E; Tue, 16 Apr 2019 09:46:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CF2866B0266; Tue, 16 Apr 2019 09:46:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBFB96B0269; Tue, 16 Apr 2019 09:46:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 6A92D6B000E for ; Tue, 16 Apr 2019 09:46:53 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id p26so2657471edy.19 for ; Tue, 16 Apr 2019 06:46:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=wYGpYk7Grim5SqOFzD8BUE/Zqht2/KBjOtsFukYIArU=; b=Ajg1ZFka58JbpZefLR9GBKrK1g2N75wb0ULA88/1e3aGAwysQ9HG0SMr3yVstDq0Jx 6g9s7Baq6BRquJqfJ+LNcpc0Zo/MPS+rd1po2j21hmTEwljat51hOKHBwQvz9ruqJN/V O0eFXX6GQkpiGtPPdBnYOveAiSfgyk7P+2MdHwg3skd/3hvouXxyVZYJcKYcWz43LrRS Y4j+qz3rhwnL1AWkrTWKYweAI8usxakNUgLMi8nS6L7TufRyDr4Dz//HI1R0J3kKoj7q J1UGvMLQXaq731KD02tHMfGytosa9TQF2S9DIK12IO3kcQFiXWt7VZ48+m5LmVly/zvn O5Og== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVwYSqp+gq+qfTtiSoGx9rXvES3GUk2jxjf5jtTQYz472swPRiz oYNVibtz4R15MEXkcU58LBPECY8OAEIycK1tpGLkA/Nc/rQbzdPEtzaMukji4IPYnSpab8RUt5F SFj4LkhFioLtyEPbX3rwg6Uzr0iaXhguAj195AfgRHE8XXLhqauW5vzrIZwyXQcGjbQ== X-Received: by 2002:a50:a4e4:: with SMTP id x33mr51420671edb.61.1555422412856; Tue, 16 Apr 2019 06:46:52 -0700 (PDT) X-Google-Smtp-Source: APXvYqxGEaXI+p6jZn4Zt6ynvNHdst++MUcCbHWGq16jcvxt5wa6+L0D/iDwzVaqoQyiMZ91V0qI X-Received: by 2002:a50:a4e4:: with SMTP id x33mr51420359edb.61.1555422407946; Tue, 16 Apr 2019 06:46:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422407; cv=none; d=google.com; s=arc-20160816; b=gTklXpTZtDwihGdJFnqiCbhGW5LGo6of1r3dViFhSbMWKfSGTz92bxZrfXheSBLGJn /daEhxM4lYIPl0tnSObe4M/Sx5AlNoUgYvHIWbBBaiiR/RLgPPFK3JoCIPW0G/ih01Fg T9x13Iy8mHnF6OJ3Xaf4z07sulVdGPGqFk/tzUlCnAkktKl+iudV5itXeJK8YoO4wVqd bhbvv+27NxoA5Zmp7MJvIts4qxOcwEwTimq49kLTNOko3LYMzxgrLTE0LtQ4Z4cEb1fU Rs/nfLv5uQOrhzesXlo/DBvT0nyndUvXQZ2u1IysPJp61X8RgAHt8UE4p8aU6uZt7OQc UhKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=wYGpYk7Grim5SqOFzD8BUE/Zqht2/KBjOtsFukYIArU=; b=tR0SZme56GHBVpwAiiioS6qmf4KcJEQ15bMFkyKNYraA/WNmG4WB1r69pToylKatw8 Kx2Fv1UReqPMUioiNHAJ+2ML+ECqreNJiwyv5zeJrCt40WlMydrmPUINej8Ien7wupWA JLZ85LJZ8/SlPz6ke1UZE4E2i43rdNY/AClanzzO1rScEAI1p8MdMXYmOgiiMWcl+oXD DT2MwanjmP3OorUYO+o85Wm0c30QuONs2zTqN4CTz+FspUWMNfgoFyvW3HRX1gfQprvp ubtvTrNW2j8quBSdl1mkCupzC1PEe7Mcw+xwiVSbxzXhHq629x3wf4SnJzuzVxaWgRCC prBw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id en2si3323203ejb.6.2019.04.16.06.46.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:47 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkZbA113297 for ; Tue, 16 Apr 2019 09:46:46 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwentmu3g-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:40 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:22 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:13 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkCpF41353468 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:12 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0FEDD4C04E; Tue, 16 Apr 2019 13:46:12 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A69A94C04A; Tue, 16 Apr 2019 13:46:10 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:10 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 16/31] mm: introduce __vm_normal_page() Date: Tue, 16 Apr 2019 15:45:07 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0028-0000-0000-0000036170BF X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0029-0000-0000-00002420A85F Message-Id: <20190416134522.17540-17-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When dealing with the speculative fault path we should use the VMA's field cached value stored in the vm_fault structure. Currently vm_normal_page() is using the pointer to the VMA to fetch the vm_flags value. This patch provides a new __vm_normal_page() which is receiving the vm_flags flags value as parameter. Note: The speculative path is turned on for architecture providing support for special PTE flag. So only the first block of vm_normal_page is used during the speculative path. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm.h | 18 +++++++++++++++--- mm/memory.c | 21 ++++++++++++--------- 2 files changed, 27 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f465bb2b049e..f14b2c9ddfd4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1421,9 +1421,21 @@ static inline void INIT_VMA(struct vm_area_struct *vma) #endif } -struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, - pte_t pte, bool with_public_device); -#define vm_normal_page(vma, addr, pte) _vm_normal_page(vma, addr, pte, false) +struct page *__vm_normal_page(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, bool with_public_device, + unsigned long vma_flags); +static inline struct page *_vm_normal_page(struct vm_area_struct *vma, + unsigned long addr, pte_t pte, + bool with_public_device) +{ + return __vm_normal_page(vma, addr, pte, with_public_device, + vma->vm_flags); +} +static inline struct page *vm_normal_page(struct vm_area_struct *vma, + unsigned long addr, pte_t pte) +{ + return _vm_normal_page(vma, addr, pte, false); +} struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t pmd); diff --git a/mm/memory.c b/mm/memory.c index 85ec5ce5c0a8..be93f2c8ebe0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -533,7 +533,8 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr, } /* - * vm_normal_page -- This function gets the "struct page" associated with a pte. + * __vm_normal_page -- This function gets the "struct page" associated with + * a pte. * * "Special" mappings do not wish to be associated with a "struct page" (either * it doesn't exist, or it exists but they don't want to touch it). In this @@ -574,8 +575,9 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr, * PFNMAP mappings in order to support COWable mappings. * */ -struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, - pte_t pte, bool with_public_device) +struct page *__vm_normal_page(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, bool with_public_device, + unsigned long vma_flags) { unsigned long pfn = pte_pfn(pte); @@ -584,7 +586,7 @@ struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, goto check_pfn; if (vma->vm_ops && vma->vm_ops->find_special_page) return vma->vm_ops->find_special_page(vma, addr); - if (vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) + if (vma_flags & (VM_PFNMAP | VM_MIXEDMAP)) return NULL; if (is_zero_pfn(pfn)) return NULL; @@ -620,8 +622,8 @@ struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, /* !CONFIG_ARCH_HAS_PTE_SPECIAL case follows: */ - if (unlikely(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))) { - if (vma->vm_flags & VM_MIXEDMAP) { + if (unlikely(vma_flags & (VM_PFNMAP|VM_MIXEDMAP))) { + if (vma_flags & VM_MIXEDMAP) { if (!pfn_valid(pfn)) return NULL; goto out; @@ -630,7 +632,7 @@ struct page *_vm_normal_page(struct vm_area_struct *vma, unsigned long addr, off = (addr - vma->vm_start) >> PAGE_SHIFT; if (pfn == vma->vm_pgoff + off) return NULL; - if (!is_cow_mapping(vma->vm_flags)) + if (!is_cow_mapping(vma_flags)) return NULL; } } @@ -2532,7 +2534,8 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - vmf->page = vm_normal_page(vma, vmf->address, vmf->orig_pte); + vmf->page = __vm_normal_page(vma, vmf->address, vmf->orig_pte, false, + vmf->vma_flags); if (!vmf->page) { /* * VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a @@ -3706,7 +3709,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, pte); update_mmu_cache(vma, vmf->address, vmf->pte); - page = vm_normal_page(vma, vmf->address, pte); + page = __vm_normal_page(vma, vmf->address, pte, false, vmf->vma_flags); if (!page) { pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; From patchwork Tue Apr 16 13:45:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E35A8161F for ; Tue, 16 Apr 2019 13:47:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3EA1288CF for ; Tue, 16 Apr 2019 13:47:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C1C24289F9; Tue, 16 Apr 2019 13:47:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0DF6A289F5 for ; Tue, 16 Apr 2019 13:47:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7295F6B0272; Tue, 16 Apr 2019 09:47:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6890F6B0273; Tue, 16 Apr 2019 09:47:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 502246B0274; Tue, 16 Apr 2019 09:47:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id 2ABC46B0272 for ; Tue, 16 Apr 2019 09:47:05 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id y6so15678807ybb.20 for ; Tue, 16 Apr 2019 06:47:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=VgB2qt03NSBRvN1xU3ANZYNuPmwAImtbKRl4F+x+2x8=; b=OFnF87wdqosKi/slonV0B8gY6mY+bzSqNsRQnXtv17wz+zjXCs1mZH/4fiyjLvD0tV ZDim9CP7c7x03DGHFNv+gzgaRw5B+1QJQUWoN9QoE49aSJnf4y69vBGmVkmOOESCTojS 4gl2RYStGUqSzIE0/iSuSnTgDtqJYm1JVzBfvk1xY7drcDta36F7nqYQ96R43oxYs6Cw +6ld8jtqgBV2RGVFHmAGftg8OFzhhCkIpViYeQR/thanEDwtiGCK1Tb0jlmJ3RJVxPqP BFqBEYRYFN26S2nodW08vjbIouG3+wBS+gqmv/Xzrpi75KgYkWpUuRb6Z0FLl9x1A9c4 Z4zw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAUe8a6Bud3DVYdBhHc/c0Qfb86Buh9weEhQe7QVAc8Dl3/k4Jys hUWuhCNppBBBOg6p0ObIK/tOL4R1UZvBuWonrRW6xK6Nfv1AMpeFAnKfRTveo4y8qw6RSsZx60w JvhVXhs63IpSvi0ODxHEYEiFYfMc5qEAdHCUIB77ISdowu+cc8+nnGvHzS7EISzZuug== X-Received: by 2002:a81:3010:: with SMTP id w16mr64610223yww.388.1555422424818; Tue, 16 Apr 2019 06:47:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqyptaVY5GIm2vfaldjrKv3GU7DNKQF7h6uzumICHW0rVrq47oH+faMftgOXoin1vbYkU28W X-Received: by 2002:a81:3010:: with SMTP id w16mr64610110yww.388.1555422423596; Tue, 16 Apr 2019 06:47:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422423; cv=none; d=google.com; s=arc-20160816; b=kg0DYAnrLcNvjTGTdzvpTnvYOGF5bQN5hGQKJbagOsz/cRdMb85rxmpV5DIXcdPTQk FGLpY2cDiE3JhJBXpQJku6YSFB4ShfA3r+69Lba8g2TM8FLHoCxJm2JkduOfKsmJAtxY b1QXo8B91RNQ4laDTd9jQT3xZVEFtU4/+OJGe9hVmIrK4UnPXS9zheLC+x2c6/gMkb7i b/wVcGTEXpDEzDMm3bOtWGiDEQPHJR/1STOg3F2M4xdSPH8GP2cnbAj7zfYHZ88FNPFV AFS0a1nnMbEJL8HDX891WgQHs0PKDlnGF1uziHc5od4IBiXX5SMFYdrUJSa9u1+8Zl2s c5Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=VgB2qt03NSBRvN1xU3ANZYNuPmwAImtbKRl4F+x+2x8=; b=LfOErS3HKuC93L3QYTDhXsK/ix8anW6+qnsEnsQEy1T7xuP9iEK+8ug/zwwk3ooFeS m/xNFQEeXLJZ/ovC6F/ORn/bv3KOfGuZpcJiPbq09qcZDESS6mJjitjB9O12cm/Xmyk+ Hseze6cHpDAp42Owy9JFYv8lW2fUmKPA8I+jat3M4tipo1pg3jgNYVLG787eJnULLpxr GyZjeUHygrnkxhvw5M0Yt3L+XlN6PjzXf5TnsLZ30EaGXg6YG4rB8z9nIvh1ePXhhP/R BqbmTpBWfWkxccCkv9GvS/NU5wKBlvPjuFfe+0lqswEBdQNXxGjjgw4AwQ82kzc0Sz6d yrFA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id 132si33888358ybo.403.2019.04.16.06.47.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkioX114207 for ; Tue, 16 Apr 2019 09:47:02 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwentmu4q-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:53 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:24 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:16 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkE4q23724124 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:14 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 905DC4C04A; Tue, 16 Apr 2019 13:46:14 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 28B704C046; Tue, 16 Apr 2019 13:46:13 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:13 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 17/31] mm: introduce __page_add_new_anon_rmap() Date: Tue, 16 Apr 2019 15:45:08 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0028-0000-0000-0000036170C0 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0029-0000-0000-00002420A860 Message-Id: <20190416134522.17540-18-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When dealing with speculative page fault handler, we may race with VMA being split or merged. In this case the vma->vm_start and vm->vm_end fields may not match the address the page fault is occurring. This can only happens when the VMA is split but in that case, the anon_vma pointer of the new VMA will be the same as the original one, because in __split_vma the new->anon_vma is set to src->anon_vma when *new = *vma. So even if the VMA boundaries are not correct, the anon_vma pointer is still valid. If the VMA has been merged, then the VMA in which it has been merged must have the same anon_vma pointer otherwise the merge can't be done. So in all the case we know that the anon_vma is valid, since we have checked before starting the speculative page fault that the anon_vma pointer is valid for this VMA and since there is an anon_vma this means that at one time a page has been backed and that before the VMA is cleaned, the page table lock would have to be grab to clean the PTE, and the anon_vma field is checked once the PTE is locked. This patch introduce a new __page_add_new_anon_rmap() service which doesn't check for the VMA boundaries, and create a new inline one which do the check. When called from a page fault handler, if this is not a speculative one, there is a guarantee that vm_start and vm_end match the faulting address, so this check is useless. In the context of the speculative page fault handler, this check may be wrong but anon_vma is still valid as explained above. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/rmap.h | 12 ++++++++++-- mm/memory.c | 8 ++++---- mm/rmap.c | 5 ++--- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 988d176472df..a5d282573093 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -174,8 +174,16 @@ void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long, bool); void do_page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long, int); -void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, - unsigned long, bool); +void __page_add_new_anon_rmap(struct page *, struct vm_area_struct *, + unsigned long, bool); +static inline void page_add_new_anon_rmap(struct page *page, + struct vm_area_struct *vma, + unsigned long address, bool compound) +{ + VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); + __page_add_new_anon_rmap(page, vma, address, compound); +} + void page_add_file_rmap(struct page *, bool); void page_remove_rmap(struct page *, bool); diff --git a/mm/memory.c b/mm/memory.c index be93f2c8ebe0..46f877b6abea 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2347,7 +2347,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * thread doing COW. */ ptep_clear_flush_notify(vma, vmf->address, vmf->pte); - page_add_new_anon_rmap(new_page, vma, vmf->address, false); + __page_add_new_anon_rmap(new_page, vma, vmf->address, false); mem_cgroup_commit_charge(new_page, memcg, false, false); __lru_cache_add_active_or_unevictable(new_page, vmf->vma_flags); /* @@ -2897,7 +2897,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) /* ksm created a completely new copy */ if (unlikely(page != swapcache && swapcache)) { - page_add_new_anon_rmap(page, vma, vmf->address, false); + __page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); __lru_cache_add_active_or_unevictable(page, vmf->vma_flags); } else { @@ -3049,7 +3049,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) } inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, vma, vmf->address, false); + __page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); __lru_cache_add_active_or_unevictable(page, vmf->vma_flags); setpte: @@ -3328,7 +3328,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg, /* copy-on-write page */ if (write && !(vmf->vma_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, vma, vmf->address, false); + __page_add_new_anon_rmap(page, vma, vmf->address, false); mem_cgroup_commit_charge(page, memcg, false, false); __lru_cache_add_active_or_unevictable(page, vmf->vma_flags); } else { diff --git a/mm/rmap.c b/mm/rmap.c index e5dfe2ae6b0d..2148e8ce6e34 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1140,7 +1140,7 @@ void do_page_add_anon_rmap(struct page *page, } /** - * page_add_new_anon_rmap - add pte mapping to a new anonymous page + * __page_add_new_anon_rmap - add pte mapping to a new anonymous page * @page: the page to add the mapping to * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped @@ -1150,12 +1150,11 @@ void do_page_add_anon_rmap(struct page *page, * This means the inc-and-test can be bypassed. * Page does not have to be locked. */ -void page_add_new_anon_rmap(struct page *page, +void __page_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, bool compound) { int nr = compound ? hpage_nr_pages(page) : 1; - VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); __SetPageSwapBacked(page); if (compound) { VM_BUG_ON_PAGE(!PageTransHuge(page), page); From patchwork Tue Apr 16 13:45:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903179 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29E9F17E1 for ; Tue, 16 Apr 2019 13:48:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B4212843C for ; Tue, 16 Apr 2019 13:48:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F04C0286CF; Tue, 16 Apr 2019 13:48:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 537472843C for ; Tue, 16 Apr 2019 13:48:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48F5D6B0292; Tue, 16 Apr 2019 09:47:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 43FEB6B0294; Tue, 16 Apr 2019 09:47:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DFE26B0295; Tue, 16 Apr 2019 09:47:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f72.google.com (mail-yw1-f72.google.com [209.85.161.72]) by kanga.kvack.org (Postfix) with ESMTP id 0ABA86B0292 for ; Tue, 16 Apr 2019 09:47:42 -0400 (EDT) Received: by mail-yw1-f72.google.com with SMTP id x66so15618571ywx.1 for ; Tue, 16 Apr 2019 06:47:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=q7pYK44vfsHNTj5O8epyA0bbX50XM1Dt0+DiR1yO2TQ=; b=TDZ9n24leFjxKryIGEHUErrCSn7wNo7IQSeqLW/1W8U5hU4TLuA6J6dqmH6ej9YJpm GFKhAGAuX1TUPbWpCFaILkmPWshLGbhiGE/cDj/AJv9f5R6VBh0C3r2eUTTn0D/uNcgY kD93NYGAZJe5fbWgiExIKUF23U1RGuKrTBUsoVZwoTFEP9r3hIUWnNFnqCtyIulkKcK/ xdQ25i15M8tUkJ3gNbP3fXZt9ZezaLFT1/llZdWEdCz6b2HHZKmPNdObbreWauJg6QAM /VWclTBKyQF0NElYvcBgRmviVWcdupOxhBAjAMbjST9xQKSBnVicugLYpl9M3ispzuyE ZZbQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVHhsnx7XL7HmTA213reB3X5K4IVWOOs7DixQ9RxO6QaM0Dnmyo o+3laR4VvgqxFmSXVRtODmFwVbMt45cZYF9WNvCs97ginpLNBqF2Gmao/R8AdIegYMC6Gjsy/O4 sjo5qXDP/DrKdajZXrfl6Fk6ZK3OO5W0QNY8SWW7so6a8SxtCWSjFiDd1zeWR4pHarQ== X-Received: by 2002:a81:3bc5:: with SMTP id i188mr65480930ywa.404.1555422461742; Tue, 16 Apr 2019 06:47:41 -0700 (PDT) X-Google-Smtp-Source: APXvYqxZydaWgDW8ot4StG3je2hDfN9E+tXdWNMyNEVr4CZxnuKR8JSnIS3wdJadMF2nnpyMaB/K X-Received: by 2002:a81:3bc5:: with SMTP id i188mr65480822ywa.404.1555422460503; Tue, 16 Apr 2019 06:47:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422460; cv=none; d=google.com; s=arc-20160816; b=vABQGQWBhbAwfh1UbgG6H7snw3rWaBwXm/xioXAHzRv3j8i34U0fHAMNT4fj2PTvWS TpLCuslJsNQ8GB4lgzVeRBB11Vngd3RvXIeO2WbLw3IBajH/jPXD6YOc927hL6SvcMaa 3dh+bb4JwwjLp5rRAXGvVnXfFiU+ULNcmrQBP9dynoHtWVwb4QITejd08PyXT+4gq9KP GecEZVccLs2T1CSG0J0rZjzSmBm6ZBLhVUxqigDX8oEZB1n1YxUP0TPYoOpb966RHx7A bN6iLtVR/2GfEDdAGEEDm4dN11/Cx3vnYsc+KEuJwRl/G8Vwhhq2nRkPtcyFg/fxIsnr i2Mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=q7pYK44vfsHNTj5O8epyA0bbX50XM1Dt0+DiR1yO2TQ=; b=IjdNr4Zy1/UIudN5VeAFMKl4VAqip4zQNi+xJPgQ5Ws03RRLT+TzNnY50U6Z/GVtr6 Hsm/2N1p3iqMG+Y/sRyWyfmPAXMEG1GYBbdITi8Pr8bW5RxAWlpx0qmoZxGw+9PF+g0j k25jeOdfCszP6pUkqnlnkOVc55IKYgr2qCJRUigGrNVUUWvBDwcRU9EA4fRkFvrVhRiN nSeOmlc+T3iWwefrqKkwH3vva1+nsQK9XPQ1IoeBLIkfOMnyMFCy9MMSNPccSsez3Kce gkpF5nLocltdgFIJOg19BfB430ct8RSFJu57Y3yMmuwPOXrGfBCE+YV/9TqkiYwm+sD0 XP+g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id x126si11773922ybc.456.2019.04.16.06.47.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:40 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlNPO091859 for ; Tue, 16 Apr 2019 09:47:39 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwfsus8fr-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:30 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:29 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:19 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkH0i8716466 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:17 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 662084C040; Tue, 16 Apr 2019 13:46:17 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A03F74C046; Tue, 16 Apr 2019 13:46:15 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:15 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Vinayak Menon Subject: [PATCH v12 18/31] mm: protect against PTE changes done by dup_mmap() Date: Tue, 16 Apr 2019 15:45:09 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-4275-0000-0000-000003287653 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-4276-0000-0000-00003837A75B Message-Id: <20190416134522.17540-19-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Vinayak Menon and Ganesh Mahendran reported that the following scenario may lead to thread being blocked due to data corruption: CPU 1 CPU 2 CPU 3 Process 1, Process 1, Process 1, Thread A Thread B Thread C while (1) { while (1) { while(1) { pthread_mutex_lock(l) pthread_mutex_lock(l) fork pthread_mutex_unlock(l) pthread_mutex_unlock(l) } } } In the details this happens because : CPU 1 CPU 2 CPU 3 fork() copy_pte_range() set PTE rdonly got to next VMA... . PTE is seen rdonly PTE still writable . thread is writing to page . -> page fault . copy the page Thread writes to page . . -> no page fault . update the PTE . flush TLB for that PTE flush TLB PTE are now rdonly So the write done by the CPU 3 is interfering with the page copy operation done by CPU 2, leading to the data corruption. To avoid this we mark all the VMA involved in the COW mechanism as changing by calling vm_write_begin(). This ensures that the speculative page fault handler will not try to handle a fault on these pages. The marker is set until the TLB is flushed, ensuring that all the CPUs will now see the PTE as not writable. Once the TLB is flush, the marker is removed by calling vm_write_end(). The variable last is used to keep tracked of the latest VMA marked to handle the error path where part of the VMA may have been marked. Since multiple VMA from the same mm may have the sequence count increased during this process, the use of the vm_raw_write_begin/end() is required to avoid lockdep false warning messages. Reported-by: Ganesh Mahendran Reported-by: Vinayak Menon Signed-off-by: Laurent Dufour Reviewed-by: Jérome Glisse --- kernel/fork.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index f8dae021c2e5..2992d2c95256 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -462,7 +462,7 @@ EXPORT_SYMBOL(free_task); static __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm) { - struct vm_area_struct *mpnt, *tmp, *prev, **pprev; + struct vm_area_struct *mpnt, *tmp, *prev, **pprev, *last = NULL; struct rb_node **rb_link, *rb_parent; int retval; unsigned long charge; @@ -581,8 +581,18 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, rb_parent = &tmp->vm_rb; mm->map_count++; - if (!(tmp->vm_flags & VM_WIPEONFORK)) + if (!(tmp->vm_flags & VM_WIPEONFORK)) { + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) { + /* + * Mark this VMA as changing to prevent the + * speculative page fault hanlder to process + * it until the TLB are flushed below. + */ + last = mpnt; + vm_raw_write_begin(mpnt); + } retval = copy_page_range(mm, oldmm, mpnt); + } if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); @@ -595,6 +605,22 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, out: up_write(&mm->mmap_sem); flush_tlb_mm(oldmm); + + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) { + /* + * Since the TLB has been flush, we can safely unmark the + * copied VMAs and allows the speculative page fault handler to + * process them again. + * Walk back the VMA list from the last marked VMA. + */ + for (; last; last = last->vm_prev) { + if (last->vm_flags & VM_DONTCOPY) + continue; + if (!(last->vm_flags & VM_WIPEONFORK)) + vm_raw_write_end(last); + } + } + up_write(&oldmm->mmap_sem); dup_userfaultfd_complete(&uf); fail_uprobe_end: From patchwork Tue Apr 16 13:45:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903139 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B570517E1 for ; Tue, 16 Apr 2019 13:47:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99DA9286B9 for ; Tue, 16 Apr 2019 13:47:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8D8CB286CF; Tue, 16 Apr 2019 13:47:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CED9A288CF for ; Tue, 16 Apr 2019 13:47:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2468C6B0010; Tue, 16 Apr 2019 09:46:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1FA276B0266; Tue, 16 Apr 2019 09:46:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 097056B0269; Tue, 16 Apr 2019 09:46:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id A75F16B0010 for ; Tue, 16 Apr 2019 09:46:53 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id s21so3585762edd.10 for ; Tue, 16 Apr 2019 06:46:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=ZfEwFFloPr/8tc9stcBjbN7acEwKiBqAEeq/s4rwt0o=; b=ibn4q/q4rmQMD4PihVcnCsBFpaWJCAmkwyRi18D9N8hR1Ma/leGcZ8FEeXVyRAHirA dRbMFgevD0v7jrn7b1W8FcfjJUCzB5M+PQolcgl/KH/y8thtUTOwugUKGE8t0qem0Gs9 ZMRg+qkMoigHkgI7tWsAh71tIrSrk1yY3DoDeJCE6iN0kkHrriFTv4wxsrPZIaXiUr8U 9bbA4V3hJL6GNoIqqZRPOjTV66NazN7dn4GjyMq9MBlpGgw6/SnC2r0qADcdJn8sCnxg mHYYQn7Anxr2c7t5u1WmyDhgEeCIKazoxorrEN51HGjUckQgf5W713LCvnhVdhbTu8B+ Sw9A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAWwGk36m4TM4udgHJevRByAIDDKTTlQ2sqN0EfhMNzJ31NeQlkl +coPl3dO0pFltl+SSfTI++12LzntRiexlbsGf2j978VNT7J/j02HoKel2UKbqIYiVrdDEqpJWLR WXxDi1l3TVCugbaPIq32nZsNw2R5pJSy7lmreWRJBfolrTH3kuuPm6tEkxiD727jNvg== X-Received: by 2002:a17:906:4d4d:: with SMTP id b13mr44319316ejv.256.1555422413042; Tue, 16 Apr 2019 06:46:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqxFka9u/Og2CyLE+DtvOB8FpNiv96H3w4T5wwocLouhKUX+Irbt4O2thfEL9D2iwpMPrS8W X-Received: by 2002:a17:906:4d4d:: with SMTP id b13mr44319244ejv.256.1555422411607; Tue, 16 Apr 2019 06:46:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422411; cv=none; d=google.com; s=arc-20160816; b=R1MDQolsus8PdvGk3oaImHlzV53CfaRZtK1iX3eRlO+isLGc/y1J0bdjFivhr6ei5S zOrA8D7AsyZoyquHw5sm2P20J6PNR8cWdrjCqofcoEaEFTv2iZcmDYWU4VbIxKlvXtNC HZWs4A9We54u6UhmyUsc6K1Xc3m5CQo4EXVzNhOdKz38Qun7fO+uMJgpR2vp279Mp9k4 5lMNpflFSVkOKwdD5nKcPdeQ+EIs6x/fRgmO7HeFJStgp3kzEgHCO59ok73ANPkQ6+Bl 6OC86G8b06e7NTuzmwQTNyROWgdEEQdguQmdfgCQm3vsDERiu1GE8+SOMvkKpz2Ca1oe P1ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=ZfEwFFloPr/8tc9stcBjbN7acEwKiBqAEeq/s4rwt0o=; b=MQM4Cd7P0/H9VKt98aL3ivUx68EwMlafgoE6JvB5mohVkEG/nutfolI6h42kPbmBZj 3/e2pIk/0EytjUv6gOZo+qsaKYGJN5pe4sbjhtk8BDC8e4N1bL7fDjUIdQUqicc/Zyku SVNHdf/bPfR9J32i0ZWqVYVm16PCJb7LRkBqOiZldJ6jpSKWba4Ar1Onb+biQDOEjXdq GklJIHbvS6zGJQMbjzx7FuLUGKeCRKqwFLdJbEbIuHb024XWEuAIEFH/EIPsm3J6CQTt 5yVPBAkrTKWhRJMBYfnjBo/6rRdSCIMWts9zQsum5oJ0LYZH5aGL2Y/k9A1kSjtCG8CX bTxw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id u8si1303533ejt.152.2019.04.16.06.46.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:51 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkJOD129673 for ; Tue, 16 Apr 2019 09:46:49 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe6cnpn8-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:47 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:31 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:22 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkLXG49545466 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:21 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 06E084C044; Tue, 16 Apr 2019 13:46:21 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8661E4C040; Tue, 16 Apr 2019 13:46:18 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:18 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 19/31] mm: protect the RB tree with a sequence lock Date: Tue, 16 Apr 2019 15:45:10 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7277 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD8F Message-Id: <20190416134522.17540-20-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Introducing a per mm_struct seqlock, mm_seq field, to protect the changes made in the MM RB tree. This allows to walk the RB tree without grabbing the mmap_sem, and on the walk is done to double check that sequence counter was stable during the walk. The mm seqlock is held while inserting and removing entries into the MM RB tree. Later in this series, it will be check when looking for a VMA without holding the mmap_sem. This is based on the initial work from Peter Zijlstra: https://lore.kernel.org/linux-mm/20100104182813.479668508@chello.nl/ Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm_types.h | 3 +++ kernel/fork.c | 3 +++ mm/init-mm.c | 3 +++ mm/mmap.c | 48 +++++++++++++++++++++++++++++++--------- 4 files changed, 46 insertions(+), 11 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e78f72eb2576..24b3f8ce9e42 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -358,6 +358,9 @@ struct mm_struct { struct { struct vm_area_struct *mmap; /* list of VMAs */ struct rb_root mm_rb; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + seqlock_t mm_seq; +#endif u64 vmacache_seqnum; /* per-thread vmacache */ #ifdef CONFIG_MMU unsigned long (*get_unmapped_area) (struct file *filp, diff --git a/kernel/fork.c b/kernel/fork.c index 2992d2c95256..3a1739197ebc 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1008,6 +1008,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->mmap = NULL; mm->mm_rb = RB_ROOT; mm->vmacache_seqnum = 0; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + seqlock_init(&mm->mm_seq); +#endif atomic_set(&mm->mm_users, 1); atomic_set(&mm->mm_count, 1); init_rwsem(&mm->mmap_sem); diff --git a/mm/init-mm.c b/mm/init-mm.c index a787a319211e..69346b883a4e 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -27,6 +27,9 @@ */ struct mm_struct init_mm = { .mm_rb = RB_ROOT, +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + .mm_seq = __SEQLOCK_UNLOCKED(init_mm.mm_seq), +#endif .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), diff --git a/mm/mmap.c b/mm/mmap.c index 13460b38b0fb..f7f6027a7dff 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -170,6 +170,24 @@ void unlink_file_vma(struct vm_area_struct *vma) } } +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static inline void mm_write_seqlock(struct mm_struct *mm) +{ + write_seqlock(&mm->mm_seq); +} +static inline void mm_write_sequnlock(struct mm_struct *mm) +{ + write_sequnlock(&mm->mm_seq); +} +#else +static inline void mm_write_seqlock(struct mm_struct *mm) +{ +} +static inline void mm_write_sequnlock(struct mm_struct *mm) +{ +} +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* * Close a vm structure and free it, returning the next. */ @@ -445,26 +463,32 @@ static void vma_gap_update(struct vm_area_struct *vma) } static inline void vma_rb_insert(struct vm_area_struct *vma, - struct rb_root *root) + struct mm_struct *mm) { + struct rb_root *root = &mm->mm_rb; + /* All rb_subtree_gap values must be consistent prior to insertion */ validate_mm_rb(root, NULL); rb_insert_augmented(&vma->vm_rb, root, &vma_gap_callbacks); } -static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root) +static void __vma_rb_erase(struct vm_area_struct *vma, struct mm_struct *mm) { + struct rb_root *root = &mm->mm_rb; + /* * Note rb_erase_augmented is a fairly large inline function, * so make sure we instantiate it only once with our desired * augmented rbtree callbacks. */ + mm_write_seqlock(mm); rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); + mm_write_sequnlock(mm); /* wmb */ } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, - struct rb_root *root, + struct mm_struct *mm, struct vm_area_struct *ignore) { /* @@ -472,21 +496,21 @@ static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, * with the possible exception of the "next" vma being erased if * next->vm_start was reduced. */ - validate_mm_rb(root, ignore); + validate_mm_rb(&mm->mm_rb, ignore); - __vma_rb_erase(vma, root); + __vma_rb_erase(vma, mm); } static __always_inline void vma_rb_erase(struct vm_area_struct *vma, - struct rb_root *root) + struct mm_struct *mm) { /* * All rb_subtree_gap values must be consistent prior to erase, * with the possible exception of the vma being erased. */ - validate_mm_rb(root, vma); + validate_mm_rb(&mm->mm_rb, vma); - __vma_rb_erase(vma, root); + __vma_rb_erase(vma, mm); } /* @@ -601,10 +625,12 @@ void __vma_link_rb(struct mm_struct *mm, struct vm_area_struct *vma, * immediately update the gap to the correct value. Finally we * rebalance the rbtree after all augmented values have been set. */ + mm_write_seqlock(mm); rb_link_node(&vma->vm_rb, rb_parent, rb_link); vma->rb_subtree_gap = 0; vma_gap_update(vma); - vma_rb_insert(vma, &mm->mm_rb); + vma_rb_insert(vma, mm); + mm_write_sequnlock(mm); } static void __vma_link_file(struct vm_area_struct *vma) @@ -680,7 +706,7 @@ static __always_inline void __vma_unlink_common(struct mm_struct *mm, { struct vm_area_struct *next; - vma_rb_erase_ignore(vma, &mm->mm_rb, ignore); + vma_rb_erase_ignore(vma, mm, ignore); next = vma->vm_next; if (has_prev) prev->vm_next = next; @@ -2674,7 +2700,7 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma, insertion_point = (prev ? &prev->vm_next : &mm->mmap); vma->vm_prev = NULL; do { - vma_rb_erase(vma, &mm->mm_rb); + vma_rb_erase(vma, mm); mm->map_count--; tail_vma = vma; vma = vma->vm_next; From patchwork Tue Apr 16 13:45:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903149 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C4DD17E1 for ; Tue, 16 Apr 2019 13:47:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B90B28947 for ; Tue, 16 Apr 2019 13:47:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5FA65288DB; Tue, 16 Apr 2019 13:47:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B463C289F9 for ; Tue, 16 Apr 2019 13:47:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EAF56B026C; Tue, 16 Apr 2019 09:46:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C5D26B026D; Tue, 16 Apr 2019 09:46:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68DA36B026E; Tue, 16 Apr 2019 09:46:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id 3F00E6B026C for ; Tue, 16 Apr 2019 09:46:59 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id x8so15735201ybp.14 for ; Tue, 16 Apr 2019 06:46:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=ubCpXHvc62vaD/IdobGXFa/dMTQCWn60cru2hVIVF7s=; b=Y/3lXKnO8QNxvUVQvoHAkc4JSrrcIALXynU567s3z+Jpz0WxwFHPZ9WsfzAQ8Qdqcf o5HTqi9j5smfngBVCqOGCYC8rRDsCb2e4mAivxaiobUHUKEynj1j0ReZDFm5AxAj6ZP5 bDq4pyoIsFObWWtNRVWsbFmCfy0NGZIN17JxOLbcnzr4JgBq/9EcVmIJPvwmOM1MvIsH Gy8csFmbwT67BZxUzHE0kXocczxqs4J5Pi70Jks+cZGr4v+dkSgj5AzKfp2IjiqixMYW wR6ZYkORjCFD+PRwdZf+ml03FZ/GBQ9Rsjgku8JPXCOkDcxzSqE3VQDr5aUcHV63FaAX 4OmA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAWUPG8b35mS9qYVlaWx6cAyBaU5XyOcpx7BWvQuWbtTT5xIKtY9 XqMgrWjRTj0EoieWJk78tgP8z8j0Tgv4jTMVt3+7Il2sfPxLIEkKzJ0nSLVGe5ukDmLJzRRBoiV J4iO/o8/hFa74If9jGDt31GNnX7L2NEEto4rnS3UQpgwICJDaOHqf9DiwYxUtXOrwpw== X-Received: by 2002:a81:350d:: with SMTP id c13mr65516074ywa.242.1555422418977; Tue, 16 Apr 2019 06:46:58 -0700 (PDT) X-Google-Smtp-Source: APXvYqywNCpiidF4uOWfVbIIPYT71ca4vcSXHnGP+nku0fb2oWhSb6rAyngXA/+S7rwyH2ATqqlb X-Received: by 2002:a81:350d:: with SMTP id c13mr65515950ywa.242.1555422417731; Tue, 16 Apr 2019 06:46:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422417; cv=none; d=google.com; s=arc-20160816; b=CeyJSk+DvMy8xDeg+AIGlpVE8MyAUQi8lP0uQapuGu0RJOynL4LNQ2RMpMituwCH+S ZZL02eqe4/xxeLpaEz2RfeHy9QwFl43lA5km3t0X3wXVEBiBAEiA1Q9qNWd6UcqHQ5kG w2Q40RqZSeF2KE+I5p6qyOOCB+iDDVw36Rzd0o5QFaVT1RsPjsaX8zZxF84p4oUG605e kG77BoWtGCkYZpmaLiCtkCCTxojHoocT+xZdO8fIj5tDcVA2gTzyao5TQXIQa7eWZXAX 5sDadVutypOYqV/ylAosHOhkbP3jREwEiLT3tKnaQvj/FHsZ7LhdI4JNdK3PXGFToL7k M5qA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=ubCpXHvc62vaD/IdobGXFa/dMTQCWn60cru2hVIVF7s=; b=HZV3GXnHcgh8NlNpgpRPrf3e/GZYLGQkVyEWO5ARFfTvSSXXsmrpI8/a8vISXGHXfI U60Xq+KyBj7jfOZ2a2NO9yuMceBekIAqsziGdRu2hX1MhBLSuFEWYbPHgcSnwriIMlvK 66v0T3SjbOq8VMLaERKPojwG8NHE9o3ptTUxR4otB/EqydaZ8txeksWlvl0UbIf9j7pc TUHX1fGVZdqrb8vT4qXbeDO1SEV9AUITE/ZQ9x6eQZOe2D7HvOOYyBrPLlF4v1bFcN1t n3599gJazgDLmIDaNvq2o9s0J16fejphSfdlKoKclYsUpw0bPqEEEcZOm8vo5v9uHmVH vqLw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id 143si17972571ywi.67.2019.04.16.06.46.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:57 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkWx4113210 for ; Tue, 16 Apr 2019 09:46:56 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwentmua0-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:54 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:34 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:24 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkMSN40894698 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:22 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 983514C040; Tue, 16 Apr 2019 13:46:22 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 326344C044; Tue, 16 Apr 2019 13:46:21 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:21 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 20/31] mm: introduce vma reference counter Date: Tue, 16 Apr 2019 15:45:11 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-4275-0000-0000-000003287655 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-4276-0000-0000-00003837A75C Message-Id: <20190416134522.17540-21-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The final goal is to be able to use a VMA structure without holding the mmap_sem and to be sure that the structure will not be freed in our back. The lockless use of the VMA will be done through RCU protection and thus a dedicated freeing service is required to manage it asynchronously. As reported in a 2010's thread [1], this may impact file handling when a file is still referenced while the mapping is no more there. As the final goal is to handle anonymous VMA in a speculative way and not file backed mapping, we could close and free the file pointer in a synchronous way, as soon as we are guaranteed to not use it without holding the mmap_sem. For sanity reason, in a minimal effort, the vm_file file pointer is unset once the file pointer is put. [1] https://lore.kernel.org/linux-mm/20100104182429.833180340@chello.nl/ Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm.h | 4 ++++ include/linux/mm_types.h | 3 +++ mm/internal.h | 27 +++++++++++++++++++++++++++ mm/mmap.c | 13 +++++++++---- 4 files changed, 43 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f14b2c9ddfd4..f761a9c65c74 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -529,6 +529,9 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + atomic_set(&vma->vm_ref_count, 1); +#endif } static inline void vma_set_anonymous(struct vm_area_struct *vma) @@ -1418,6 +1421,7 @@ static inline void INIT_VMA(struct vm_area_struct *vma) INIT_LIST_HEAD(&vma->anon_vma_chain); #ifdef CONFIG_SPECULATIVE_PAGE_FAULT seqcount_init(&vma->vm_sequence); + atomic_set(&vma->vm_ref_count, 1); #endif } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 24b3f8ce9e42..6a6159e11a3f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -285,6 +285,9 @@ struct vm_area_struct { /* linked list of VM areas per task, sorted by address */ struct vm_area_struct *vm_next, *vm_prev; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + atomic_t vm_ref_count; +#endif struct rb_node vm_rb; /* diff --git a/mm/internal.h b/mm/internal.h index 9eeaf2b95166..302382bed406 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -40,6 +40,33 @@ void page_writeback_init(void); vm_fault_t do_swap_page(struct vm_fault *vmf); + +extern void __free_vma(struct vm_area_struct *vma); + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static inline void get_vma(struct vm_area_struct *vma) +{ + atomic_inc(&vma->vm_ref_count); +} + +static inline void put_vma(struct vm_area_struct *vma) +{ + if (atomic_dec_and_test(&vma->vm_ref_count)) + __free_vma(vma); +} + +#else + +static inline void get_vma(struct vm_area_struct *vma) +{ +} + +static inline void put_vma(struct vm_area_struct *vma) +{ + __free_vma(vma); +} +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma, unsigned long floor, unsigned long ceiling); diff --git a/mm/mmap.c b/mm/mmap.c index f7f6027a7dff..c106440dcae7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -188,6 +188,12 @@ static inline void mm_write_sequnlock(struct mm_struct *mm) } #endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ +void __free_vma(struct vm_area_struct *vma) +{ + mpol_put(vma_policy(vma)); + vm_area_free(vma); +} + /* * Close a vm structure and free it, returning the next. */ @@ -200,8 +206,8 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) vma->vm_ops->close(vma); if (vma->vm_file) fput(vma->vm_file); - mpol_put(vma_policy(vma)); - vm_area_free(vma); + vma->vm_file = NULL; + put_vma(vma); return next; } @@ -990,8 +996,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (next->anon_vma) anon_vma_merge(vma, next); mm->map_count--; - mpol_put(vma_policy(next)); - vm_area_free(next); + put_vma(next); /* * In mprotect's case 6 (see comments on vma_merge), * we must remove another next too. It would clutter From patchwork Tue Apr 16 13:45:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903141 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1A550161F for ; Tue, 16 Apr 2019 13:47:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 024DC28650 for ; Tue, 16 Apr 2019 13:47:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E8E7E28A08; Tue, 16 Apr 2019 13:47:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2DC2128650 for ; Tue, 16 Apr 2019 13:47:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 504916B0269; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4B57B6B0266; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E4756B026B; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id CA9946B0266 for ; Tue, 16 Apr 2019 09:46:54 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id p90so10846365edp.11 for ; Tue, 16 Apr 2019 06:46:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=OckpsbzGyYbhhd2ScW+lfhM+L5+Xl4cLJVeT4TNmxBw=; b=FY7PexDUud8eGryDyZHwgwIOEJIEVMrE9wIrtWHssfCj94Eb89avkMi+kkr7OgJ66E iico7d26ZswN/fpG/Qd5UyNGppL7VutTS3ysSn1jKspj3QhfshekQ6vuXShRRuWwDtKx wXg7+NLKyXUlNLNq25tkDAl2A/zcMLFY5gllK9+2JxkVWUgiP3+3Vq2OolCxfEIV6hMy hBtAdrkKBuh+Ge8gKoOtp+o0wykgDntB1BekFKUElo2ogRgSC9yELTvatPj3732iM+yW 3+5OJ87JRF3Q1jLHnyjUwucCYbE68H/LinC2gu5JayoKEo2xdTrHq/ZZQRWs8cRcsomt pk/w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAV2BMGAZ5G7165O4duhhNB3FJU6cXeJBrD5VwiUtO4QDUNIOgD7 hm4c3k5UCJJc6YMkIv7QUgwsma0NB+FznpliEtd3cco2GnP4F5MspCJFE1rUDoS1KSfVRAoD5qB LVk+9oCJ+A5YbPQ87lF3XzNgE6vqI18X9KZ5LGpEORCGtM4bJAytM64FSYvpH5sj83g== X-Received: by 2002:a17:906:1906:: with SMTP id a6mr44062901eje.236.1555422414277; Tue, 16 Apr 2019 06:46:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqyk7Se6H0w8KDC+hSPywKPsg2QItz2UqrS82mCEStbXzqe/mna0yk2NjrXTAg/I3alhUEUZ X-Received: by 2002:a17:906:1906:: with SMTP id a6mr44062798eje.236.1555422412408; Tue, 16 Apr 2019 06:46:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422412; cv=none; d=google.com; s=arc-20160816; b=cdTH9/xKMIoTV1ATXODXs9KZGadtWM0+9v4z4TgQMlDDzMY8/mIuglH/cgPvDtpoZ9 S5Y7v+ClbtFHLIedR5m1A8iQASPHMbvNL5MDUT5Sx+TCh8EBs0LQDJ+lgaPBBbdTUf4M 4+OYrAnhw7pYdgQiq5UJLh7oE6OxPrgamNuYvv7SSeai4vsng0cFCImvE6ERbAr5w68G ESF4ZMCZjLPnTX6+ERBcdBf2IumI4AklOxBJScNMokETm8UuhJ2GvZRf23jr/n3LSfSr 2NNUcBjZO7K4r7Qa7DzZqRucF2glUaEXSAd8oWbT4ygISmPtnS/Ovx08F5SatTC21qJl IkfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=OckpsbzGyYbhhd2ScW+lfhM+L5+Xl4cLJVeT4TNmxBw=; b=OGPMjsVwktL4IyO/RRqLZbUpkbBcxMkZVIrSsMiyGWP1AqaGUf5djS4tNBiaxRCov4 zAnoACepXXb0PHfy1IUWGcAHM0e8cWs7cFAF8tl+LfspavvaBtQvBOHWlJ0TXaMNOHNi eMh8VJSAm3JVub/3MRe7URORJHbKsvUypebqqVr4iRuwpok667yhdJ0cuWp040dMUjoz ZSVxRrKgbqQu5L+nalMM34oBObBHy8tKAoDZAP8mDEDVLk+hSyx3fsKYZkfuDtK3Zgi4 oybThLTWLgtE58QN2Eb5PTWT5ox9XWuRk4eDkyP3GWgfXtllfrrLGEMQjPdQCLWcF/5h 60/Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id c15si1855425edy.208.2019.04.16.06.46.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:52 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkRJx100074 for ; Tue, 16 Apr 2019 09:46:51 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwdvq6yv5-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:48 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:36 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:26 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkPQA33620156 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:25 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 19B424C040; Tue, 16 Apr 2019 13:46:25 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A7D134C046; Tue, 16 Apr 2019 13:46:23 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:23 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 21/31] mm: Introduce find_vma_rcu() Date: Tue, 16 Apr 2019 15:45:12 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-4275-0000-0000-000003287656 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-4276-0000-0000-00003837A75D Message-Id: <20190416134522.17540-22-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=958 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This allows to search for a VMA structure without holding the mmap_sem. The search is repeated while the mm seqlock is changing and until we found a valid VMA. While under the RCU protection, a reference is taken on the VMA, so the caller must call put_vma() once it not more need the VMA structure. At the time a VMA is inserted in the MM RB tree, in vma_rb_insert(), a reference is taken to the VMA by calling get_vma(). When removing a VMA from the MM RB tree, the VMA is not release immediately but at the end of the RCU grace period through vm_rcu_put(). This ensures that the VMA remains allocated until the end the RCU grace period. Since the vm_file pointer, if valid, is released in put_vma(), there is no guarantee that the file pointer will be valid on the returned VMA. Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- include/linux/mm_types.h | 1 + mm/internal.h | 5 ++- mm/mmap.c | 76 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 78 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 6a6159e11a3f..9af6694cb95d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -287,6 +287,7 @@ struct vm_area_struct { #ifdef CONFIG_SPECULATIVE_PAGE_FAULT atomic_t vm_ref_count; + struct rcu_head vm_rcu; #endif struct rb_node vm_rb; diff --git a/mm/internal.h b/mm/internal.h index 302382bed406..1e368e4afe3c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -55,7 +55,10 @@ static inline void put_vma(struct vm_area_struct *vma) __free_vma(vma); } -#else +extern struct vm_area_struct *find_vma_rcu(struct mm_struct *mm, + unsigned long addr); + +#else /* CONFIG_SPECULATIVE_PAGE_FAULT */ static inline void get_vma(struct vm_area_struct *vma) { diff --git a/mm/mmap.c b/mm/mmap.c index c106440dcae7..34bf261dc2c8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -179,6 +179,18 @@ static inline void mm_write_sequnlock(struct mm_struct *mm) { write_sequnlock(&mm->mm_seq); } + +static void __vm_rcu_put(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + put_vma(vma); +} +static void vm_rcu_put(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(!RB_EMPTY_NODE(&vma->vm_rb), vma); + call_rcu(&vma->vm_rcu, __vm_rcu_put); +} #else static inline void mm_write_seqlock(struct mm_struct *mm) { @@ -190,6 +202,8 @@ static inline void mm_write_sequnlock(struct mm_struct *mm) void __free_vma(struct vm_area_struct *vma) { + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) + VM_BUG_ON_VMA(!RB_EMPTY_NODE(&vma->vm_rb), vma); mpol_put(vma_policy(vma)); vm_area_free(vma); } @@ -197,11 +211,24 @@ void __free_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it, returning the next. */ -static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) +static struct vm_area_struct *__remove_vma(struct vm_area_struct *vma) { struct vm_area_struct *next = vma->vm_next; might_sleep(); + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT) && + !RB_EMPTY_NODE(&vma->vm_rb)) { + /* + * If the VMA is still linked in the RB tree, we must release + * that reference by calling put_vma(). + * This should only happen when called from exit_mmap(). + * We forcely clear the node to satisfy the chec in + * __free_vma(). This is safe since the RB tree is not walked + * anymore. + */ + RB_CLEAR_NODE(&vma->vm_rb); + put_vma(vma); + } if (vma->vm_ops && vma->vm_ops->close) vma->vm_ops->close(vma); if (vma->vm_file) @@ -211,6 +238,13 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) return next; } +static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) +{ + if (IS_ENABLED(CONFIG_SPECULATIVE_PAGE_FAULT)) + VM_BUG_ON_VMA(!RB_EMPTY_NODE(&vma->vm_rb), vma); + return __remove_vma(vma); +} + static int do_brk_flags(unsigned long addr, unsigned long request, unsigned long flags, struct list_head *uf); SYSCALL_DEFINE1(brk, unsigned long, brk) @@ -475,7 +509,7 @@ static inline void vma_rb_insert(struct vm_area_struct *vma, /* All rb_subtree_gap values must be consistent prior to insertion */ validate_mm_rb(root, NULL); - + get_vma(vma); rb_insert_augmented(&vma->vm_rb, root, &vma_gap_callbacks); } @@ -491,6 +525,14 @@ static void __vma_rb_erase(struct vm_area_struct *vma, struct mm_struct *mm) mm_write_seqlock(mm); rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks); mm_write_sequnlock(mm); /* wmb */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + /* + * Ensure the removal is complete before clearing the node. + * Matched by vma_has_changed()/handle_speculative_fault(). + */ + RB_CLEAR_NODE(&vma->vm_rb); + vm_rcu_put(vma); +#endif } static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma, @@ -2331,6 +2373,34 @@ struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr) EXPORT_SYMBOL(find_vma); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +/* + * Like find_vma() but under the protection of RCU and the mm sequence counter. + * The vma returned has to be relaesed by the caller through the call to + * put_vma() + */ +struct vm_area_struct *find_vma_rcu(struct mm_struct *mm, unsigned long addr) +{ + struct vm_area_struct *vma = NULL; + unsigned int seq; + + do { + if (vma) + put_vma(vma); + + seq = read_seqbegin(&mm->mm_seq); + + rcu_read_lock(); + vma = find_vma(mm, addr); + if (vma) + get_vma(vma); + rcu_read_unlock(); + } while (read_seqretry(&mm->mm_seq, seq)); + + return vma; +} +#endif + /* * Same as find_vma, but also return a pointer to the previous VMA in *pprev. */ @@ -3231,7 +3301,7 @@ void exit_mmap(struct mm_struct *mm) while (vma) { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - vma = remove_vma(vma); + vma = __remove_vma(vma); } vm_unacct_memory(nr_accounted); } From patchwork Tue Apr 16 13:45:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903151 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84751161F for ; Tue, 16 Apr 2019 13:47:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F94828947 for ; Tue, 16 Apr 2019 13:47:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 52E66289F9; Tue, 16 Apr 2019 13:47:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9EDDE286E2 for ; Tue, 16 Apr 2019 13:47:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F7D06B026D; Tue, 16 Apr 2019 09:47:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1A8DD6B026E; Tue, 16 Apr 2019 09:47:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0755B6B026F; Tue, 16 Apr 2019 09:47:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f199.google.com (mail-yb1-f199.google.com [209.85.219.199]) by kanga.kvack.org (Postfix) with ESMTP id CED106B026D for ; Tue, 16 Apr 2019 09:47:00 -0400 (EDT) Received: by mail-yb1-f199.google.com with SMTP id 204so15703107ybf.5 for ; Tue, 16 Apr 2019 06:47:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=TiaW3nQS0in308DNydmwNBVevWj1nVl3s3LJYDlW8UY=; b=Fiyq6hXziFdIcK/8H/Y99XHQA9qDl25Q3zhhuYM4ZC2qQqy9mTuqo+jrYK38Bgobf5 3T0lzDb4iSI1/LBw87KNnpLk20/klRe/h3k+78Uqa0AKJVebiCab/cEwJsNnnux8qOM5 hb2DAwfisejSgu5nnWTPpakT4If9aAMTqh0oGYcx5dO56wX6vgc+Bk59sHlmfJQ4rbS8 aIzRyAmOf+ZN3/i7Wj6GRjPPd9b+2VRk+GR0cT1MjdEctlbaU5BNua4ThfznT2l4SSql 9Y0ia9N/1+PXQe7VArm8VFSl5qS2FMl6H9h31Om2pkShLIBQztcAxznJi6Nv1xTMqDmm bxkQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAW/KpaYjuX97avRs2XR4v0ZP0SNWN13Sq7xmJVQDMOXljAxlkco TS1omLG4hq8gspLFwLMPV/UfCgeT0bzv5MF1IWKl2GwhkPUgavwEJIIqoh+R25UNpiMKWTytP+c ZJ9GsitotmCXaDp0AZjfwROzhQzbz+Aj+Td9QLjVGKHmMJPSDIYzpx92ZU8NY9Kh+Bw== X-Received: by 2002:a25:bc0a:: with SMTP id i10mr50946296ybh.121.1555422420477; Tue, 16 Apr 2019 06:47:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqwmtLFVfRpe9OPy/MTrJI41mfc1zHARKAQF+2vNFTroyufQYPh5+52z7AceuliExXz+Jvjx X-Received: by 2002:a25:bc0a:: with SMTP id i10mr50946151ybh.121.1555422418749; Tue, 16 Apr 2019 06:46:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422418; cv=none; d=google.com; s=arc-20160816; b=Xw1eOkZ/koWbf5idKuIP2byZJoyz0MeINkY2o8aNYNbnQgkpHSoASav45gfuMQBaUf bENaSi8dagDmeiyQf1jCc7sn5iAySyMNDpLxlH5v+el57yc9TVMPqTaN7R+YXzfq/fiT KXRNjcPcM6ygK7lcABTfMPyFwc7zj2f3NKtMmb7pGHzqJue/YMScUW0Z6hQUVc3XCQhN LMAi9wg+11gYo/bgjhogadFRyFXSteBn+Umv8kIEipYcy3EMyz2HE2DtBL7AhjOqBmLT mtXMjOF8Ebi5Ld/Pz63RA1sVpx31i+faJSuKwywOvvf+ySk0g+p9ncH9R4b3E/f+rnnm aj0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=TiaW3nQS0in308DNydmwNBVevWj1nVl3s3LJYDlW8UY=; b=TTafiG8r8oku7lql88Zot3pLjZyS/7WajfuOItCgL1urhDqa2j6hSmZVIKQJxmHvF8 gfkhwR2fL8IYRJukKBomYwbAlqDZO8VDtb5yBzzjpD3NP+rTaTvo53W7hT1T8rf7bczF q2GLEMP56Fd1ymzbFi3BAdHdJ2fmgfYcL42BT3fMHQ/IKkjfs/joukdZZmSOgZrVjJYb T+3Ba5be/QGWZLUZIJKtQ9gRvoG6UWLmcK2DR4GH4knN9BvMbicyrbIDdJ+MTW/PH7y3 IToDUIIzEOgNr8mInYDkvo367xsWejgcppTF5m3El3YLM1ULBoxrEoDoQc1o/shtnH7P HpTA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id i62si34253444ybi.399.2019.04.16.06.46.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:58 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkUfj113100 for ; Tue, 16 Apr 2019 09:46:57 -0400 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwentmufy-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:51 -0400 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:40 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:30 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkTOR38535316 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:29 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0090B4C040; Tue, 16 Apr 2019 13:46:29 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3894F4C04A; Tue, 16 Apr 2019 13:46:26 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:26 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 22/31] mm: provide speculative fault infrastructure Date: Tue, 16 Apr 2019 15:45:13 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0012-0000-0000-0000030F6F04 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0013-0000-0000-00002147A864 Message-Id: <20190416134522.17540-23-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Peter Zijlstra Provide infrastructure to do a speculative fault (not holding mmap_sem). The not holding of mmap_sem means we can race against VMA change/removal and page-table destruction. We use the SRCU VMA freeing to keep the VMA around. We use the VMA seqcount to detect change (including umapping / page-table deletion) and we use gup_fast() style page-table walking to deal with page-table races. Once we've obtained the page and are ready to update the PTE, we validate if the state we started the fault with is still valid, if not, we'll fail the fault with VM_FAULT_RETRY, otherwise we update the PTE and we're done. Signed-off-by: Peter Zijlstra (Intel) [Manage the newly introduced pte_spinlock() for speculative page fault to fail if the VMA is touched in our back] [Rename vma_is_dead() to vma_has_changed() and declare it here] [Fetch p4d and pud] [Set vmd.sequence in __handle_mm_fault()] [Abort speculative path when handle_userfault() has to be called] [Add additional VMA's flags checks in handle_speculative_fault()] [Clear FAULT_FLAG_ALLOW_RETRY in handle_speculative_fault()] [Don't set vmf->pte and vmf->ptl if pte_map_lock() failed] [Remove warning comment about waiting for !seq&1 since we don't want to wait] [Remove warning about no huge page support, mention it explictly] [Don't call do_fault() in the speculative path as __do_fault() calls vma->vm_ops->fault() which may want to release mmap_sem] [Only vm_fault pointer argument for vma_has_changed()] [Fix check against huge page, calling pmd_trans_huge()] [Use READ_ONCE() when reading VMA's fields in the speculative path] [Explicitly check for __HAVE_ARCH_PTE_SPECIAL as we can't support for processing done in vm_normal_page()] [Check that vma->anon_vma is already set when starting the speculative path] [Check for memory policy as we can't support MPOL_INTERLEAVE case due to the processing done in mpol_misplaced()] [Don't support VMA growing up or down] [Move check on vm_sequence just before calling handle_pte_fault()] [Don't build SPF services if !CONFIG_SPECULATIVE_PAGE_FAULT] [Add mem cgroup oom check] [Use READ_ONCE to access p*d entries] [Replace deprecated ACCESS_ONCE() by READ_ONCE() in vma_has_changed()] [Don't fetch pte again in handle_pte_fault() when running the speculative path] [Check PMD against concurrent collapsing operation] [Try spin lock the pte during the speculative path to avoid deadlock with other CPU's invalidating the TLB and requiring this CPU to catch the inter processor's interrupt] [Move define of FAULT_FLAG_SPECULATIVE here] [Introduce __handle_speculative_fault() and add a check against mm->mm_users in handle_speculative_fault() defined in mm.h] [Abort if vm_ops->fault is set instead of checking only vm_ops] [Use find_vma_rcu() and call put_vma() when we are done with the VMA] Signed-off-by: Laurent Dufour --- include/linux/hugetlb_inline.h | 2 +- include/linux/mm.h | 30 +++ include/linux/pagemap.h | 4 +- mm/internal.h | 15 ++ mm/memory.c | 344 ++++++++++++++++++++++++++++++++- 5 files changed, 389 insertions(+), 6 deletions(-) diff --git a/include/linux/hugetlb_inline.h b/include/linux/hugetlb_inline.h index 0660a03d37d9..9e25283d6fc9 100644 --- a/include/linux/hugetlb_inline.h +++ b/include/linux/hugetlb_inline.h @@ -8,7 +8,7 @@ static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma) { - return !!(vma->vm_flags & VM_HUGETLB); + return !!(READ_ONCE(vma->vm_flags) & VM_HUGETLB); } #else diff --git a/include/linux/mm.h b/include/linux/mm.h index f761a9c65c74..ec609cbad25a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -381,6 +381,7 @@ extern pgprot_t protection_map[16]; #define FAULT_FLAG_USER 0x40 /* The fault originated in userspace */ #define FAULT_FLAG_REMOTE 0x80 /* faulting for non current tsk/mm */ #define FAULT_FLAG_INSTRUCTION 0x100 /* The fault was during an instruction fetch */ +#define FAULT_FLAG_SPECULATIVE 0x200 /* Speculative fault, not holding mmap_sem */ #define FAULT_FLAG_TRACE \ { FAULT_FLAG_WRITE, "WRITE" }, \ @@ -409,6 +410,10 @@ struct vm_fault { gfp_t gfp_mask; /* gfp mask to be used for allocations */ pgoff_t pgoff; /* Logical page offset based on vma */ unsigned long address; /* Faulting virtual address */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + unsigned int sequence; + pmd_t orig_pmd; /* value of PMD at the time of fault */ +#endif pmd_t *pmd; /* Pointer to pmd entry matching * the 'address' */ pud_t *pud; /* Pointer to pud entry matching @@ -1524,6 +1529,31 @@ int invalidate_inode_page(struct page *page); #ifdef CONFIG_MMU extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags); + +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +extern vm_fault_t __handle_speculative_fault(struct mm_struct *mm, + unsigned long address, + unsigned int flags); +static inline vm_fault_t handle_speculative_fault(struct mm_struct *mm, + unsigned long address, + unsigned int flags) +{ + /* + * Try speculative page fault for multithreaded user space task only. + */ + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + return VM_FAULT_RETRY; + return __handle_speculative_fault(mm, address, flags); +} +#else +static inline vm_fault_t handle_speculative_fault(struct mm_struct *mm, + unsigned long address, + unsigned int flags) +{ + return VM_FAULT_RETRY; +} +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + extern int fixup_user_fault(struct task_struct *tsk, struct mm_struct *mm, unsigned long address, unsigned int fault_flags, bool *unlocked); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2e8438a1216a..2fcfaa910007 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -457,8 +457,8 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma, pgoff_t pgoff; if (unlikely(is_vm_hugetlb_page(vma))) return linear_hugepage_index(vma, address); - pgoff = (address - vma->vm_start) >> PAGE_SHIFT; - pgoff += vma->vm_pgoff; + pgoff = (address - READ_ONCE(vma->vm_start)) >> PAGE_SHIFT; + pgoff += READ_ONCE(vma->vm_pgoff); return pgoff; } diff --git a/mm/internal.h b/mm/internal.h index 1e368e4afe3c..ed91b199cb8c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -58,6 +58,21 @@ static inline void put_vma(struct vm_area_struct *vma) extern struct vm_area_struct *find_vma_rcu(struct mm_struct *mm, unsigned long addr); + +static inline bool vma_has_changed(struct vm_fault *vmf) +{ + int ret = RB_EMPTY_NODE(&vmf->vma->vm_rb); + unsigned int seq = READ_ONCE(vmf->vma->vm_sequence.sequence); + + /* + * Matches both the wmb in write_seqlock_{begin,end}() and + * the wmb in vma_rb_erase(). + */ + smp_rmb(); + + return ret || seq != vmf->sequence; +} + #else /* CONFIG_SPECULATIVE_PAGE_FAULT */ static inline void get_vma(struct vm_area_struct *vma) diff --git a/mm/memory.c b/mm/memory.c index 46f877b6abea..6e6bf61c0e5c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -522,7 +522,8 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr, if (page) dump_page(page, "bad pte"); pr_alert("addr:%p vm_flags:%08lx anon_vma:%p mapping:%p index:%lx\n", - (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index); + (void *)addr, READ_ONCE(vma->vm_flags), vma->anon_vma, + mapping, index); pr_alert("file:%pD fault:%pf mmap:%pf readpage:%pf\n", vma->vm_file, vma->vm_ops ? vma->vm_ops->fault : NULL, @@ -2082,6 +2083,118 @@ int apply_to_page_range(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(apply_to_page_range); +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +static bool pte_spinlock(struct vm_fault *vmf) +{ + bool ret = false; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + pmd_t pmdval; +#endif + + /* Check if vma is still valid */ + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) { + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + spin_lock(vmf->ptl); + return true; + } + +again: + local_irq_disable(); + if (vma_has_changed(vmf)) + goto out; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* + * We check if the pmd value is still the same to ensure that there + * is not a huge collapse operation in progress in our back. + */ + pmdval = READ_ONCE(*vmf->pmd); + if (!pmd_same(pmdval, vmf->orig_pmd)) + goto out; +#endif + + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + if (unlikely(!spin_trylock(vmf->ptl))) { + local_irq_enable(); + goto again; + } + + if (vma_has_changed(vmf)) { + spin_unlock(vmf->ptl); + goto out; + } + + ret = true; +out: + local_irq_enable(); + return ret; +} + +static bool pte_map_lock(struct vm_fault *vmf) +{ + bool ret = false; + pte_t *pte; + spinlock_t *ptl; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + pmd_t pmdval; +#endif + + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) { + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + return true; + } + + /* + * The first vma_has_changed() guarantees the page-tables are still + * valid, having IRQs disabled ensures they stay around, hence the + * second vma_has_changed() to make sure they are still valid once + * we've got the lock. After that a concurrent zap_pte_range() will + * block on the PTL and thus we're safe. + */ +again: + local_irq_disable(); + if (vma_has_changed(vmf)) + goto out; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* + * We check if the pmd value is still the same to ensure that there + * is not a huge collapse operation in progress in our back. + */ + pmdval = READ_ONCE(*vmf->pmd); + if (!pmd_same(pmdval, vmf->orig_pmd)) + goto out; +#endif + + /* + * Same as pte_offset_map_lock() except that we call + * spin_trylock() in place of spin_lock() to avoid race with + * unmap path which may have the lock and wait for this CPU + * to invalidate TLB but this CPU has irq disabled. + * Since we are in a speculative patch, accept it could fail + */ + ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); + pte = pte_offset_map(vmf->pmd, vmf->address); + if (unlikely(!spin_trylock(ptl))) { + pte_unmap(pte); + local_irq_enable(); + goto again; + } + + if (vma_has_changed(vmf)) { + pte_unmap_unlock(pte, ptl); + goto out; + } + + vmf->pte = pte; + vmf->ptl = ptl; + ret = true; +out: + local_irq_enable(); + return ret; +} +#else static inline bool pte_spinlock(struct vm_fault *vmf) { vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); @@ -2095,6 +2208,7 @@ static inline bool pte_map_lock(struct vm_fault *vmf) vmf->address, &vmf->ptl); return true; } +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ /* * handle_pte_fault chooses page fault handler according to an entry which was @@ -2999,6 +3113,14 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) ret = check_stable_address_space(vma->vm_mm); if (ret) goto unlock; + /* + * Don't call the userfaultfd during the speculative path. + * We already checked for the VMA to not be managed through + * userfaultfd, but it may be set in our back once we have lock + * the pte. In such a case we can ignore it this time. + */ + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + goto setpte; /* Deliver the page fault to userland, check inside PT lock */ if (userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -3041,7 +3163,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) goto unlock_and_release; /* Deliver the page fault to userland, check inside PT lock */ - if (userfaultfd_missing(vma)) { + if (!(vmf->flags & FAULT_FLAG_SPECULATIVE) && + userfaultfd_missing(vma)) { pte_unmap_unlock(vmf->pte, vmf->ptl); mem_cgroup_cancel_charge(page, memcg, false); put_page(page); @@ -3836,6 +3959,15 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) pte_t entry; if (unlikely(pmd_none(*vmf->pmd))) { + /* + * In the case of the speculative page fault handler we abort + * the speculative path immediately as the pmd is probably + * in the way to be converted in a huge one. We will try + * again holding the mmap_sem (which implies that the collapse + * operation is done). + */ + if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; /* * Leave __pte_alloc() until later: because vm_ops->fault may * want to allocate huge page, and if we expose page table @@ -3843,7 +3975,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * concurrent faults and from rmap lookups. */ vmf->pte = NULL; - } else { + } else if (!(vmf->flags & FAULT_FLAG_SPECULATIVE)) { /* See comment in pte_alloc_one_map() */ if (pmd_devmap_trans_unstable(vmf->pmd)) return 0; @@ -3852,6 +3984,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * pmd from under us anymore at this point because we hold the * mmap_sem read mode and khugepaged takes it in write mode. * So now it's safe to run pte_offset_map(). + * This is not applicable to the speculative page fault handler + * but in that case, the pte is fetched earlier in + * handle_speculative_fault(). */ vmf->pte = pte_offset_map(vmf->pmd, vmf->address); vmf->orig_pte = *vmf->pte; @@ -3874,6 +4009,8 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (!vmf->pte) { if (vma_is_anonymous(vmf->vma)) return do_anonymous_page(vmf); + else if (vmf->flags & FAULT_FLAG_SPECULATIVE) + return VM_FAULT_RETRY; else return do_fault(vmf); } @@ -3971,6 +4108,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, vmf.pmd = pmd_alloc(mm, vmf.pud, address); if (!vmf.pmd) return VM_FAULT_OOM; +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + vmf.sequence = raw_read_seqcount(&vma->vm_sequence); +#endif if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) { ret = create_huge_pmd(&vmf); if (!(ret & VM_FAULT_FALLBACK)) @@ -4004,6 +4144,204 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, return handle_pte_fault(&vmf); } +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +/* + * Tries to handle the page fault in a speculative way, without grabbing the + * mmap_sem. + */ +vm_fault_t __handle_speculative_fault(struct mm_struct *mm, + unsigned long address, + unsigned int flags) +{ + struct vm_fault vmf = { + .address = address, + }; + pgd_t *pgd, pgdval; + p4d_t *p4d, p4dval; + pud_t pudval; + int seq; + vm_fault_t ret = VM_FAULT_RETRY; + struct vm_area_struct *vma; +#ifdef CONFIG_NUMA + struct mempolicy *pol; +#endif + + /* Clear flags that may lead to release the mmap_sem to retry */ + flags &= ~(FAULT_FLAG_ALLOW_RETRY|FAULT_FLAG_KILLABLE); + flags |= FAULT_FLAG_SPECULATIVE; + + vma = find_vma_rcu(mm, address); + if (!vma) + return ret; + + /* rmb <-> seqlock,vma_rb_erase() */ + seq = raw_read_seqcount(&vma->vm_sequence); + if (seq & 1) + goto out_put; + + /* + * Can't call vm_ops service has we don't know what they would do + * with the VMA. + * This include huge page from hugetlbfs. + */ + if (vma->vm_ops && vma->vm_ops->fault) + goto out_put; + + /* + * __anon_vma_prepare() requires the mmap_sem to be held + * because vm_next and vm_prev must be safe. This can't be guaranteed + * in the speculative path. + */ + if (unlikely(!vma->anon_vma)) + goto out_put; + + vmf.vma_flags = READ_ONCE(vma->vm_flags); + vmf.vma_page_prot = READ_ONCE(vma->vm_page_prot); + + /* Can't call userland page fault handler in the speculative path */ + if (unlikely(vmf.vma_flags & VM_UFFD_MISSING)) + goto out_put; + + if (vmf.vma_flags & VM_GROWSDOWN || vmf.vma_flags & VM_GROWSUP) + /* + * This could be detected by the check address against VMA's + * boundaries but we want to trace it as not supported instead + * of changed. + */ + goto out_put; + + if (address < READ_ONCE(vma->vm_start) + || READ_ONCE(vma->vm_end) <= address) + goto out_put; + + if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE, + flags & FAULT_FLAG_INSTRUCTION, + flags & FAULT_FLAG_REMOTE)) { + ret = VM_FAULT_SIGSEGV; + goto out_put; + } + + /* This is one is required to check that the VMA has write access set */ + if (flags & FAULT_FLAG_WRITE) { + if (unlikely(!(vmf.vma_flags & VM_WRITE))) { + ret = VM_FAULT_SIGSEGV; + goto out_put; + } + } else if (unlikely(!(vmf.vma_flags & (VM_READ|VM_EXEC|VM_WRITE)))) { + ret = VM_FAULT_SIGSEGV; + goto out_put; + } + +#ifdef CONFIG_NUMA + /* + * MPOL_INTERLEAVE implies additional checks in + * mpol_misplaced() which are not compatible with the + *speculative page fault processing. + */ + pol = __get_vma_policy(vma, address); + if (!pol) + pol = get_task_policy(current); + if (pol && pol->mode == MPOL_INTERLEAVE) + goto out_put; +#endif + + /* + * Do a speculative lookup of the PTE entry. + */ + local_irq_disable(); + pgd = pgd_offset(mm, address); + pgdval = READ_ONCE(*pgd); + if (pgd_none(pgdval) || unlikely(pgd_bad(pgdval))) + goto out_walk; + + p4d = p4d_offset(pgd, address); + p4dval = READ_ONCE(*p4d); + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) + goto out_walk; + + vmf.pud = pud_offset(p4d, address); + pudval = READ_ONCE(*vmf.pud); + if (pud_none(pudval) || unlikely(pud_bad(pudval))) + goto out_walk; + + /* Huge pages at PUD level are not supported. */ + if (unlikely(pud_trans_huge(pudval))) + goto out_walk; + + vmf.pmd = pmd_offset(vmf.pud, address); + vmf.orig_pmd = READ_ONCE(*vmf.pmd); + /* + * pmd_none could mean that a hugepage collapse is in progress + * in our back as collapse_huge_page() mark it before + * invalidating the pte (which is done once the IPI is catched + * by all CPU and we have interrupt disabled). + * For this reason we cannot handle THP in a speculative way since we + * can't safely identify an in progress collapse operation done in our + * back on that PMD. + * Regarding the order of the following checks, see comment in + * pmd_devmap_trans_unstable() + */ + if (unlikely(pmd_devmap(vmf.orig_pmd) || + pmd_none(vmf.orig_pmd) || pmd_trans_huge(vmf.orig_pmd) || + is_swap_pmd(vmf.orig_pmd))) + goto out_walk; + + /* + * The above does not allocate/instantiate page-tables because doing so + * would lead to the possibility of instantiating page-tables after + * free_pgtables() -- and consequently leaking them. + * + * The result is that we take at least one !speculative fault per PMD + * in order to instantiate it. + */ + + vmf.pte = pte_offset_map(vmf.pmd, address); + vmf.orig_pte = READ_ONCE(*vmf.pte); + barrier(); /* See comment in handle_pte_fault() */ + if (pte_none(vmf.orig_pte)) { + pte_unmap(vmf.pte); + vmf.pte = NULL; + } + + vmf.vma = vma; + vmf.pgoff = linear_page_index(vma, address); + vmf.gfp_mask = __get_fault_gfp_mask(vma); + vmf.sequence = seq; + vmf.flags = flags; + + local_irq_enable(); + + /* + * We need to re-validate the VMA after checking the bounds, otherwise + * we might have a false positive on the bounds. + */ + if (read_seqcount_retry(&vma->vm_sequence, seq)) + goto out_put; + + mem_cgroup_enter_user_fault(); + ret = handle_pte_fault(&vmf); + mem_cgroup_exit_user_fault(); + + put_vma(vma); + + /* + * The task may have entered a memcg OOM situation but + * if the allocation error was handled gracefully (no + * VM_FAULT_OOM), there is no need to kill anything. + * Just clean up the OOM state peacefully. + */ + if (task_in_memcg_oom(current) && !(ret & VM_FAULT_OOM)) + mem_cgroup_oom_synchronize(false); + return ret; + +out_walk: + local_irq_enable(); +out_put: + put_vma(vma); + return ret; +} +#endif /* CONFIG_SPECULATIVE_PAGE_FAULT */ + /* * By the time we get here, we already hold the mm semaphore * From patchwork Tue Apr 16 13:45:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903187 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 371C217E1 for ; Tue, 16 Apr 2019 13:48:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A1DE2843C for ; Tue, 16 Apr 2019 13:48:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D48B286CF; Tue, 16 Apr 2019 13:48:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 758622843C for ; Tue, 16 Apr 2019 13:48:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A13276B029A; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 997716B029B; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 836DE6B029D; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 284A26B029B for ; Tue, 16 Apr 2019 09:48:03 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id e6so10970553edi.20 for ; Tue, 16 Apr 2019 06:48:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=Jz5rqUkE6WFM4DKQsFaVvFygOBtBuOfuq1p+t1n4dxE=; b=DRKJudefvI4Rm86eUpUuK9/bRtq6/65XTPBK1GcklwRkZTvya+OB+Rr27Zf8l11PoE FIqVwbHO1N+WAnNp39jyBWDcREigRlxIZqblAMrT5b7jvcKRzD1YjVl1RUyIRzfO4I5Q 1d9BxaaqAItr6re5nvDymP1C5oSO8bMiUMFPWNzhQRpu0SJ1WYN2v18TuCSwn0FbylEh WQn8mAAoIqtfx0nB0yobLy0GbPDIyqK+5MQOhfb709L29iNXzHZTLGqf0V8wPi+BTJd9 boV9UwRbCHteqtDnv2G8sDGZLMLMO3DclLF+r+CHX90A5o7uLYjuAbZgPE1F3PWueBmR fhYw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAXTX53MxHIur1mLgcSWsGWizzIHbULjQ7RGtrszKG/zTugVv7RJ yMI6Owy2/kpgZtE/q+F5xT2JzWTSJR7WOzcdtUTGop93ZZ638rvgxQ0H/CZNdgFSOnAjuIFY0ly ztUP7gF2lQ4rLmJyz086HZp7pDRp50KXoJl+3Z3qYT18cckYge6wUbzLVY0+11X9btA== X-Received: by 2002:a50:a7a6:: with SMTP id i35mr50296119edc.96.1555422482652; Tue, 16 Apr 2019 06:48:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqyhSyTnGcxWEbPKdSLEuN8TYB3ABYWUHOwmeYxRBJ1PQhQTX8TH68P1kooHu7DjDx6GDdTz X-Received: by 2002:a50:a7a6:: with SMTP id i35mr50296009edc.96.1555422480608; Tue, 16 Apr 2019 06:48:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422480; cv=none; d=google.com; s=arc-20160816; b=wIaWdwayg5urejySgyLnImYXPgf5oNz6wuzqG9P/hN6/FqoQwKTyl6Z0sdhE16HZQI cOeQwqg+q5/0XTdTsqJY5pNIVVkA8JoXmvbVvd8c8GpMieSADmZzn0FqPlYYC89F6/Cf GMMvqEAqe6OgyuzNoh/YqtnAkR22g3Nqcumn/idLBd29Lt7JBx2GlXADiZyKaNhRRcFp qr51K2t2b0JggimcILIvSS75P0jmPFzc1/Enn4vLpxtoDBlUp/xk6TgE1XIypWAa5L1v TZCvY+ZvLzX8X9KQ15BBN3G69n5TDpmkU4mNbg9orjxvfHNoHGf41ZAPzyKUUEAP+fRj bR+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=Jz5rqUkE6WFM4DKQsFaVvFygOBtBuOfuq1p+t1n4dxE=; b=DbuBQhmZPMSDNyb+XpQN2wUiX16CVdQ242q9255OVLhnkkXETx77hJy6DIyX/PHQiK teupCCEHn5/SWHZO6zz3/IKnFqikivin357HV5RuXRUVRmdf4XursnK+XIfCsr+6qSi1 CtADqZVQBxMPzzSf+F0dcn6EAPoDnNRzSudJnQBI1D2UAmjPxSmpqQRKv0uYBkGgC7SN /1b/DvprQqyYny3N++1HCXLSZlpxkGH1YeKXxmRibCv9Pr1OrqGY+gqCkkU8C6y9ihdj lXO0whT6855X5rKJAoJpriVHoapWOmBFbVpuT/cmQdbXJ8RhY4t1PO4yi4yINOhQZJBE vv/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id c45si15955833ede.103.2019.04.16.06.48.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:48:00 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlGXl060623 for ; Tue, 16 Apr 2019 09:47:59 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwe3k5vd6-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:20 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:43 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:33 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkVc961145266 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:31 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 80BDA4C050; Tue, 16 Apr 2019 13:46:31 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 102364C046; Tue, 16 Apr 2019 13:46:30 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:29 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org, Vinayak Menon Subject: [PATCH v12 23/31] mm: don't do swap readahead during speculative page fault Date: Tue, 16 Apr 2019 15:45:14 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0008-0000-0000-000002DA6FB6 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0009-0000-0000-00002246A83F Message-Id: <20190416134522.17540-24-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=934 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Vinayak Menon faced a panic because one thread was page faulting a page in swap, while another one was mprotecting a part of the VMA leading to a VMA split. This raise a panic in swap_vma_readahead() because the VMA's boundaries were not more matching the faulting address. To avoid this, if the page is not found in the swap, the speculative page fault is aborted to retry a regular page fault. Reported-by: Vinayak Menon Signed-off-by: Laurent Dufour Reviewed-by: Jérôme Glisse --- mm/memory.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 6e6bf61c0e5c..1991da97e2db 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2900,6 +2900,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) lru_cache_add_anon(page); swap_readpage(page, true); } + } else if (vmf->flags & FAULT_FLAG_SPECULATIVE) { + /* + * Don't try readahead during a speculative page fault + * as the VMA's boundaries may change in our back. + * If the page is not in the swap cache and synchronous + * read is disabled, fall back to the regular page + * fault mechanism. + */ + delayacct_clear_flag(DELAYACCT_PF_SWAPIN); + ret = VM_FAULT_RETRY; + goto out; } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); From patchwork Tue Apr 16 13:45:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903143 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8511817E1 for ; Tue, 16 Apr 2019 13:47:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6112E288DB for ; Tue, 16 Apr 2019 13:47:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 52E4D288F4; Tue, 16 Apr 2019 13:47:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76C8F286B9 for ; Tue, 16 Apr 2019 13:47:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70EAF6B0266; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 68FAF6B026C; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43EDC6B026A; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f69.google.com (mail-yw1-f69.google.com [209.85.161.69]) by kanga.kvack.org (Postfix) with ESMTP id 1B2A96B0269 for ; Tue, 16 Apr 2019 09:46:55 -0400 (EDT) Received: by mail-yw1-f69.google.com with SMTP id j62so15657861ywe.3 for ; Tue, 16 Apr 2019 06:46:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=WKVMnBgOI6B/J82OoWTSlIuYM7akWgmuMADBH08y4RQ=; b=ZuPwkU0aCMaZ7NVWQoAQRHRyJ8TSjIL5tv/m5KTVNZJiQ6Ul69rCCf+VRPHXBvMn9w Lx4ZCxxEUhwVTSV5/satKUgXo4CfzOLKUO62TEBZKxVa6ebjUD0ru2cIZwfp00JHutVx e48Mp+hrggvY5CuhkQXqEfVbEvWEPm0X6VFUeJ/xskeweFJ2bNEB1FvIMzn2jGSa7F7l klwxoEYjKOHK0SxQiqOq7VGqa+F3NE8aAk0NaSxrwROTzHciskyjZkW5VRzKl2rm2dv/ beaOBQk1xGlKprZpCMQBBvLDzXPwWF7gAMoLtNtemIcy/Gzwh2uOqsCNhxB4uvoKb7bl AKHg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAWeTMMCkek/4ojzwBhuABSLhaivuIbUf5YMWagEZVPJUQC0m3S3 5W5gGyAi/jNoQxQFBo2zhb8Mgqv8hRLbZdkTTCKzR9YEzs/niXH8bFtoFosU2sjroWUTSHhhbdC 0hTU9I5VlSo/xfxkjsZs3/Sp46W43HDpnDClRNw6XPGbI1kHVJXeFmFNMLFdQ+scpjA== X-Received: by 2002:a81:3253:: with SMTP id y80mr64586434ywy.63.1555422414777; Tue, 16 Apr 2019 06:46:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqy6DtSnNAjMahPxwsTjFVQpJ0APoWsyYH6afLqcxOEBZX5Hs9GQaP1oUOznnfgc68+Y6vNH X-Received: by 2002:a81:3253:: with SMTP id y80mr64586308ywy.63.1555422413186; Tue, 16 Apr 2019 06:46:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422413; cv=none; d=google.com; s=arc-20160816; b=hbI8D6FXCNbzKHF1lHKZIjwP99RzyHfJHkd3qoFMOzyYTX3ALP8I9pncHN8oioNewR EqavznOWMYQQKQyqkbipyvsh1TQhpEWaPIht29IGkYACVgaSd+rIyLnqGPGekgVh/B3Q qh8AEurTi03LmSzulesL9xLvaOIEMCp2wJc6HRQXLu2hWsEwOwX3UPIQfoU/J9C4eZH1 OPv92KeEZI1vhKCr2QCjEPNjGboUuaubCvUnfwEZcxeo509mnajevOv9/qssI1Dpv5QM s3e0zoaUpW7oyRMKOCNwrEvwesUN7ZKAOeKZL2Ot1dUSizwMWO9sATlAystbiB3L66KM /zww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=WKVMnBgOI6B/J82OoWTSlIuYM7akWgmuMADBH08y4RQ=; b=ZlICak8N4g/nHX5ocpz3hHLr2mEdrB2Ug+KLLQCfWvFY8ssDqC7HyE9/I0taComoHq 4BWSXJ4BIDiFjNsEjjsji2u23WMD3YHPq657Immxt5x8ZYCdpK0DuOXLn1EGRA7zT+Pv wA8X6yIbW+p3Xg81r1nD51lZxpVAvU8dw6UxJDUMwmDEiDLqgPpUczRzea6Z0QYft8rN sxveXI2TWYvd4U3hmiHHi3jYmzpMcRXw5uC/84khEu7b/btG3IsjsubCi3pQvFpYLm6v Grvcbq8+meyPCK7mdqPCGqlTY2F5dXQqYh57ib07bREZ9TnJDqKmc1KmS0hwgYU+OQs4 RdRw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id w5si34891547yww.420.2019.04.16.06.46.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:46:53 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkmvl059220 for ; Tue, 16 Apr 2019 09:46:52 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe36nymw-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:50 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:46 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:36 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkYua30736478 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:34 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 623894C040; Tue, 16 Apr 2019 13:46:34 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8FE474C04A; Tue, 16 Apr 2019 13:46:32 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:32 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 24/31] mm: adding speculative page fault failure trace events Date: Tue, 16 Apr 2019 15:45:15 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0008-0000-0000-000002DA6FB7 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0009-0000-0000-00002246A840 Message-Id: <20190416134522.17540-25-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch a set of new trace events to collect the speculative page fault event failures. Signed-off-by: Laurent Dufour --- include/trace/events/pagefault.h | 80 ++++++++++++++++++++++++++++++++ mm/memory.c | 57 ++++++++++++++++++----- 2 files changed, 125 insertions(+), 12 deletions(-) create mode 100644 include/trace/events/pagefault.h diff --git a/include/trace/events/pagefault.h b/include/trace/events/pagefault.h new file mode 100644 index 000000000000..d9438f3e6bad --- /dev/null +++ b/include/trace/events/pagefault.h @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM pagefault + +#if !defined(_TRACE_PAGEFAULT_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_PAGEFAULT_H + +#include +#include + +DECLARE_EVENT_CLASS(spf, + + TP_PROTO(unsigned long caller, + struct vm_area_struct *vma, unsigned long address), + + TP_ARGS(caller, vma, address), + + TP_STRUCT__entry( + __field(unsigned long, caller) + __field(unsigned long, vm_start) + __field(unsigned long, vm_end) + __field(unsigned long, address) + ), + + TP_fast_assign( + __entry->caller = caller; + __entry->vm_start = vma->vm_start; + __entry->vm_end = vma->vm_end; + __entry->address = address; + ), + + TP_printk("ip:%lx vma:%lx-%lx address:%lx", + __entry->caller, __entry->vm_start, __entry->vm_end, + __entry->address) +); + +DEFINE_EVENT(spf, spf_vma_changed, + + TP_PROTO(unsigned long caller, + struct vm_area_struct *vma, unsigned long address), + + TP_ARGS(caller, vma, address) +); + +DEFINE_EVENT(spf, spf_vma_noanon, + + TP_PROTO(unsigned long caller, + struct vm_area_struct *vma, unsigned long address), + + TP_ARGS(caller, vma, address) +); + +DEFINE_EVENT(spf, spf_vma_notsup, + + TP_PROTO(unsigned long caller, + struct vm_area_struct *vma, unsigned long address), + + TP_ARGS(caller, vma, address) +); + +DEFINE_EVENT(spf, spf_vma_access, + + TP_PROTO(unsigned long caller, + struct vm_area_struct *vma, unsigned long address), + + TP_ARGS(caller, vma, address) +); + +DEFINE_EVENT(spf, spf_pmd_changed, + + TP_PROTO(unsigned long caller, + struct vm_area_struct *vma, unsigned long address), + + TP_ARGS(caller, vma, address) +); + +#endif /* _TRACE_PAGEFAULT_H */ + +/* This part must be outside protection */ +#include diff --git a/mm/memory.c b/mm/memory.c index 1991da97e2db..509851ad7c95 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -81,6 +81,9 @@ #include "internal.h" +#define CREATE_TRACE_POINTS +#include + #if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST) #warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid. #endif @@ -2100,8 +2103,10 @@ static bool pte_spinlock(struct vm_fault *vmf) again: local_irq_disable(); - if (vma_has_changed(vmf)) + if (vma_has_changed(vmf)) { + trace_spf_vma_changed(_RET_IP_, vmf->vma, vmf->address); goto out; + } #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* @@ -2109,8 +2114,10 @@ static bool pte_spinlock(struct vm_fault *vmf) * is not a huge collapse operation in progress in our back. */ pmdval = READ_ONCE(*vmf->pmd); - if (!pmd_same(pmdval, vmf->orig_pmd)) + if (!pmd_same(pmdval, vmf->orig_pmd)) { + trace_spf_pmd_changed(_RET_IP_, vmf->vma, vmf->address); goto out; + } #endif vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); @@ -2121,6 +2128,7 @@ static bool pte_spinlock(struct vm_fault *vmf) if (vma_has_changed(vmf)) { spin_unlock(vmf->ptl); + trace_spf_vma_changed(_RET_IP_, vmf->vma, vmf->address); goto out; } @@ -2154,8 +2162,10 @@ static bool pte_map_lock(struct vm_fault *vmf) */ again: local_irq_disable(); - if (vma_has_changed(vmf)) + if (vma_has_changed(vmf)) { + trace_spf_vma_changed(_RET_IP_, vmf->vma, vmf->address); goto out; + } #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* @@ -2163,8 +2173,10 @@ static bool pte_map_lock(struct vm_fault *vmf) * is not a huge collapse operation in progress in our back. */ pmdval = READ_ONCE(*vmf->pmd); - if (!pmd_same(pmdval, vmf->orig_pmd)) + if (!pmd_same(pmdval, vmf->orig_pmd)) { + trace_spf_pmd_changed(_RET_IP_, vmf->vma, vmf->address); goto out; + } #endif /* @@ -2184,6 +2196,7 @@ static bool pte_map_lock(struct vm_fault *vmf) if (vma_has_changed(vmf)) { pte_unmap_unlock(pte, ptl); + trace_spf_vma_changed(_RET_IP_, vmf->vma, vmf->address); goto out; } @@ -4187,47 +4200,60 @@ vm_fault_t __handle_speculative_fault(struct mm_struct *mm, /* rmb <-> seqlock,vma_rb_erase() */ seq = raw_read_seqcount(&vma->vm_sequence); - if (seq & 1) + if (seq & 1) { + trace_spf_vma_changed(_RET_IP_, vma, address); goto out_put; + } /* * Can't call vm_ops service has we don't know what they would do * with the VMA. * This include huge page from hugetlbfs. */ - if (vma->vm_ops && vma->vm_ops->fault) + if (vma->vm_ops && vma->vm_ops->fault) { + trace_spf_vma_notsup(_RET_IP_, vma, address); goto out_put; + } /* * __anon_vma_prepare() requires the mmap_sem to be held * because vm_next and vm_prev must be safe. This can't be guaranteed * in the speculative path. */ - if (unlikely(!vma->anon_vma)) + if (unlikely(!vma->anon_vma)) { + trace_spf_vma_notsup(_RET_IP_, vma, address); goto out_put; + } vmf.vma_flags = READ_ONCE(vma->vm_flags); vmf.vma_page_prot = READ_ONCE(vma->vm_page_prot); /* Can't call userland page fault handler in the speculative path */ - if (unlikely(vmf.vma_flags & VM_UFFD_MISSING)) + if (unlikely(vmf.vma_flags & VM_UFFD_MISSING)) { + trace_spf_vma_notsup(_RET_IP_, vma, address); goto out_put; + } - if (vmf.vma_flags & VM_GROWSDOWN || vmf.vma_flags & VM_GROWSUP) + if (vmf.vma_flags & VM_GROWSDOWN || vmf.vma_flags & VM_GROWSUP) { /* * This could be detected by the check address against VMA's * boundaries but we want to trace it as not supported instead * of changed. */ + trace_spf_vma_notsup(_RET_IP_, vma, address); goto out_put; + } if (address < READ_ONCE(vma->vm_start) - || READ_ONCE(vma->vm_end) <= address) + || READ_ONCE(vma->vm_end) <= address) { + trace_spf_vma_changed(_RET_IP_, vma, address); goto out_put; + } if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE, flags & FAULT_FLAG_INSTRUCTION, flags & FAULT_FLAG_REMOTE)) { + trace_spf_vma_access(_RET_IP_, vma, address); ret = VM_FAULT_SIGSEGV; goto out_put; } @@ -4235,10 +4261,12 @@ vm_fault_t __handle_speculative_fault(struct mm_struct *mm, /* This is one is required to check that the VMA has write access set */ if (flags & FAULT_FLAG_WRITE) { if (unlikely(!(vmf.vma_flags & VM_WRITE))) { + trace_spf_vma_access(_RET_IP_, vma, address); ret = VM_FAULT_SIGSEGV; goto out_put; } } else if (unlikely(!(vmf.vma_flags & (VM_READ|VM_EXEC|VM_WRITE)))) { + trace_spf_vma_access(_RET_IP_, vma, address); ret = VM_FAULT_SIGSEGV; goto out_put; } @@ -4252,8 +4280,10 @@ vm_fault_t __handle_speculative_fault(struct mm_struct *mm, pol = __get_vma_policy(vma, address); if (!pol) pol = get_task_policy(current); - if (pol && pol->mode == MPOL_INTERLEAVE) + if (pol && pol->mode == MPOL_INTERLEAVE) { + trace_spf_vma_notsup(_RET_IP_, vma, address); goto out_put; + } #endif /* @@ -4326,8 +4356,10 @@ vm_fault_t __handle_speculative_fault(struct mm_struct *mm, * We need to re-validate the VMA after checking the bounds, otherwise * we might have a false positive on the bounds. */ - if (read_seqcount_retry(&vma->vm_sequence, seq)) + if (read_seqcount_retry(&vma->vm_sequence, seq)) { + trace_spf_vma_changed(_RET_IP_, vma, address); goto out_put; + } mem_cgroup_enter_user_fault(); ret = handle_pte_fault(&vmf); @@ -4346,6 +4378,7 @@ vm_fault_t __handle_speculative_fault(struct mm_struct *mm, return ret; out_walk: + trace_spf_vma_notsup(_RET_IP_, vma, address); local_irq_enable(); out_put: put_vma(vma); From patchwork Tue Apr 16 13:45:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903161 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3980117E1 for ; Tue, 16 Apr 2019 13:48:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1CBEE2867F for ; Tue, 16 Apr 2019 13:48:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0FD412874E; Tue, 16 Apr 2019 13:48:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 962712867F for ; Tue, 16 Apr 2019 13:48:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8622B6B0273; Tue, 16 Apr 2019 09:47:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8157A6B0274; Tue, 16 Apr 2019 09:47:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68ACB6B0275; Tue, 16 Apr 2019 09:47:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by kanga.kvack.org (Postfix) with ESMTP id 43A636B0273 for ; Tue, 16 Apr 2019 09:47:07 -0400 (EDT) Received: by mail-yb1-f200.google.com with SMTP id y6so15678866ybb.20 for ; Tue, 16 Apr 2019 06:47:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=UAHX411TVfkHtYtx3rwTP/R30VyZz8uurL0FWWNuBRg=; b=igTsLL661aI6Y14h0YUn361eGffPN3aSFebQkrNriz0+tqDgmq1rcN4Mu08Lm2Xvfg 1LvzUUEaXqbl9Q9JFFW6cbytcPFUH8ER3uRULm6eW88gnaB4A8oV7f5XJDMoqye1CPXS z7ZzVqhtanqLjJgKhBgJNq594pwpnJlgVSofdontqZlNTgwAgwHx/PxloUJ1v+RJ/sih vLEhtrRmCy+h8xfYVws18k3rtrvb2T+X3PdHkUZt/OmoTmayffazzAOPT+Cxy4ve0twW jKlBvwXzAxbKfnaIRxp9yZKmnJxQLun7qMurUJfM7Um6LEqK/tdIbTHjdmyV3hs37gna E4OA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAXcjJuUz7r+Vd53rquTHlc4UtXnrjAxXwyfjcwnbiXr3NZ2we4n Mne/2MgbvfxHsfTczbwba2aw9mAtGtumiUcY7GiMNHQSENTg81EcMQNby5PM7YpS2JoBpUK2ps6 qWVf0PaKAQo2/9vL1APwURZiBPalgPSrTQglNfvI3NXIbxCwyTsCUsHrLOfF3KGIA5g== X-Received: by 2002:a81:1d49:: with SMTP id d70mr67013274ywd.84.1555422427015; Tue, 16 Apr 2019 06:47:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqxHiPemEeekaTMNp8S/DxryBG/c/OtYnJiXFuSwdRptu7Z3457YoWkGAenFt0Z3uyQCzc2T X-Received: by 2002:a81:1d49:: with SMTP id d70mr67013191ywd.84.1555422426040; Tue, 16 Apr 2019 06:47:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422426; cv=none; d=google.com; s=arc-20160816; b=EaCADxEC5WsLtlAsutcelXOtdymlAzMgzsVdXK2Ghvhar22aBPlJbJq9Mg2YxpjzL6 Nf2cjzU8EKj+pgxFTn6LOmCQWUv9TJT7h5ANziajrIfPjjRrKmTTj0ImU5bMbw9u2keJ NuclGlxyQyIO2ZYn5F+gxe4jJoJSPOn6CI7fO/JGgpVytfdElPO+nH3vxJLSOVnWwfsH jokxTpDm87GIYECQ7roUpRo5OWr5pXWhbuwceLnpmS/ABxPZQr6m+ii3G9hpLzmZQDqW Pjs8ZQA91KOeUGFowZfAK+k/UYxrDZZf3fNX3weFaH5h8PhrK15WjBldU/LQXRqlaD1C tRAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=UAHX411TVfkHtYtx3rwTP/R30VyZz8uurL0FWWNuBRg=; b=Pr2QSiXAywjpSD9Hj/Lt8elUJiWEUx9qMcvjZqJZ6tLzVp1el11FfqbQY7GaumD4c3 TpsHA9x/hEHOgxDKkbXSen+MTJpzyRmA7UIQnp7KWuBRTEhvpVBmEhXrq27txjjU016D 6/7YNFnwBIMbo02Gv52ORhf0xfL4XwHXJv0IuqCHjNzTWV1RNik53qJ997S7+pB/6crL DFe5YNA/OYOvRipaDunU8NElwskywHT9tra4sLkv8iV4wS7kQZw8OsIKUIUUOZSjAOtx cIh6KglnknHaFCRLBCNR+DXFfYyv+76FjGW39Zmg4ndgTvw9SWU6jz3aokNnDUYWq3GX udDw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id u129si3612603ybf.268.2019.04.16.06.47.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:06 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkpwE059599 for ; Tue, 16 Apr 2019 09:47:05 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe36nypd-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:01 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:49 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:38 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkbc850200614 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:37 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 473624C04E; Tue, 16 Apr 2019 13:46:37 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A910F4C04A; Tue, 16 Apr 2019 13:46:35 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:35 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 25/31] perf: add a speculative page fault sw event Date: Tue, 16 Apr 2019 15:45:16 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F727D X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD95 Message-Id: <20190416134522.17540-26-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add a new software event to count succeeded speculative page faults. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- include/uapi/linux/perf_event.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h index 7198ddd0c6b1..3b4356c55caa 100644 --- a/include/uapi/linux/perf_event.h +++ b/include/uapi/linux/perf_event.h @@ -112,6 +112,7 @@ enum perf_sw_ids { PERF_COUNT_SW_EMULATION_FAULTS = 8, PERF_COUNT_SW_DUMMY = 9, PERF_COUNT_SW_BPF_OUTPUT = 10, + PERF_COUNT_SW_SPF = 11, PERF_COUNT_SW_MAX, /* non-ABI */ }; From patchwork Tue Apr 16 13:45:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D7BE0161F for ; Tue, 16 Apr 2019 13:47:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BB6AC288DB for ; Tue, 16 Apr 2019 13:47:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AC269288F4; Tue, 16 Apr 2019 13:47:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 256AE288F4 for ; Tue, 16 Apr 2019 13:47:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA51C6B0270; Tue, 16 Apr 2019 09:47:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E29EB6B0272; Tue, 16 Apr 2019 09:47:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF5CB6B0273; Tue, 16 Apr 2019 09:47:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id A742E6B0270 for ; Tue, 16 Apr 2019 09:47:04 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id 204so15703212ybf.5 for ; Tue, 16 Apr 2019 06:47:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=tfdZgFh2HKDjltbc+4Gn/jcI1gLnMp7/QqrTaygkNoI=; b=RaeVjWKigX6Zn2MvxlOlWtyBe4kaBujpqvGREHoKwBNUqE3MEa99P2TZmj9kor+Auu eyunKrIZudz/cYFNhknqKBLsTT/mFZ13cPIOCHAWwXsgi2CFWWUlODe/t3TLNA82tVHm c82KV2eimzw5ZpMPqdCqKFEll0j9a8d1XTCRJuvTjlqYiTItT0Zu8pC+1ueUUthsf6M9 ttrOqO6vKwgdVK9O5Trb5dfOe/q/LEw2F6NDOrc1HSQN294MsHdq2a8d2DHQgPfIfWS6 p7ogwVIJnMrTrYLTU6CCqIWj52oHxvIbSMVMgcnlKriXZan55ajkSPUL+7c8T8rV+TdS hkQA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAWE55ah8b8utyPwHZ2edAqIPVUywdS7c/HfPX47fTHe7Yl/J26k 1gmuKNI+ymRnNFROt+2/VBBqO9xLNaeEIstWkAckUivW8EtZP7oJq0d+Uga9uNbTDDPl6k7RYxc 5tzWJhXvP6W9BuJBHY2KVlduDC1Bl0dDK56hrgSY2DvTt8p37deYPflvvNE6TOjLkjA== X-Received: by 2002:a81:1390:: with SMTP id 138mr63912526ywt.230.1555422424366; Tue, 16 Apr 2019 06:47:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqw4cSNjRsO9WT0/7KmlcO9K+5yvbnoJi+ham/ICSpPpxXsyRa/GzOL+KV5ycDV3obFGOfiM X-Received: by 2002:a81:1390:: with SMTP id 138mr63912449ywt.230.1555422423450; Tue, 16 Apr 2019 06:47:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422423; cv=none; d=google.com; s=arc-20160816; b=EpUYyTVVFzHz9OviQFXVTaHDOz4tpLpH2FJDnDUSmzxQgwGWgQ86Wh0F63HL6mkqi8 43gPjOeqPBSMO6gV2ZgW+Cz/3LuVpfvZnE2fVwEVOqSGN2UDnF2S5e9Q38kkfGbJ+CZL W+lw8FLmpeaZCqtDexIkE+bjCUAzBmcBY0ZFXjC6ycpBdvbrSZ7fhCaWmYMzZXnMHLJG NxqxchVtkimoGyGkC/TA3t5LQ/bWXd7ebPPHZA+kQxZeuP81D9sMILSRGhb7Sj7R/GaD Ss64BQd0ItVV/luti2u6wVItOVLZ9Y8SFQuOEGL09dHk2xjPMr3B9udlTV2rzx2+/op/ NzfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=tfdZgFh2HKDjltbc+4Gn/jcI1gLnMp7/QqrTaygkNoI=; b=uS4c7rtHmvP55Nv9+Y/PEBlFVPCbKk2WHQM84Rkn/yoJUXRzuYKgrVJEEpYEVqaegN FWst8mM4Oi5gZbs65qgpYkcMa+cnnzdGvtxq4OeiwcQPbifhzGBjmP4Qb2lgn6n+4J+j is5jlBDKFuURLdv5iUSkW/zYYAIPPinIkpHI2rWYWVCx1QrYv6cwUjBdF3XV8jeiM312 e6InvdTsB2qRzOKXH7aR2M64DRe8OkuYKo6t9/8ivIWFmziyJC1xf5xyb7XaQv0T4mQp QT9wMDzEAsY0XyPKPUF3jO+YYzJGWNnVRq4rM/BvA81YEeLlBfebe00YwI16AkOzpmj5 rvCA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id 198si33407466ywh.53.2019.04.16.06.47.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:03 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkUZ3022324 for ; Tue, 16 Apr 2019 09:47:02 -0400 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe8rwcf4-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:57 -0400 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:52 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:42 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkeG417104934 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:41 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CF1644C05C; Tue, 16 Apr 2019 13:46:39 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 60C684C058; Tue, 16 Apr 2019 13:46:38 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:38 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 26/31] perf tools: add support for the SPF perf event Date: Tue, 16 Apr 2019 15:45:17 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0020-0000-0000-00000330705D X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0021-0000-0000-00002182B50A Message-Id: <20190416134522.17540-27-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add support for the new speculative faults event. Acked-by: David Rientjes Signed-off-by: Laurent Dufour --- tools/include/uapi/linux/perf_event.h | 1 + tools/perf/util/evsel.c | 1 + tools/perf/util/parse-events.c | 4 ++++ tools/perf/util/parse-events.l | 1 + tools/perf/util/python.c | 1 + 5 files changed, 8 insertions(+) diff --git a/tools/include/uapi/linux/perf_event.h b/tools/include/uapi/linux/perf_event.h index 7198ddd0c6b1..3b4356c55caa 100644 --- a/tools/include/uapi/linux/perf_event.h +++ b/tools/include/uapi/linux/perf_event.h @@ -112,6 +112,7 @@ enum perf_sw_ids { PERF_COUNT_SW_EMULATION_FAULTS = 8, PERF_COUNT_SW_DUMMY = 9, PERF_COUNT_SW_BPF_OUTPUT = 10, + PERF_COUNT_SW_SPF = 11, PERF_COUNT_SW_MAX, /* non-ABI */ }; diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c index 66d066f18b5b..1f3bea4379b2 100644 --- a/tools/perf/util/evsel.c +++ b/tools/perf/util/evsel.c @@ -435,6 +435,7 @@ const char *perf_evsel__sw_names[PERF_COUNT_SW_MAX] = { "alignment-faults", "emulation-faults", "dummy", + "speculative-faults", }; static const char *__perf_evsel__sw_name(u64 config) diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c index 5ef4939408f2..effa8929cc90 100644 --- a/tools/perf/util/parse-events.c +++ b/tools/perf/util/parse-events.c @@ -140,6 +140,10 @@ struct event_symbol event_symbols_sw[PERF_COUNT_SW_MAX] = { .symbol = "bpf-output", .alias = "", }, + [PERF_COUNT_SW_SPF] = { + .symbol = "speculative-faults", + .alias = "spf", + }, }; #define __PERF_EVENT_FIELD(config, name) \ diff --git a/tools/perf/util/parse-events.l b/tools/perf/util/parse-events.l index 7805c71aaae2..d28a6edd0a95 100644 --- a/tools/perf/util/parse-events.l +++ b/tools/perf/util/parse-events.l @@ -324,6 +324,7 @@ emulation-faults { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_EM dummy { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_DUMMY); } duration_time { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_DUMMY); } bpf-output { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_BPF_OUTPUT); } +speculative-faults|spf { return sym(yyscanner, PERF_TYPE_SOFTWARE, PERF_COUNT_SW_SPF); } /* * We have to handle the kernel PMU event cycles-ct/cycles-t/mem-loads/mem-stores separately. diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c index dda0ac978b1e..c617a4751549 100644 --- a/tools/perf/util/python.c +++ b/tools/perf/util/python.c @@ -1200,6 +1200,7 @@ static struct { PERF_CONST(COUNT_SW_ALIGNMENT_FAULTS), PERF_CONST(COUNT_SW_EMULATION_FAULTS), PERF_CONST(COUNT_SW_DUMMY), + PERF_CONST(COUNT_SW_SPF), PERF_CONST(SAMPLE_IP), PERF_CONST(SAMPLE_TID), From patchwork Tue Apr 16 13:45:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903155 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98C1517E1 for ; Tue, 16 Apr 2019 13:47:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7CA87286E2 for ; Tue, 16 Apr 2019 13:47:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 703ED289FC; Tue, 16 Apr 2019 13:47:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F3755286E2 for ; Tue, 16 Apr 2019 13:47:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AABD46B026F; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A5A1C6B0270; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 948096B0272; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 2F8256B026F for ; Tue, 16 Apr 2019 09:47:03 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id p26so2657705edy.19 for ; Tue, 16 Apr 2019 06:47:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=G5RcpMbs5Qpi7XAuTIhInlGH2eGdEH42UsKUzadUQCU=; b=iCbTpXfCr5SslcjWm9bhs5AQi6orJuGDw9+zg0z8/9jUvdm1b5wISoDaplfg5659RJ SWXqkkR0LCoxfIvE5v2vlEO+c1klfWmmfHHUkcmrZRHZ1HP2fd5AWgEEhMibFjtMJ0Ww A5ZUYVG87OR9ZKBrzd0ZWZWJMtGeCR2mfKe8jK6vWazIpZWi4D84CAegs2ncRI5MXc6d g0s3wT9THBib5gUyE6Ltr+svrBFCx0OXYpyP5WHFXH+fxgKZbBGW98qtp/Rj3op62FC6 MxUVdjZRizBKja+UXqq6K6cqZAQ7TX9sXs7aS/aeaxcMfvE4TMLiFVPB/2w1EyuSfZRO nk1A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVNvlfDoIvfv7UG8upC51oL6bwIutqoRFC0luclA91W3BK8kv88 1nH6PJ0npQJz5Z0fzFasS3+tPFVBKCIphdvse215cZgkzdx8Q1TOfwDgfUzhC7wEqoFMWlXWScz v/NuBmVCxcK0xzixAHxoWmYf7VwPe7VXUlAYaNKiTTxL4wwvI1uAl0qUhXvwh4/MNlg== X-Received: by 2002:a17:906:46d1:: with SMTP id k17mr16143396ejs.104.1555422422668; Tue, 16 Apr 2019 06:47:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqyMou6fTPXL5/Lx5d4W4cNctLAxMysGQwSlVt+aRHQlEbqcdCCT+WE1w9t5QxsyUP7+y2Ru X-Received: by 2002:a17:906:46d1:: with SMTP id k17mr16143317ejs.104.1555422421148; Tue, 16 Apr 2019 06:47:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422421; cv=none; d=google.com; s=arc-20160816; b=0+74KHkmxCqE4/RIF1ZnH+gJCQRYj+BXoszlVlLBsiLg5xjom9U94BJ03y3noNSjxZ UsSumxYorbF4wONWEYC3zjAY9raN4RfY+FiDPWNcJxlWrSMaOxrobAgRZ0WQOaiRhXH+ VZYBRBNXLmZ7OyjJWFN77uiRlvCDkTqv6AM1mQP0300+lr3RBsD88GGkNUJ965zE8JSl +LzTYE+pNuHGeoJaKtb5aLinbNwTXuOxIbFLdD11tL7tb5gdWIGHEjdvuvY9HHk0d6+A +s9HxXCXhtmcjCGdRz+V5GGMwrtPH2Gc5GMd9BnJ7esP58XqQtrwjmq1c2fFE5X1ywgO 2l4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=G5RcpMbs5Qpi7XAuTIhInlGH2eGdEH42UsKUzadUQCU=; b=lrYnYey/u+RJx1uHA1aclZcaKtFBGJRnrlQjXdGbvMeM+Lgfa3bgStNztbzdnupNMO 4dZpQMWc4qOl07lLApzCR2c8XrSwdcSBIXmcLj4njClXCmBq+mJLCgdRQg7VD0SDP+ai 9iyPiwNjPta7oF7Jn+cXEUPbNRL3K5/P/bwrp8CshR1mVR+jfNbAPqqKhufyMA6Q5omU xTmKaAdF1CYuvBTFYEVW61VqQ/tKHGj3ZBDYgVfinGBhEka2789QkxJnFQMAmDgdEm3z d0z5aLIo+9VjSv8jTu7DapcZBsSu4nCOw89SPwxDF6cgd54w2kqlNUaZXnSgfWoaleo8 Gm4A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id d7si244793edx.319.2019.04.16.06.47.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:01 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkS1b100127 for ; Tue, 16 Apr 2019 09:46:59 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwdvq70e7-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:46:59 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:55 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:43 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkgCs32833784 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:42 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5135C4C04E; Tue, 16 Apr 2019 13:46:42 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DDCD34C05A; Tue, 16 Apr 2019 13:46:40 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:40 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 27/31] mm: add speculative page fault vmstats Date: Tue, 16 Apr 2019 15:45:18 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7280 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD97 Message-Id: <20190416134522.17540-28-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add speculative_pgfault vmstat counter to count successful speculative page fault handling. Also fixing a minor typo in include/linux/vm_event_item.h. Signed-off-by: Laurent Dufour --- include/linux/vm_event_item.h | 3 +++ mm/memory.c | 3 +++ mm/vmstat.c | 5 ++++- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 47a3441cf4c4..137666e91074 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -109,6 +109,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_SWAP SWAP_RA, SWAP_RA_HIT, +#endif +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + SPECULATIVE_PGFAULT, #endif NR_VM_EVENT_ITEMS }; diff --git a/mm/memory.c b/mm/memory.c index 509851ad7c95..c65e8011d285 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4367,6 +4367,9 @@ vm_fault_t __handle_speculative_fault(struct mm_struct *mm, put_vma(vma); + if (ret != VM_FAULT_RETRY) + count_vm_event(SPECULATIVE_PGFAULT); + /* * The task may have entered a memcg OOM situation but * if the allocation error was handled gracefully (no diff --git a/mm/vmstat.c b/mm/vmstat.c index a7d493366a65..93f54b31e150 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1288,7 +1288,10 @@ const char * const vmstat_text[] = { "swap_ra", "swap_ra_hit", #endif -#endif /* CONFIG_VM_EVENTS_COUNTERS */ +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + "speculative_pgfault", +#endif +#endif /* CONFIG_VM_EVENT_COUNTERS */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA */ From patchwork Tue Apr 16 13:45:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903169 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1DE09161F for ; Tue, 16 Apr 2019 13:48:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F2F292843C for ; Tue, 16 Apr 2019 13:48:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E6F5A286CF; Tue, 16 Apr 2019 13:48:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6EFC72867F for ; Tue, 16 Apr 2019 13:48:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 861386B0277; Tue, 16 Apr 2019 09:47:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C1EE6B0278; Tue, 16 Apr 2019 09:47:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68BAE6B0279; Tue, 16 Apr 2019 09:47:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 18EC76B0278 for ; Tue, 16 Apr 2019 09:47:20 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id g1so10962859edm.16 for ; Tue, 16 Apr 2019 06:47:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=k9/z2p3j44DZBGh5PTqmYkMKjzVk66Vx/Q6VNa38JtM=; b=MquTjq2kRIrluY7/skOCnmPS6F3roK6HRaK1Izm3URFikYxRQtBYbsc/dh6OfP4Z8w JT2Q7l9T1yiiIufRk1ztQXm+lTJRvjeIVDOzStuDPsVEcj4qkYAcXd4z4yvNtCUEZboM P+j3U0tb+pfRCTIR4iD6yizU8Pl3DyVOijqM7eVxgd7b1qddrBugmchSbehBQr18n9u4 8kOKqkqfg1DDCtNx699aUni5qtmbwHnJ5uAOUwAB4sVs9k5LQxHbXEyWmXvGmIghxHmK jkV7gSuejxPepkxa5eKycdKrpBwZ3mmbUBV6Ou8YHxk0NaQIZ8IzP0a46kqlReT7VyDp EFaA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAUiQ3ab1RRi6/Ge0o6awhysfB65okh+1yFP6QHYMAAjEa8HHEM2 Z+AmTCguNS8cqfYza11jEnKTt99eHbOFbBtvH08QT7GSiF3Nk49pmL6bm+uSY9HmY83Z/LyEXlw VYJY/n8PaDoXp2jFmDHeHCphpM508Kn2e4+2f4UN63aA4xNm/1sSMv9ekg7iFaPaQJg== X-Received: by 2002:a50:92a6:: with SMTP id k35mr21816409eda.100.1555422439591; Tue, 16 Apr 2019 06:47:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqxDpvLmJC+GfM+uXEbiQB+HNdZ3lUt8g3Pig/cx0wZ7DxaHt47Nut0G7rTYnEUQeVVSBRUP X-Received: by 2002:a50:92a6:: with SMTP id k35mr21816329eda.100.1555422438185; Tue, 16 Apr 2019 06:47:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422438; cv=none; d=google.com; s=arc-20160816; b=hKr04n3J35CXygI92idexF6sm7YtZF3NgyYyFMehJA+7Z4NxDVs9FYqRXJkjniLBeu LEt4ri9CuRgf0X+cZ8HSKpehK5VrS0vztRvx44894t5k573QgW/VIoJUFioTTvs7V1CG /EkeIwU0wNKp8BLuThZQkn4FCrw0YBJdOqotSPkfdkfwCbOxApkey++bIdF7ggjiuq1P vTBABqwT21frtpul/UMkv5BK3TZnfdluTRlw7p30T2EGgWaDyE6uBWrPY89mAa12EOMH +PKPsdiEw8nPm867orQdpnaSMRjaf0UYEcEtBgzsT78vuDazeO5JfgEIx+0RAxrZktpX rngg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=k9/z2p3j44DZBGh5PTqmYkMKjzVk66Vx/Q6VNa38JtM=; b=MJn4dCF9oujSm1LDBcPlFGLA0+1AezxhXNP0qkI9ju0p/bt33wdAf4ffK+o2qxGQGk Q+Nmnxyd3rbqWV/bIaltMbdGXGtUG6zOIrOLlNIDLKWlnNskEI9zzpvvO0wsn0zBU88G ISeZS+Gy7WBBjdxnfzuJgnI/B40oF49opwyPNrxY6RKrfN8kYb+gRqN/h8q4vrwzSvJW AmfXtbBF+sd6Z3RdFLThhdf7EwDb4llqC1LLlIGcO9Y61joeQO6P5ysSDfBenL8YhlSt Shmh4xXsYdgLK2IiG3lu3QL209ru7a5IT8FWb0mC7nlHtCrjAmNUeJ6gtOgKd2HvuDJ2 cOnw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id w23si4997066eji.29.2019.04.16.06.47.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:18 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkcdK175887 for ; Tue, 16 Apr 2019 09:47:16 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwfbattea-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:15 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:58 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:47 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkjdC54067278 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:45 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E76AD4C05C; Tue, 16 Apr 2019 13:46:44 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6053D4C052; Tue, 16 Apr 2019 13:46:43 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:43 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 28/31] x86/mm: add speculative pagefault handling Date: Tue, 16 Apr 2019 15:45:19 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0016-0000-0000-0000026F7281 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0017-0000-0000-000032CBBD98 Message-Id: <20190416134522.17540-29-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Peter Zijlstra Try a speculative fault before acquiring mmap_sem, if it returns with VM_FAULT_RETRY continue with the mmap_sem acquisition and do the traditional fault. Signed-off-by: Peter Zijlstra (Intel) [Clearing of FAULT_FLAG_ALLOW_RETRY is now done in handle_speculative_fault()] [Retry with usual fault path in the case VM_ERROR is returned by handle_speculative_fault(). This allows signal to be delivered] [Don't build SPF call if !CONFIG_SPECULATIVE_PAGE_FAULT] [Handle memory protection key fault] Signed-off-by: Laurent Dufour --- arch/x86/mm/fault.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 667f1da36208..4390d207a7a1 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1401,6 +1401,18 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif + /* + * Do not try to do a speculative page fault if the fault was due to + * protection keys since it can't be resolved. + */ + if (!(hw_error_code & X86_PF_PK)) { + fault = handle_speculative_fault(mm, address, flags); + if (fault != VM_FAULT_RETRY) { + perf_sw_event(PERF_COUNT_SW_SPF, 1, regs, address); + goto done; + } + } + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1499,6 +1511,8 @@ void do_user_addr_fault(struct pt_regs *regs, } up_read(&mm->mmap_sem); + +done: if (unlikely(fault & VM_FAULT_ERROR)) { mm_fault_error(regs, hw_error_code, address, fault); return; From patchwork Tue Apr 16 13:45:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903171 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35AC817E1 for ; Tue, 16 Apr 2019 13:48:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19A492867F for ; Tue, 16 Apr 2019 13:48:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D7F4286E2; Tue, 16 Apr 2019 13:48:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A1C82867F for ; Tue, 16 Apr 2019 13:48:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 419B66B028B; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3AD8F6B028C; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21CC36B028D; Tue, 16 Apr 2019 09:47:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id C824F6B028B for ; Tue, 16 Apr 2019 09:47:28 -0400 (EDT) Received: by mail-ed1-f72.google.com with SMTP id s21so3586554edd.10 for ; Tue, 16 Apr 2019 06:47:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=D74TI4wTqaLm+uQLuyy7fuTf7fEfjbDu3aa1CZ6Jvec=; b=cWzK8FXPDD2prZ/1VZvUVw0oqIi7LXr+uA6b1rUUr95ZMSIIt/dH2RqgEFjin9Vfmt cWC7tk+1lffPazJ0Y0QPBO9LJe7ciLqmGhBjp4fZ3M9nT3F2bKBuu5msMnrvcRuVmeRT oD60njF/6QrH+S4TIUNe4UuvPX6YL2KJWmrMmOuXxeomogjYAgEtwO9z/511UbZFbUvl Ied5T1a/QC2V94jXH79Q4AIFA/xNHoSwsEQIrKIv9KdKhPSPPqQhbQpwcLwLkjH9v8OG s35zjR4Bi8UKTKiSUtrzRRf1dlArrNPaAz1KsFP0GQ07dkon0gD3kfACRsjKCkcng3jN kX3Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAUHg+Y6XLd4npp6eu8gIx1GSF56M37WUFTLHEmu30TIG5clsPw0 oJAJ3NRmlTU5lAAxWbKvkkAY/QWytrmHlUyTly05cCfMEBWxh+Mv91DiBY1RDgUbyA5E+9gzQhm 7ZH6D5rdE7KAlDqgegQQcaPhSu+5SHx0F2hoMbnXBJL7Cx4Zn5CgMsZB0fD4FIP6jMA== X-Received: by 2002:a17:906:4017:: with SMTP id v23mr44333747ejj.40.1555422448192; Tue, 16 Apr 2019 06:47:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqziSk415UcXA4F2az/8c1fe0MSM83jAVJCkP/zuCJUOtj2pdxub+F3wC+1DVI7oOT1+3VMT X-Received: by 2002:a17:906:4017:: with SMTP id v23mr44333660ejj.40.1555422446576; Tue, 16 Apr 2019 06:47:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422446; cv=none; d=google.com; s=arc-20160816; b=g++QoKbOvWJ1r05RmNVwumkxBxF9oBVBZF7JzorhhmtRdHv4L+s2lTVN2rZabVW95G XUP1sXrQDxbWMlyW54AqBzKjps/OcEASpBWsaYHTCtWSWcSL4GBIEWXgMaiZ0Yk14iY5 N3HIdO/HYPhGamz4Yx71JMbmVF1fAGPApd2wEetRaTDbTa0j5xn4bLu6onot3bExfd3I 5PfzOEsDiZ159NzX89sX17eurYmucrm66fBbXxMblspj7E3e0nZKCLoWvWbTMbfH0Idl Qn93EXbOznreN/Y9AR1RXJcGQXhkoLFOqK+H1OS1dNixBaeApB3pjarUxpQQCgfFEDcI hKBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=D74TI4wTqaLm+uQLuyy7fuTf7fEfjbDu3aa1CZ6Jvec=; b=mP3q5OLS8lgoL6TQwPZm4XfesSGSE9UzWk4W+A7ijCU1LTk22OKqH4mghtN/MOEC+L o3oAc3LLdryopwn/HZfD7+As8TiNBlBQNTPoGPdXjZJ6aZb96xSX2y+HzuSfdee85wWt aoHj+n/99NCwZxo55pkgOsuTpc7jZKRhkXp3HrxcycOpRYt7vn1+FTsYrHDtaeAHD1dq z9l/8mjWO8yydhgO6YUFMApKHJkYku36cjOJuh3+FAcQ8z+8wnjybaDF3tkSzIZdSxaE y/R0I0mkMfN+YF981VEO7dqYlGtCmlpgfi2c3/g+47Kbc1ELnmHoKEf8fV7+t+vynk9o stVg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id w2si7537413edh.436.2019.04.16.06.47.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:26 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) client-ip=148.163.158.5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDlHPk060649 for ; Tue, 16 Apr 2019 09:47:25 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rwe3k5vms-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:21 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:46:59 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:49 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDklaL24838310 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:47 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A0ADD4C044; Tue, 16 Apr 2019 13:46:47 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1BCED4C05E; Tue, 16 Apr 2019 13:46:46 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:45 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 29/31] powerpc/mm: add speculative page fault Date: Tue, 16 Apr 2019 15:45:20 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0008-0000-0000-000002DA6FC4 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0009-0000-0000-00002246A849 Message-Id: <20190416134522.17540-30-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=998 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch enable the speculative page fault on the PowerPC architecture. This will try a speculative page fault without holding the mmap_sem, if it returns with VM_FAULT_RETRY, the mmap_sem is acquired and the traditional page fault processing is done. The speculative path is only tried for multithreaded process as there is no risk of contention on the mmap_sem otherwise. Signed-off-by: Laurent Dufour --- arch/powerpc/mm/fault.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index ec74305fa330..5d48016073cb 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -491,6 +491,21 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; + /* + * Try speculative page fault before grabbing the mmap_sem. + * The Page fault is done if VM_FAULT_RETRY is not returned. + * But if the memory protection keys are active, we don't know if the + * fault is due to key mistmatch or due to a classic protection check. + * To differentiate that, we will need the VMA we no more have, so + * let's retry with the mmap_sem held. + */ + fault = handle_speculative_fault(mm, address, flags); + if (fault != VM_FAULT_RETRY && (IS_ENABLED(CONFIG_PPC_MEM_KEYS) && + fault != VM_FAULT_SIGSEGV)) { + perf_sw_event(PERF_COUNT_SW_SPF, 1, regs, address); + goto done; + } + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -600,6 +615,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, up_read(¤t->mm->mmap_sem); +done: if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); From patchwork Tue Apr 16 13:45:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903163 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADA52161F for ; Tue, 16 Apr 2019 13:48:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8ED7E2843C for ; Tue, 16 Apr 2019 13:48:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 81CC6286CF; Tue, 16 Apr 2019 13:48:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 10EC32843C for ; Tue, 16 Apr 2019 13:48:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FDA06B0003; Tue, 16 Apr 2019 09:47:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8ACB46B0006; Tue, 16 Apr 2019 09:47:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7747D6B0274; Tue, 16 Apr 2019 09:47:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) by kanga.kvack.org (Postfix) with ESMTP id 529E26B0003 for ; Tue, 16 Apr 2019 09:47:11 -0400 (EDT) Received: by mail-yb1-f198.google.com with SMTP id k2so4779916ybp.17 for ; Tue, 16 Apr 2019 06:47:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=EstedJQx7ODAKKF9fts52ZGdTMASFCC7o1iQQyKc1w4=; b=uBN9vNIeEOrRnLISMZwOk7EF3dO5OpbAncgq+QpdB6uDrDlWxXOprUi7YsCh2uybMe wgbJOpvpWvXsI5kgiRRnuHCYT05g0q9ORGwDUXWdZRIDEOHXlmm6QzYjhXd0BOSJeRsO EyQJQiAC4pDbKkHup2Uha7tpg6lmgHKI/rV4lOJqbH4gXaUdKaiPLc4loBvV+UjW8NaG qwoHSzB9j7fZaHvXyByVMGCLtVgKSamw66IXTSYfLfyNnyyyXJ+WZsyJB7xiWIIAW3YE caKkiz+yQO+YYh/NTBAbttxhpDDI5EWVfvgUeVFlWN0LMGdwlm/OrrsfljZF8375NNQw Cxmg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAVoQ2IDg25zUHsRu9P4sHOmx9rkVD5biMytfn94VG5yg6j4ikHF FdoMD2ISZZ6KNHkiV3FVtvsO2Fx5mj+qLIRdzmNWrQyPAnhrVJB+PoKmJiRlRJ3J5LHLEmIBa59 IUv6maI5gAzejCBdqV9pwX+Gm24J1/tS0oHQYe1deRc7zPQrfl4j58vzWOQkR2R/3Sw== X-Received: by 2002:a81:9404:: with SMTP id l4mr64707946ywg.89.1555422431021; Tue, 16 Apr 2019 06:47:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqxfJ3M155anGVMAocL+OgyeOZIdZ0j2+fM2LGeLhchAcIFZudpbF6Gq5Gh8No+wlXmGpp0U X-Received: by 2002:a81:9404:: with SMTP id l4mr64707832ywg.89.1555422429847; Tue, 16 Apr 2019 06:47:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422429; cv=none; d=google.com; s=arc-20160816; b=phQufYB8VOkVGHOI+WOwqxeS134yF/Du2ArGapZeiuNQav1fk95HJigl1WC5G5UssK xDts3MIn9/SWt+OAmgnOXWj7Y8QN9fnfa1esib6Eeh40s39wcATf9jXS8uquSTdcsS5n /JUPuk2N4jgq4KkWwD9AvKJvACW88vUn9OLZluCGmvlKae7AK+d8tULfPH7XrNCGyQHv BV/PLDvKLzog4YMXIyOUzbk9qyJTDT4zCyc/hdilwe5MQlrXmwC/S1tvzxN5cLV/lSrq y/UaQRCjMxq16OVLcNTCJRqfHya/foU0gikSxY6u3Yq4/DFKF48HnB7XAkyG144Lhjpc 2Qzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=EstedJQx7ODAKKF9fts52ZGdTMASFCC7o1iQQyKc1w4=; b=VlVdNgsoxs6JYycZSn/gM12dB8wu2cXZ2n25WQ23XwOLgLkr/mG34vnjI+w6fp2npQ WpwPnBycbRlbbdJ+7NMgotpjL1cQH/aPjyuZ1PFRJVW94NdKt0Krc8H3l+Ic5WcD2zUU vvXFJcD01PwUgF0WjShXt/4Fy/dd0xCDgaR9lP+sKTPC++aJ5card8sFM2QnxHPlfYw/ z9i7NSjiMKIHIypDylRZb1gIAfkR5DpcNKRJBTZRo7DcYLpSGs7k5fR++Tr6/GFLzjP8 mKKo3uS63ICcp55x9HazMuanvDEod5mghsVmL84fkwwQNRODaejOMydUtRisiy/00+Qc s/XQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id s1si32040778ybs.76.2019.04.16.06.47.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:09 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkWxY113195 for ; Tue, 16 Apr 2019 09:47:09 -0400 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwentmv3g-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:07 -0400 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:47:01 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:51 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDknUt21561348 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:49 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 594DE4C044; Tue, 16 Apr 2019 13:46:49 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D75944C05E; Tue, 16 Apr 2019 13:46:47 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:47 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 30/31] arm64/mm: add speculative page fault Date: Tue, 16 Apr 2019 15:45:21 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0012-0000-0000-0000030F6F13 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0013-0000-0000-00002147A86B Message-Id: <20190416134522.17540-31-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Mahendran Ganesh This patch enables the speculative page fault on the arm64 architecture. I completed spf porting in 4.9. From the test result, we can see app launching time improved by about 10% in average. For the apps which have more than 50 threads, 15% or even more improvement can be got. Signed-off-by: Ganesh Mahendran [handle_speculative_fault() is no more returning the vma pointer] Signed-off-by: Laurent Dufour --- arch/arm64/mm/fault.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 4f343e603925..b5e2a93f9c21 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -485,6 +485,16 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); + /* + * let's try a speculative page fault without grabbing the + * mmap_sem. + */ + fault = handle_speculative_fault(mm, addr, mm_flags); + if (fault != VM_FAULT_RETRY) { + perf_sw_event(PERF_COUNT_SW_SPF, 1, regs, addr); + goto done; + } + /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -535,6 +545,8 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr, } up_read(&mm->mmap_sem); +done: + /* * Handle the "normal" (no error) case first. */ From patchwork Tue Apr 16 13:45:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Laurent Dufour X-Patchwork-Id: 10903165 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0A7A17E1 for ; Tue, 16 Apr 2019 13:48:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C53242843C for ; Tue, 16 Apr 2019 13:48:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B961E286CF; Tue, 16 Apr 2019 13:48:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F5782843C for ; Tue, 16 Apr 2019 13:48:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D68966B0006; Tue, 16 Apr 2019 09:47:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D19056B0274; Tue, 16 Apr 2019 09:47:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB9646B0275; Tue, 16 Apr 2019 09:47:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-yw1-f70.google.com (mail-yw1-f70.google.com [209.85.161.70]) by kanga.kvack.org (Postfix) with ESMTP id 97AEA6B0006 for ; Tue, 16 Apr 2019 09:47:12 -0400 (EDT) Received: by mail-yw1-f70.google.com with SMTP id j63so15584221ywb.15 for ; Tue, 16 Apr 2019 06:47:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:in-reply-to:references:mime-version :content-transfer-encoding:message-id; bh=rgeNuvJuXBGvLDIQM3tb0bsPHtZK+4OhBnf+76arYR0=; b=kpRnJMQBLv7xn4A+lJhLqrT8OKpffY+qr5j1q43qWrHfzq2iyFH3Dm5UHNRi5jmhye NSORmzWMu5wa1pxLBWVZdkUngduiRQ6NUtfNRCo/0nLHaQoPud/T0WZKM3Gwpee3fZsY LVTaGml+Zv1sEjKXjxRdRiRJJraWgefuKx0nHkYcQSxN6MhBw0ClAADm6oaO/oUQfS/D JXAyeLEQw3/Od1vlEKUpOlNk8J5AedjqA8R/dDmZsHeR8BXRLJS32A3zbOikME0Ym4VF dWEYMHwC21WZtVC79Fvghs5r7RtIJHH9yb2sau/oibJuui5NUsgkPDpui7KSv7MWGL52 ZkQw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com X-Gm-Message-State: APjAAAU7Dss4GTp0PvMfkRue5QZ4Lfnw+KgFc5z29KP1ByRhDrLLo1MK xNfCqLRdPIRoepZdAFAWW7yp4KEf4VTXje9NuPUSapv9iNlYHmhHjlikSNN1TtUneJ6VWN6X9AK J1FHdNZ/lEb+QfyiKSRlPH5jOd7PeZdFcPBO/Dm0uB3OE0aDeIPCeCetswFSU9hGBxA== X-Received: by 2002:a0d:d58d:: with SMTP id x135mr65956971ywd.396.1555422432311; Tue, 16 Apr 2019 06:47:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqxps3DJ431u8XdXJbQIW70c29mUyo17UqdrmuTJ/yuZnV4tRqgP4x1giE+vRUY0L85M7ezJ X-Received: by 2002:a0d:d58d:: with SMTP id x135mr65956857ywd.396.1555422431064; Tue, 16 Apr 2019 06:47:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555422431; cv=none; d=google.com; s=arc-20160816; b=M5W3Yc8OrQgkNKcOxpaTZrQ96k1FcBRcUP3ec0EMtEsq9DJWKsccC6szMLxqaHRnys UxvwlIMA16dmdZHDisguyrjwjo7A7SUJHcMgSAPxctR8rw2uaE577R0BR5FHNM7+Crvk agx4rC5Nv+/gEApBTQbmRT4oyTYMQo9uWUzScgI31RQhsF952HTNyRX72tGc/zu8Unv4 SrRyvIpqkYY6ilyoLWrnDDHNvYPh7GVK3jgUYn6dpHfXVFRfncEMagvwiCQ4oQOO6Did SB8vYqSkrsRIWGXSMtD18IIuCNocsk2k9AS5gpoMGg2Va5TPhOop2YTBGHTaSalud/8C eI/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:content-transfer-encoding:mime-version:references :in-reply-to:date:subject:cc:to:from; bh=rgeNuvJuXBGvLDIQM3tb0bsPHtZK+4OhBnf+76arYR0=; b=PjxYDB0jh/YSdfKlGXCuTUFtHJLAi2vZ+JLb3pEQBuD+NEqjjkA5Ph7BmEd4f4MrfE 1dvcCiCikBpRXOzekXSfwBks+zQaVCgHKs5QQlVmU+EoSwYl17OR6wwwrmlceM1C+zo1 3aewiM8hh2RizF3mBaP5Z+vIV4Pqdt+R1ndOvKHbavFCtO0NtJXE0ZgU9tY9PfLTdQJQ gVbbYOwN/ypZ8oudEFBDUtK/71PJsTnu193L/4nvFFosYJJ+0jEvJkvCWGXfJAiwNmbK lyGcQcwcayahQ8GkWQ+ewq0qwIG7AvwWTzU+QlD+Bbsw3gSzEdZrN9woq/WbwdiwkAwl sQjA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com. [148.163.156.1]) by mx.google.com with ESMTPS id x144si20350614ywd.157.2019.04.16.06.47.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 Apr 2019 06:47:11 -0700 (PDT) Received-SPF: pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) client-ip=148.163.156.1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of ldufour@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=ldufour@linux.ibm.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ibm.com Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3GDkIaC129607 for ; Tue, 16 Apr 2019 09:47:10 -0400 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2rwe6cnqfw-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 16 Apr 2019 09:47:08 -0400 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 16 Apr 2019 14:47:04 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 16 Apr 2019 14:46:53 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3GDkqd060686582 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 16 Apr 2019 13:46:52 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DE5414C066; Tue, 16 Apr 2019 13:46:51 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6804E4C046; Tue, 16 Apr 2019 13:46:50 +0000 (GMT) Received: from nimbus.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.33]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 16 Apr 2019 13:46:50 +0000 (GMT) From: Laurent Dufour To: akpm@linux-foundation.org, mhocko@kernel.org, peterz@infradead.org, kirill@shutemov.name, ak@linux.intel.com, dave@stgolabs.net, jack@suse.cz, Matthew Wilcox , aneesh.kumar@linux.ibm.com, benh@kernel.crashing.org, mpe@ellerman.id.au, paulus@samba.org, Thomas Gleixner , Ingo Molnar , hpa@zytor.com, Will Deacon , Sergey Senozhatsky , sergey.senozhatsky.work@gmail.com, Andrea Arcangeli , Alexei Starovoitov , kemi.wang@intel.com, Daniel Jordan , David Rientjes , Jerome Glisse , Ganesh Mahendran , Minchan Kim , Punit Agrawal , vinayak menon , Yang Shi , zhong jiang , Haiyan Song , Balbir Singh , sj38.park@gmail.com, Michel Lespinasse , Mike Rapoport Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, haren@linux.vnet.ibm.com, npiggin@gmail.com, paulmck@linux.vnet.ibm.com, Tim Chen , linuxppc-dev@lists.ozlabs.org, x86@kernel.org Subject: [PATCH v12 31/31] mm: Add a speculative page fault switch in sysctl Date: Tue, 16 Apr 2019 15:45:22 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190416134522.17540-1-ldufour@linux.ibm.com> References: <20190416134522.17540-1-ldufour@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19041613-0020-0000-0000-000003307064 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041613-0021-0000-0000-00002182B514 Message-Id: <20190416134522.17540-32-ldufour@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-16_05:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=781 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904160093 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This allows to turn on/off the use of the speculative page fault handler. By default it's turned on. Signed-off-by: Laurent Dufour --- include/linux/mm.h | 3 +++ kernel/sysctl.c | 9 +++++++++ mm/memory.c | 3 +++ 3 files changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index ec609cbad25a..f5bf13a2197a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1531,6 +1531,7 @@ extern vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags); #ifdef CONFIG_SPECULATIVE_PAGE_FAULT +extern int sysctl_speculative_page_fault; extern vm_fault_t __handle_speculative_fault(struct mm_struct *mm, unsigned long address, unsigned int flags); @@ -1538,6 +1539,8 @@ static inline vm_fault_t handle_speculative_fault(struct mm_struct *mm, unsigned long address, unsigned int flags) { + if (unlikely(!sysctl_speculative_page_fault)) + return VM_FAULT_RETRY; /* * Try speculative page fault for multithreaded user space task only. */ diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 9df14b07a488..3a712e52c14a 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1295,6 +1295,15 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, .extra2 = &two, }, +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT + { + .procname = "speculative_page_fault", + .data = &sysctl_speculative_page_fault, + .maxlen = sizeof(sysctl_speculative_page_fault), + .mode = 0644, + .proc_handler = proc_dointvec, + }, +#endif { .procname = "panic_on_oom", .data = &sysctl_panic_on_oom, diff --git a/mm/memory.c b/mm/memory.c index c65e8011d285..a12a60891350 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -83,6 +83,9 @@ #define CREATE_TRACE_POINTS #include +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT +int sysctl_speculative_page_fault = 1; +#endif #if defined(LAST_CPUPID_NOT_IN_PAGE_FLAGS) && !defined(CONFIG_COMPILE_TEST) #warning Unfortunate NUMA and NUMA Balancing config, growing page-frame for last_cpupid.