From patchwork Tue Apr 4 13:58:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13200238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FEB2C77B6C for ; Tue, 4 Apr 2023 13:59:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA71B900002; Tue, 4 Apr 2023 09:59:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C5877900004; Tue, 4 Apr 2023 09:59:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF7AA900003; Tue, 4 Apr 2023 09:59:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A0C6B900002 for ; Tue, 4 Apr 2023 09:59:05 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 74525160EA0 for ; Tue, 4 Apr 2023 13:59:05 +0000 (UTC) X-FDA: 80643865050.18.DAC7717 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id C4B2E1C0017 for ; Tue, 4 Apr 2023 13:59:03 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wOhh5MI6; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680616743; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=52BqLkPH7JK+YVgxAe2fuxeKTYdfJ/oXPMwp85njK/A=; b=T8MhFpCrL94iopFsv5+qSHSCi2gEUKRPUUPgD/gM1jpxinXJnMNxxPxiku+pDolr31LfMo muG1BmI0UEsvNdmYOC/ROT7jNEGaIBFgRY9JIlCdXMI8G0XYjbMzMpdPhi3QQqYNJ9NtO2 9nFls3gaUZV2MsXn2pJ0GdSGKxQUOKM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wOhh5MI6; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680616743; a=rsa-sha256; cv=none; b=vmdyx2f5GAf7Ex5oEM3QTl+ug67b6QCT7L/47m8CCY95aCbj1IfDB3/BaVYlTvGIzaip8m siUwlsR9nxVUsX22+h/nJ9lg8z5sOid99Sj2sBPJHZYThTB6xjq5TZL8ExscOhJnyPLZTH XNzfs99mYHtFSmjEQT+bq9lxYkGSsec= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=52BqLkPH7JK+YVgxAe2fuxeKTYdfJ/oXPMwp85njK/A=; b=wOhh5MI6+I4NGse+vua1IB8f4A hZZCG95HJrdfyK8JqiDCfGDmfPoatJwd6OUBpDjbZLS1rpRg/INo4Zm7QMNbZUmHWr1+FTPGtFy+1 4tXsE9cGWheBFU8hBh3jvu+vwuyrSNhEA0qwcC3LvgvRLxmpKZKvbxXay4xYHkbEz0oy8m2BdpdO2 XfBSQZsFz3NVZ2WSiG8L6CJs9EpamaXyYXMfqe5PifW8V2fCb0gwJ63XmPTArO/gT6m84+xT34D6x AiB3JXxroKqcRT8kYynhag8ZVBv5Avf0fW91FS2TVRCCvnKSgSfVnHl2ZxjaIvJQoz7e2eN9c4SLG /LGz7mag==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pjhBU-00FPcf-4A; Tue, 04 Apr 2023 13:58:52 +0000 From: "Matthew Wilcox (Oracle)" To: Suren Baghdasaryan Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH 1/6] mm: Allow per-VMA locks on file-backed VMAs Date: Tue, 4 Apr 2023 14:58:45 +0100 Message-Id: <20230404135850.3673404-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230404135850.3673404-1-willy@infradead.org> References: <20230404135850.3673404-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C4B2E1C0017 X-Rspam-User: X-Stat-Signature: oq7wu1fdwf6j19e5f8tw49hh9fe9azq6 X-HE-Tag: 1680616743-968674 X-HE-Meta: U2FsdGVkX1/aXHRPNBOJGIoeuLp72tP0ab8o6EHSx/yBg2O+pCK5W1KK2OO/1pUzndDy4aDwXSEanQP9FSHHoN1DT3+e5m7IscjNN4Osw5A8VnX6O2cRKUj9qXNc1SFmOsG0myAYKNR/zbonmb5JxxJdblKfM8CZ0dI+8MLMtyPai2D1SpxN0UXEbEqsjVOteRRDK+K9YKtjuTukYtjwde/DI7dXoi/yOvmVUx6H473fz7MbUUw81950aYppLy3FtIpRT2UkY/mJZXP5Ce2MSHBUqh69WBjU+SHMx3k1jK1XZ2L02iOF6g+Vq1Zj+Dnj2zo1b96zUDydr3vrQmU3BFaRqLaH/QLopXhgbYFta5oJAKZafRfX0QQ2BS7dtrCPgNJPjEYepe/0hzyE/bTi3wYSBuHk09CU/pRxoPbS7v/Y43bFhH0M6sEc92CzrGz9LkMGz0H09WYXlyk5fTAW4eQcVxftuE8wZe1AZkka7Az3WZbzD9J833Rwwp+2KskDcJjK/lsBjoRwwJvAOzBEkgoGPOvL0fZEwxa9KiO174PE7UKJB8ndk0RNw4d+a481fMZQ3mBXUenm8qihphKL1/YGb9DtUFZsAxkIWi81vTqFplOEQXVUn/oE/6P/clVY6i9aYkU3xgG+bCVMM1xO9c6HNHrN2vHnMkP6Tb+WXHcmDvipWejqjWP3WeF6LP+ww0FA0V85qeOGoaHkhffb6Ety5/mXAyJElZxhJoVOtxzx8gPfyWalIl2Io1OxY3raggacKPSl4qX8/gRD8/vziGV0n71Krnpia+w3C92rxVQPCh4F3Angu6Ja/m5l4mGwQ8PWCjNv02G3/aa+NTAMuPzimHYfHgjxrMnOBKO2f/kdTbi1ro3exvnZS+36woLJ4SdRWjxBr6bj0gCtQSxeUUrIpk5w0a0bPUXLQxGoj6ryN3UpGSOLAbZChAoLUOt9bRWII9s2hUv/AW58jgg 1Wf4MKJH FRqOgppErRZG4zSXH7FHfbjNJDPEE0rxZ7vv6sJJGXejttg3lLOaSUDaP+L5SyFL1aE1nqcADAyBH359IwxyicpXcwtYzVcHulQJ1ZAIWA3ivzN4SR0qG/WXMsL7IhRXJFNs9Z8Ut/vxQyhTSXS/SA4lrv0MGnyJptP+qurGd3QCLCAd7bBRunfWyamUKh9rz5ck3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The fault path will immediately fail in handle_mm_fault(), so this is the minimal step which allows the per-VMA lock to be taken on file-backed VMAs. There may be a small performance reduction as a little unnecessary work will be done on each page fault. See later patches for the improvement. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index fdaec7772fff..f726f85f0081 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5223,6 +5223,9 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, flags & FAULT_FLAG_REMOTE)) return VM_FAULT_SIGSEGV; + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) + return VM_FAULT_RETRY; + /* * Enable the memcg OOM handling for faults triggered in user * space. Kernel faults are handled more gracefully. @@ -5275,12 +5278,8 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma) goto inval; - /* Only anonymous vmas are supported for now */ - if (!vma_is_anonymous(vma)) - goto inval; - /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ - if (!vma->anon_vma) + if (vma_is_anonymous(vma) && !vma->anon_vma) goto inval; if (!vma_start_read(vma)) From patchwork Tue Apr 4 13:58:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13200240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39271C77B76 for ; Tue, 4 Apr 2023 13:59:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCEC9900005; Tue, 4 Apr 2023 09:59:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7E1E900003; Tue, 4 Apr 2023 09:59:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF78F900005; Tue, 4 Apr 2023 09:59:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A1127900003 for ; Tue, 4 Apr 2023 09:59:09 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 77121A0962 for ; Tue, 4 Apr 2023 13:59:09 +0000 (UTC) X-FDA: 80643865218.05.D78BACC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 8501F100021 for ; Tue, 4 Apr 2023 13:59:07 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Yw+GAfLt; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680616747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jRN9LMIQY9dyThVaq2oo0Jj1gstfJQImVo8wGgQY6/Q=; b=rX6YTkQVT1g2crD9G7nN/kIXo1cXPrW4gjvVtWakRVuxZVs07ZpWiTmC9JKJcow5WpSKzk LHqcfciDU7YTfezqXFob8GsW2MiiJ65MGENrbQumw81Ryu5niQZYzWZXxP2wAzthTpJXTY FTBV+pMrDwOeIPBQPk0wpIZAiwjKRKk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Yw+GAfLt; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680616747; a=rsa-sha256; cv=none; b=oNGyOwf2dnF0mtyY0sAgw1mbAqPZ0TX+hXM3nW0iwTyDSVDjQuXuV0a8IwIN6KGpvP3nLv O7LU6bVqO/h6UWp0ZBiUKdnJT6dvheEcDZOLIZKoBBNCOZYuiuNT0be9QTcmjdyatbyoPa uEzsCBCJKzU1ibra2C9RbVyeZoT8Hhw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=jRN9LMIQY9dyThVaq2oo0Jj1gstfJQImVo8wGgQY6/Q=; b=Yw+GAfLtiLfrvZGq+OER5Mn4PY +J0pXFeOBou2VBOR4k2CLiBm1tUeMHkv9r0ZhgGnFjFTfK+4BvkUG4x73vi5rXykhGS9VcgC+/FkX mfSd42Q8SVIYoUvx7B34jKjN9JSy7gzP2dNtFDh4TmIUOsJ2bwppQsAvzQvl9rcxQobgu2f+XEzRV X1tipMKhYfhar5o0Mfx0vwkXW0BAGIPHZlC3GVBT1QD5K9g4yiGE9emh9rG0GslClpPdkW6FLbRS8 9wnhkIFz0AOs2gDOASmoWlKVlASpOYQlrQRlKXEvSbe0BoyJ9zmdd6Si5hRfh9pnromJ/fdl3haiu te7v2ZAw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pjhBU-00FPch-6s; Tue, 04 Apr 2023 13:58:52 +0000 From: "Matthew Wilcox (Oracle)" To: Suren Baghdasaryan Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH 2/6] mm: Move FAULT_FLAG_VMA_LOCK check from handle_mm_fault() Date: Tue, 4 Apr 2023 14:58:46 +0100 Message-Id: <20230404135850.3673404-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230404135850.3673404-1-willy@infradead.org> References: <20230404135850.3673404-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8501F100021 X-Stat-Signature: gnru9tg1pg1o79qhrxoa59bhuutwstqh X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680616747-527928 X-HE-Meta: U2FsdGVkX1/qKk+0B8lCzBBuncYWf/aWXdYDMxOzDvOWk/yrX+rqQikkMMSvgeDqiUfCYzeSGQr73wuB+xUp5Z6Ws0QM7IMoKBof6ZgZqEt5lOKBqrJzDbIYGGI3DiEPNvWLLtTryHMUkA9zF+IO98VoSfSNzJbONLHQm04XOaZmiU3+GBPhtrU3TjjFzWz9kpcODxYd73PCmN3nL4mw69ii3e/YAhN0XR4irGjvIicY4pbnOYn4sg3Eq/SkJPMZRXfnff2ZAHSkuFXzE8RoVYbm4aY8t9FIme7v3zBzB0GSMYx+L/4UHzZucqqQTrYbYTNi5goP2EP7wty4YGD48sSU8gp3KFRt5xpuCDwv/Ax7wIDlOwhW/YrsimeVx0P4K/oynGcV8Ol4vmIHYS8SVaN94T/LFtottHFf28VgYpIXsEJq6lABF287sjAvUG9SwpWaHyk56HEJDLvDv3Wv7TpNQR5BAiIbT2MFrelwNkCBcPefEhzMNDvi1zdgTGimAB8ncoRjGXfJs7vIi9tB1qD6S2/Wo6+8UiwqMG9VzFGyugqUiTBj0GW8qa3DYXHhaDwT4Q/DjY53uAZ6unlgK1a36vocr+gKX0uhtUkuY9CwfqvplOWh9UTtZua013cT+fbhElJNi5EZmBpCsBNWXaZy28KOdWxm6Vny29Q1Mgmv4o/6hfCTuh3L4q/LjzkBkMRDFmIGZG930E0ScM9PxP3errxRNRMwSv3tbHo66Gla1CFOvRa9hdc2QRqT5eQBXto0Jkbhys6KMsZ0vVGQFDeoSZG+LkSe2uNLbcj1SlcBDJwzpa8JAJnqxUSvhCr6PyzH9NPuPpJfPsDcQBq3QJu2R0sp4ilkrXyXfWSYBS6rQevomsVxKi9xa/H92rZEZnzgF4JsX4pZi3kAzc6e+Ix+co10EP8VoYcg5vze7onMh934NQtJWQYYmsfG1pZD+abjxx1UxjYg/GxxbXj 4zgx2NTP YM0ILyFb5tK6v1AiWoH4eQbrVlWB/7mHExh+FJ1o4k4d7pP3GsFyn59iTwkegxs/YPpml4ah7TvFLHvzA4EKRk8FXxGodb4mxUJr+VNWUhRCSH67y/4x5spGyGR6gp7TBjA5ubYWXV8xx/l2iV/W1D7UJGgMyKUAVWq31VgGazV9DiY1Bbr54MSPNf61pJMUyFsolt6k5BLSkas0zPB7/F4yYiLwqXea0DKwXzs2Gzfcx67bDeK/4c2qT8w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Handle a little more of the page fault path outside the mmap sem. The hugetlb path doesn't need to check whether the VMA is anonymous; the VM_HUGETLB flag is only set on hugetlbfs VMAs. There should be no performance change from the previous commit; this is simply a step to ease bisection of any problems. Signed-off-by: Matthew Wilcox (Oracle) --- mm/hugetlb.c | 4 ++++ mm/memory.c | 14 +++++++------- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index efc443a906fa..39f168e3518f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6052,6 +6052,10 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); + /* TODO: Handle faults under the VMA lock */ + if (flags & FAULT_FLAG_VMA_LOCK) + return VM_FAULT_RETRY; + /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate diff --git a/mm/memory.c b/mm/memory.c index f726f85f0081..fc1f0ef9a7a5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4992,10 +4992,10 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) } /* - * By the time we get here, we already hold the mm semaphore - * - * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __folio_lock_or_retry(). + * On entry, we hold either the VMA lock or the mmap_lock + * (FAULT_FLAG_VMA_LOCK tells you which). If VM_FAULT_RETRY is set in + * the result, the mmap_lock is not held on exit. See filemap_fault() + * and __folio_lock_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) @@ -5014,6 +5014,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; + if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) + return VM_FAULT_RETRY; + pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) @@ -5223,9 +5226,6 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, flags & FAULT_FLAG_REMOTE)) return VM_FAULT_SIGSEGV; - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) - return VM_FAULT_RETRY; - /* * Enable the memcg OOM handling for faults triggered in user * space. Kernel faults are handled more gracefully. From patchwork Tue Apr 4 13:58:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13200236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E3EEC6FD1D for ; Tue, 4 Apr 2023 13:58:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAAEF6B0078; Tue, 4 Apr 2023 09:58:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A5AF56B007D; Tue, 4 Apr 2023 09:58:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 949AE6B007E; Tue, 4 Apr 2023 09:58:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 840696B0078 for ; Tue, 4 Apr 2023 09:58:58 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5EA64C0DA4 for ; Tue, 4 Apr 2023 13:58:58 +0000 (UTC) X-FDA: 80643864756.21.B57C7C5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id D5EFA14000B for ; Tue, 4 Apr 2023 13:58:55 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=H5QsLN89; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680616736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+X4Ep4ykQThQZtLA9ICZLzgCY3jpfgFwwzOJniiRp5g=; b=b7LYa5TEJbJw3c7sgj7nYaOdRDk1GFZp+CASuj0CdziW8hjp5cnZWlqEfVrZt7ScYMij/z eEJVrEDQ76quV82LcCsc2uqoaQSwu6dqAkEn3wMhMPCGM0nOj7ZIysOAu2OR//2vWhs1Vw 4dLBCzbNQ3SHOodkWI9ClU9mhH1z3r0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=H5QsLN89; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680616736; a=rsa-sha256; cv=none; b=YmEp9P7x7CR2XD3PZA5420lY/Q0P5BSedBB1GZKDKigdYO/U9glRVYcECo95kL6PYTi9F8 vu8a+Mwm9HdyCWGDACbSJEfHscD0rFx9PDyYEMKyrUQ2QPh9hlLLwrGjAGwc+g1u/JG/sp jH10sti/acPti5DRbLZAyo4u6lFz/H4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+X4Ep4ykQThQZtLA9ICZLzgCY3jpfgFwwzOJniiRp5g=; b=H5QsLN89pXHLK2DxF+z8kahswh xwcY3DxZ1JJ63xik1ZPiYMfP0hiCt2hH6DLAmD0RwO6stP30snJo6nQkAIRshQB46ggRt6TKnZb/a +ul0nI/Wjm8XXJH/i9Eu4lF3rCyYODJ1S2QhRkF10bcYOWMk/39U18ohLDwMV7wedlULvpuHvQy/o 6U9l368xFkHT9E1bYat91pOle+g7m42oWvMtNSpjvX9WZx8CYeRo/wjjTQq1RTbxeKwPBhBMI4yfb 3gGodTonrj+IEPRJd/Co6SGLo15XYOrccSYAxLWsBl8W5nG0TPiF/EIhsN8YcU/u863dAT14iRUlQ JBqsXQyQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pjhBU-00FPcj-9Z; Tue, 04 Apr 2023 13:58:52 +0000 From: "Matthew Wilcox (Oracle)" To: Suren Baghdasaryan Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH 3/6] mm: Move FAULT_FLAG_VMA_LOCK check into handle_pte_fault() Date: Tue, 4 Apr 2023 14:58:47 +0100 Message-Id: <20230404135850.3673404-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230404135850.3673404-1-willy@infradead.org> References: <20230404135850.3673404-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D5EFA14000B X-Stat-Signature: qenewz954fy5jccjxp174geme95oz4in X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680616735-721957 X-HE-Meta: U2FsdGVkX19Y7HMQ+oSfEPMTcZL+1TC51b3MQugeG724ItwvkxV5V/v81GGry9zKcBrbrOspnfHPUio8KXp56j07CPrQXV0sgGUmEJzHXiMunZ7xTpnLlTFjLW+vpCfrrS3fHV4sDZvxZekEtMnMttJl0Xc9G+EOwXLmbD+2pp8GuK9ztrRxjokP3wVX5A82LbCawjnCziDL9p7bavg/C6hLYn4hNL+WqSgQYNVwtzACEUq7KG8g5nQ13SlSLTuf8mq+NpNs5zXSj4YuKkT/0Ig7joswmi2JhC0DmKTvHLPUdO4kCpLVp/LJFVfdpZJrBdTVKL7k7crBUZZRDbRdsfBvneE1nRj1tXUxcZ8YhRjNjrpOL4adeUYLqKvHqP8Fjbo66He6fe4MrXjVUmUJNRNMIKScX5dBXZ1y8Zt2TEA0tZWNN2VkSreVxuJXZLXCDHkft4dkbdASBvVdiJKuROoPZUSYsjUNkHkLcNjHtYbnO/pizsPj+adMmDTpJE7v/zWjySrErt/Fd2x3WOLHculhsbNzmbTS/eVqkxAdCfYhyjBcv4NEgu/QBk25w9rLYJQn28AmM1NjLpHulkjRBygHLzeUk7//2a4D380dPEPt8PAt/Gxt/PeHhyXaRj7NtgF4YD/G7Qa3IPFnn6Mn+m4VFiAHZsGpaXz1N4iq0mJ6o0T8FccuPgJgSUUzaaPuOcLrUS1lMoFZimjURF7+uE8tGGJPiodepaz1Q4uXgfPhf8r7o+U7gvmLxN+WaI5Sx3xCrUx5usgTPSZpjjK2luHUe5w8m0UH0pEutxOg0uCpuHuiqEHQaCShx3EoF4iBL9LMdzQEEf96Gljh5uXykwG6glAwHU/+YYyRFWj9pTuErQTkpuTwku3HRmTRgkJN3mj4WvvGs+y7d8ZGTPhpz4TGUzIIpyGvwZL96vh08V7hePtb3+LyKf4VEnlomlDkFD6sn4PU+RnB0eGCd65 Qhbf96lm NPdZwwCtU8lq+f7u7cfmStI1SWJyJ9yRu60RTI+4lNZa5ZJf2DGQugNlhFhwZGahiG5sdbGIWSk+ocfQo474VgtLpBdQjlyWUeks9ik/U5a2i4pbEBWsqRODDg5hb2pgMzpK1Dsc8g0I+6/kavNL1czhqviIHZNTas9qogEK/UcPkHQDbPNYV9lKAOp//pDc/89XyTkxUixAyRjGEXEPizeQqHi9zaR5DpQ7QH5MvBQ0QQNTS5qfL0hjt3A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Push the check down from __handle_mm_fault(). There's a mild upside to this patch in that we'll allocate the page tables while under the VMA lock rather than the mmap lock, reducing the hold time on the mmap lock, since the retry will find the page tables already populated. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index fc1f0ef9a7a5..a2e27403e4f1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4896,6 +4896,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) + return VM_FAULT_RETRY; + if (unlikely(pmd_none(*vmf->pmd))) { /* * Leave __pte_alloc() until later: because vm_ops->fault may @@ -5014,9 +5017,6 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, p4d_t *p4d; vm_fault_t ret; - if ((flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) - return VM_FAULT_RETRY; - pgd = pgd_offset(mm, address); p4d = p4d_alloc(mm, pgd, address); if (!p4d) From patchwork Tue Apr 4 13:58:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13200237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17822C761AF for ; Tue, 4 Apr 2023 13:59:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A84AB6B007D; Tue, 4 Apr 2023 09:59:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A0DB2900002; Tue, 4 Apr 2023 09:59:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D4376B0080; Tue, 4 Apr 2023 09:59:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7ED306B007D for ; Tue, 4 Apr 2023 09:59:00 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4158BABD51 for ; Tue, 4 Apr 2023 13:59:00 +0000 (UTC) X-FDA: 80643864840.07.ABEDAD6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 65F9920004 for ; Tue, 4 Apr 2023 13:58:58 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TxwFEXAt; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680616738; a=rsa-sha256; cv=none; b=VuaLikC8M0FZzzdjeDu/9ty+TuSFbfOeDZUCPH9C1ZS7LHms5KpKj3Qgyfu1hIJfrkWmUc I8R77yBllYW6m1A0rE5lNQn8IPfUMg0y7Juk8Ndv4Mg25xC4+sfuApoEFpgWxpO+UqELif OoG5FYFG42C9xWWA7UJSEOzb0h5GJj0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TxwFEXAt; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680616738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fp3DXW4B3dj0QEKkbnSR4pkaFzIpRXPOad7xMgXDJaQ=; b=CzJhF2jiDMPKQp2OUpQx+VtLXnW2I200XXAr6JjUF3gHF3uH0/M1R1gYdPSQLZ7Fx7lIuf mkssr4rU0/A4znc2EzhzeQ3xi+Gf2YrfJku77lol7KYfyG5C4uL8Qro0oQsCB8OOrGNUGr 21K2A4lNj5u5yGiYkhzhKbroDjBL82M= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Fp3DXW4B3dj0QEKkbnSR4pkaFzIpRXPOad7xMgXDJaQ=; b=TxwFEXAtSCPKXbeHQbszWeprHo VxqiwXK91lJo0UbDX3RXW4XNyqSskCcZ+9c6WpaI49UbctxbS8wZqEs3nPaFmlxhiHDuV8vPlJax2 YyzP6h7vKnv5ar2DQ7nkf41EsuJBzI5y3wKFEZhWVvhRMEWky3yvIorFwMObUfedc0Ox57hBgNG4L hJdS5UNhY9mxtMwjqVdTEK1iqVrDNyxYcuLW4sE/QpfuP1jUoT8MUTWkYQUjLGPjz2slZ/+T3Dmu8 uN5UE/fYsoduKgaE2VcvgZ2054Ku5h1J2ox/V4pURx85nuWl/nVa5VEMNVMnWzhBlMND2LjjWjjuw v1Yks0fw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pjhBU-00FPcl-Co; Tue, 04 Apr 2023 13:58:52 +0000 From: "Matthew Wilcox (Oracle)" To: Suren Baghdasaryan Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH 4/6] mm: Move FAULT_FLAG_VMA_LOCK check down in handle_pte_fault() Date: Tue, 4 Apr 2023 14:58:48 +0100 Message-Id: <20230404135850.3673404-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230404135850.3673404-1-willy@infradead.org> References: <20230404135850.3673404-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 65F9920004 X-Rspamd-Server: rspam01 X-Stat-Signature: 51sf3wfexie9owgwcf1rbbbfotiaxsmb X-HE-Tag: 1680616738-468638 X-HE-Meta: U2FsdGVkX19KO7aOcREWZ45CX0AV/ZFlTyuoCZcF5LQaDpi0kNVxTumxA/REUBb+AtrpCXJtBj2F2Wl1xwaoadRQyisKlGkjsxAfdskYbxx+/BYjwjk3s0Y8C7PWmrEaX8K3AXLqKpMzO9dbzeCmum+SzoOF+MjrKBXn/cDhqUuuTdzGdzjDcu74NLX05GRf0XMjCGXMFs/P2SUu8fPAZhI3SJd2lAYCg4sdZvc0D8d98GNaYah4FwU8h6kClPi3HF2vGy3cN+Xc+B1nzUsRZMEeAdoQR7yXnSVJ3J6iFxrQyyruxzryGfGm5oVRTFN23d0IJwVIrisMgODMaVu0EA3eXkFRrom236d9GDnjiuaCezDCVGiWUNWZosRnrrIkqdcI638XfPkWxdrETl+x8k5sAlxUsXWZ4NxggC3WbfBaeo1n/JMhIUS0rIM6RXlM6yEWcFoLjDSy9FSJfikmn/bXH8LBxvkFxY2ouKSn4F+y0NJkLC7K/baqwDFwkGGFE5yF+THVFiEgJzKS2sRqjMr4iK7DpBd89sbBGOhq3u6zN3YrfIHnnY11uVTXsbsbe7Zp+eW0wanvOFtwzjTTllmSleOy8ziJs0oulnH72f66IRbv96ZHwMkCjE+fkzrtYOTLobphuwPQDIuvxGE99dG1KqEUHxWSyBX2Qx8vQ32zwGxt3BQjzMAGx4RviPPPOYy/sQqKMQytarq8hD7sG9aGf4uPJDTR/bPPnY/NBiTO6HlF4RPM+Y0KEtGLugJrXCnU0w+F2uP/rf/s3KJX3AR9bJvsGWTC9CiUG+gjqytlgbJ1M7Ysv+7yLYwixNoVYu/6a72EbQ1HnkZjp1Y+Qk5p/pp+8r65iEUHIqdC326V15S2idM2QdlQdh1hxTz+4Ucis31/FG4cG9ml6+inR10rlQaTEwM4ep8jQiWxVJZMav45EohoaL3ASpLmorwJyn9vyk8ZWD2kflAKJv/ sAaBNarS bcV72/nrRdH4oCIjgSFmVx9eIQiwyK1eIV6JJh8ua2xacht75MctkJOqOyLfYhr1vJ6AM6JTic5IXEVhDEwnXG+a5rYpy+dsEhZA1N4m4L14GFli1ez/uFR0YNT9Llmi1l3XfNWD7EgSbFqMY0ITxoFwgQKVkubucULLwi7De3kuhAwwU152G00tsZjLboQRcUFMZPrhSURNHYpubxQMHzri/akqoj1ki2grPj645laiPYidich/Zd5+1Vg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call do_pte_missing(), do_wp_page() and do_numa_page() under the VMA lock. The first two gain the check, but do_numa_page() looks safe to run outside the mmap lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a2e27403e4f1..dc2baddc6040 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3329,6 +3329,9 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct folio *folio = NULL; + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vma)) + return VM_FAULT_RETRY; + if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -3644,6 +3647,8 @@ static vm_fault_t do_pte_missing(struct vm_fault *vmf) { if (vma_is_anonymous(vmf->vma)) return do_anonymous_page(vmf); + else if (vmf->flags & FAULT_FLAG_VMA_LOCK) + return VM_FAULT_RETRY; else return do_fault(vmf); } @@ -4896,9 +4901,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { pte_t entry; - if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) - return VM_FAULT_RETRY; - if (unlikely(pmd_none(*vmf->pmd))) { /* * Leave __pte_alloc() until later: because vm_ops->fault may @@ -4957,6 +4959,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); + if ((vmf->flags & FAULT_FLAG_VMA_LOCK) && !vma_is_anonymous(vmf->vma)) + return VM_FAULT_RETRY; + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); entry = vmf->orig_pte; From patchwork Tue Apr 4 13:58:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13200242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC34CC77B62 for ; Tue, 4 Apr 2023 13:59:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D5D8280001; Tue, 4 Apr 2023 09:59:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85F1A900003; Tue, 4 Apr 2023 09:59:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 726A5280001; Tue, 4 Apr 2023 09:59:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6086E900003 for ; Tue, 4 Apr 2023 09:59:15 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2A3CCABD49 for ; Tue, 4 Apr 2023 13:59:15 +0000 (UTC) X-FDA: 80643865470.06.354CC45 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 7F68014000B for ; Tue, 4 Apr 2023 13:59:13 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=it1WBTzC; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680616753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NLYKRwFNq+8FLihqhRGQQaNAsp7ufPwRcv+8u3H1cA4=; b=4Ilpp+lttTSIZIvfyyCXb0c7dLpO/of3lZhs0H7w7kdp6ofT8EylGfLwoqZu9vRccEIfhN xYBM62YFqPpd57L4Shz3viD2K9HNww42f5SU7fBqo3cWsOnES5mAAApvkfUFmO/iHErgT/ e9+PG9gEHOp2MLrHqpS99QZyphpYUps= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=it1WBTzC; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680616753; a=rsa-sha256; cv=none; b=IkoNUlpzSbAcmrvOsWdJOu3tO34HUg6JpRuhl/Ln5fHT7qmCDk8t11Xy6gsaD5DzUmvc6f OPrTcDu+HpaVXcLwV63oJSVlk40bjM5wpwHP2jaAzo4QzWzWyMyC2W5JNE6sc/Dn7K8vEt 2dDQFbLbkvKj/+KexmsRukifX4Ticyw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=NLYKRwFNq+8FLihqhRGQQaNAsp7ufPwRcv+8u3H1cA4=; b=it1WBTzCbBaRNto+t074tt0Ush V1CMwLewtl1Ml7qU5ah8lyoolevYZpl6ajFurNLcf3LgTOT5eRim6LguAdfOxqGY5KEmjEvFKc9f/ +wvpgAB26L2MfoJtXNLnOKrezsaXjg6iGehplYusPKYuFCqJdzS+yQpirPEX3VaC45ImCMI4OkdGP wRk3ZC+AnxvMeyglwYxclUIi1BDH0vOhR1425jfurrFBzJJFYBfKMttWXuXOr9KIPAHI8kX8pNfA/ ciGWPILcbwdfVQwL0k+Z/SoXgBLlOW889RFA/2HwuHRfWOIw942qXzIBiuPRQq1jgZ9GFnnrlZBoU X1ZgHIag==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pjhBU-00FPcn-FW; Tue, 04 Apr 2023 13:58:52 +0000 From: "Matthew Wilcox (Oracle)" To: Suren Baghdasaryan Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH 5/6] mm: Move the FAULT_FLAG_VMA_LOCK check down from do_pte_missing() Date: Tue, 4 Apr 2023 14:58:49 +0100 Message-Id: <20230404135850.3673404-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230404135850.3673404-1-willy@infradead.org> References: <20230404135850.3673404-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7F68014000B X-Stat-Signature: c8xx4otb48mxnaxhyqi551gcufnccqbq X-HE-Tag: 1680616753-633140 X-HE-Meta: U2FsdGVkX1+csjZXDFDFBIaPgKfi2fB1fQ0Ey6NgxmbwDN/u70V8syueehrzKmJjg5vxdSfYM3OTuSCUhVHbz0Me8qohJcr80/l6qxLE+aMrvPB5bon6ABMt8hCD8LHgXDxv/ViL7SinusCUJkMjZQxcQ+3OKS61gI0a8ybGU+vU/bs7SirTaUOXkwttjHNHqNSijn65Tkro1r0lWgnvwK8Euq2Leal95l9iWsIVjMH/8wUXoOIHg2VHUqz0om55BlosZ0DxnETXg+6q7pCKKAv0sa4X6UCkJkTBFivuMHShNRW0eLfgTJ4ovhlUkTCIt6WulsO2HYmRjaNhXqhmO2P+DwZbVXu2/Qkf+mPin9raqLPxY+sLzeijNvMsLSLcVlQbIBAGZMwCyJG4I+/CPbAQWgRPmcvkUjr93ZRLkBUpVfHK7gafq6ezMLJ4QZP9wuiYBsM6AWBhN+CO/uxeOKgWFlpTZjhgc5v+48slFaB0wHMldAV0kLfPUhhO0FIQoqFxqKdWi783OH54S2zcu57QoIZA/48F1l2AfIsp/b7WGKt5JjhKRmtx0VCDshHrUDYpMciMoVGFXh792ifSztFAEfkn7p6jMRkWu4ah1FOluyJOnVwXqqMTA4XI2j1bqk7mGBJzw61OET7nRk4KBHm3f4i7dnfLSs9GXISxv1ENiUKoI0IbZl3rhUsXn/617mcB3ahGZhV3bBGFYoYV70tIlgz+pKEzex5VplMcD4+cRszQINFfxo4ThaiieN5NrPnupvkOoRSYym5GCdF0CCkC9dx8gyo4IfkHOiDCH0IN9wPBEZlNtR+dq+QSzJw5clbicbok+983XyNRjEcAy9iBeBSma3O9eU0YavTwr7EZL0gT43hmH90BMe9jiKQDXuD8nZlFaxSF9SYwaS9/3ZRYRqrAhUllgHba21uPqga9I5uxLCPhzuuiX2YuyHNU48wALOvKHJ6httD6Vho NMb6HN+h lOOpri7z72FsyvOQDJm37ZcEtIsJJ5uRomPb+6MBxyxnmlC3YRjlbzF/S7vjlGW9jgXJYaiPjZ5seNLJZKSF/PUFwdYFNGK6OopXeBh0Ks+kRKNHriL0Z3dTJ93J9n8Box7g195dno0XA2lEGl6PJF0tdXYjpfLtR3bAVpftR1KrVWFxqSddfRNVGTq/AOVK5Wg1jtjaDjPrfdfByqEeVB4EMQ8gfs6ueVAxOh24H627lVQfUamZchImXZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Perform the check at the start of do_read_fault(), do_cow_fault() and do_shared_fault() instead. Should be no performance change from the last commit. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dc2baddc6040..9952bebd25b4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3647,8 +3647,6 @@ static vm_fault_t do_pte_missing(struct vm_fault *vmf) { if (vma_is_anonymous(vmf->vma)) return do_anonymous_page(vmf); - else if (vmf->flags & FAULT_FLAG_VMA_LOCK) - return VM_FAULT_RETRY; else return do_fault(vmf); } @@ -4523,6 +4521,8 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) { vm_fault_t ret = 0; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + return VM_FAULT_RETRY; /* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or @@ -4550,6 +4550,9 @@ static vm_fault_t do_cow_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + return VM_FAULT_RETRY; + if (unlikely(anon_vma_prepare(vma))) return VM_FAULT_OOM; @@ -4589,6 +4592,9 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; vm_fault_t ret, tmp; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + return VM_FAULT_RETRY; + ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret; From patchwork Tue Apr 4 13:58:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13200241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 945CFC6FD1D for ; Tue, 4 Apr 2023 13:59:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E2C7900006; Tue, 4 Apr 2023 09:59:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2924E900003; Tue, 4 Apr 2023 09:59:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15A9A900006; Tue, 4 Apr 2023 09:59:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 034BB900003 for ; Tue, 4 Apr 2023 09:59:13 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C61E3140F35 for ; Tue, 4 Apr 2023 13:59:12 +0000 (UTC) X-FDA: 80643865344.30.DF65AEB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 1CAA74001B for ; Tue, 4 Apr 2023 13:59:10 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hkP38WXd; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680616751; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IgHYdieXRfLvREO+xFvQYpyGVelRpE2byQDY/NWcPVw=; b=lkAF4Je7ADfgo5IRHiilTiBqFg61ecH64P+169iQsleDBPim78QwFZP3L57AbBxxS5GCtc eDXx6jgRdrM3wu8YI4zKdcbKxd7q5T29CcWURT2XsWmBiKzdvfTxgo5OzZMvg/UK4m7Bf8 59pNGwJkzHl0zSZfFMSBUSxsna2FRbs= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hkP38WXd; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680616751; a=rsa-sha256; cv=none; b=cRjY6s/m2sfr8KW+KsRM9Hpxh2j/F8Iay7pIEy7JO7LP63ABupTUBEtO41hyNGDWyDdNZk e/c6U8VteQEnFmGBobS9ovJRc9bRI5mxEJdJyP93nhWg1q+AKxa4Yworqb6bSDwWUF7u2d tmxo2mz6QKFqzI2XuySJMusoqasOK8Q= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=IgHYdieXRfLvREO+xFvQYpyGVelRpE2byQDY/NWcPVw=; b=hkP38WXdxSRQI9qE4hdjSm+MIG dGsRyyxfG7qIxgAkpnHEL3nMtKCJK4cfzXA9camTfntUJN/qZyyI05FnwSdTv3jzAMZ++uIFUj/FG tYeiw5hX2sPahC/yykQOSeiOs+w/+wq3p8d4lI22HJqr5aieEKPQMBDF1fNeYy1Urdg+lUreny2qZ MqaF+0qam1AhUsPqvAkKcxeR7LF2tjcYHY99iYM92L2Bh42AKNUQH87AxBJA8cBViykfIvlUhGxuV +mMA05WyJsgPlB/21C4fBDOd/E8TGta1WocWkCzcVFFpXqODYKZbzvCJXfV0KOAQQsMOSvPPmElnQ AR/B9/MA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pjhBU-00FPcp-II; Tue, 04 Apr 2023 13:58:52 +0000 From: "Matthew Wilcox (Oracle)" To: Suren Baghdasaryan Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, Punit Agrawal Subject: [PATCH 6/6] mm: Run the fault-around code under the VMA lock Date: Tue, 4 Apr 2023 14:58:50 +0100 Message-Id: <20230404135850.3673404-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230404135850.3673404-1-willy@infradead.org> References: <20230404135850.3673404-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1CAA74001B X-Rspam-User: X-Stat-Signature: nmap343daanuzogi3oa4z6xo3a7uon5u X-HE-Tag: 1680616750-570363 X-HE-Meta: U2FsdGVkX1/6V4G1AX12KvuI3RGypL5HCb1uYWblL/Fr4VgWQR+ObhgteWpOsn5y15lZJIyO+/GX5hXpOzGlQygBXO29Tyc9rxycX0hBU2gd15QY88+zkAlmqUPTvlM7Uc5Rnsl+jItMptsktZym7dzUp2qlYJUyP9CMynxQCPk3CpJ07JGaOpohC2E20A7It7JDqNDYXfQxkOWD93O//2Ky4pRNPB2xvMBG4dgX6Icn91RNqWlALZyGxuYbFZwcE8XZB3do09f/x59c3BeI495eXYdfY9XA4zzI+85YpPR+fN4h7hG8yuixeibBI7R3ZNPy/+HvGGp2DfeY4n9JSmIJJY+6Cbhr+gd3g7ngDZsj3tckE6n3yt72JqJ9NENu/NVNfr3cw+co8dUvcXZPXTD2dmuh7wa+GdQ5rmx4vOlFXFIq8qTTcnXqpWu7B5VFVwEQ4qotCGjhiGAn4DN7a0wGQXw089f78pDqK0YRypfSWVwc4Af/vTa1v//EeES5buHJerKck61EAr+VdnTY33MUF5z/0TSnVjWKC6O+DupT+ZMIn0OZfYntbo6NJt1vyuWjx4KmdGiBZQ4HzwX/Q+DGcO/vs8hcYpoxvzP89Fy80fL394R74UCm/yxfRtMgX2rMJsNq3dehKCGQNp6q1e1OjDBqGjZdLT0v++w/TGb816gqGjesfiGk+nR+Yt3Zas1/dHFlt9NJuTGS4zpKGHsTM9sv3sXKWm6so6Oklvc66oAO5vMm5aJpiUZELWpwfumGCANBTSzXgGY/RCGXFSzVVD8Vcef/SXoU/FhDHJwhQupsZSCGeCXDJGuBM7uTvZPzV8YHVieRqAMVGctwBfcDhrmP8Ld0azr+t0MFM0I70/FZtgL5ghFboN10w7u/+i6DhdnnDqCiquGNm9qhulIoL+EUSLIxvomrOz8wFAWxTi+Oq7COWWNqmfj7irhKiWIajEfm4epaycTRdW5 RFK+w17P JQM/lgqxy2fdZLQAD7ZAYNdgl+WY2/rAO+6KaGcwqZOIa+OA6Pv97Gg3YcjnvfF1rDfxyQjdOJZhWxsRMZJss09pT7HU7KkOvWiXV/Zr3NOYw0lO+KX3eyvfMWmlYkHGvvJRMWcJNRU0RzHqi9UPN3D3Lv6vUJHRLxq1EikhLcEo8mtyUCod2G5XNr4mdyTRUjTTH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The map_pages fs method should be safe to run under the VMA lock instead of the mmap lock. This should have a measurable reduction in contention on the mmap lock. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 9952bebd25b4..590ccaf6f7fb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4521,8 +4521,6 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) { vm_fault_t ret = 0; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) - return VM_FAULT_RETRY; /* * Let's call ->map_pages() first and use ->fault() as fallback * if page by the offset is not ready to be mapped (cold cache or @@ -4534,6 +4532,8 @@ static vm_fault_t do_read_fault(struct vm_fault *vmf) return ret; } + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + return VM_FAULT_RETRY; ret = __do_fault(vmf); if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY))) return ret;