From patchwork Fri Jun 9 00:51:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13273053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9505EC7EE29 for ; Fri, 9 Jun 2023 00:52:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C2DC8E0005; Thu, 8 Jun 2023 20:52:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D57C8E0001; Thu, 8 Jun 2023 20:52:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09D2B8E0005; Thu, 8 Jun 2023 20:52:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F07978E0001 for ; Thu, 8 Jun 2023 20:52:10 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AE4351A0397 for ; Fri, 9 Jun 2023 00:52:10 +0000 (UTC) X-FDA: 80881382820.06.A6C64E0 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf26.hostedemail.com (Postfix) with ESMTP id E210A140008 for ; Fri, 9 Jun 2023 00:52:08 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="gXDUez/e"; spf=pass (imf26.hostedemail.com: domain of 3t3eCZAYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3t3eCZAYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686271928; a=rsa-sha256; cv=none; b=do7rDF/17vHJOBvpwocPm2Ya+sHQAfRyuCkXBExCl+kGiooAK3rZl/6hLi+dR3Jsjp8HPf ux3vAgvzr0bh8UZLpjNBw08mfGvXp179CtGbmrptT0tU5wdQz3Ly9yGB6TfAx1GPmh1bvH LH8dEfsnKqIFnMDvOrZl4iSzQDKeH0k= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="gXDUez/e"; spf=pass (imf26.hostedemail.com: domain of 3t3eCZAYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3t3eCZAYKCNoOQNAJ7CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686271928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PkmNN9bDB8D6ky9B1T3x3QkjIj3UMnGVw+OccLVTcQI=; b=pF9Pblb6pmU9aIfpiCrHnfqgTZXv2Ve8r9GsA4/NGiDTOky+yuMT9U3dQLrrjtX/7/0XAs PIo0/e1Yl+5E9ieQjVYjUrCwl27fCXsV2YRkn166KyQZsQexeM3M+r/3Ko33G9zGJybW9m OLTQ4Brzesxpec/jKimDRs7MDvEH1ho= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-babb79a17b8so1742711276.0 for ; Thu, 08 Jun 2023 17:52:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686271928; x=1688863928; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PkmNN9bDB8D6ky9B1T3x3QkjIj3UMnGVw+OccLVTcQI=; b=gXDUez/eWeoXfvtzBxoMFqX6yzChXfTjDT+EJvNOmhXnnvN012eKsotIpflrgQF8Lp 2fyTd8do5aQagZSZ9UACoo1VzNRvUvy51OShxvvsm8sbDywK/VrQvSv+sRssJTGwZVIC O3ehpAZc1poO5bYiy+cj10MKTJW6cZFiOgpPC3NY9qfT+DR6GY8WwwiHc1OHk8IdiUQQ 0ZsriYgMP54GSTwssMvXRD/MDAyF/nV4E+VqHs3oPq/UGb1xuVtyNqZYuiLhpsjiEGIs wnGJGf8nQGo+PvFSaShaFDV7MeHfsOJeL0uf+hxmZhAZVLSptRntOtdojyaNM4xUuxW5 WCcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686271928; x=1688863928; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PkmNN9bDB8D6ky9B1T3x3QkjIj3UMnGVw+OccLVTcQI=; b=HvEohrHLx9JyynucQlYyo1VXfSV2UnjdvTpVj6Eo8CzLGE0zGzCU7qJzxZBoOsiRL3 2IcBjjeQ5X7xVPrPWYSne9WGT3lCM1htR/Uxt+8jgGIJ/e4ucmkKiT/QTPKcyrg7gGuQ j+4NHjYf3eFeBL7PX9viqYhUxIsBKzpkPgGwEmR6U5+kUgKT1SAkmdoevfCzMJ/qCIf0 l5FA6vEvonbkw06lkspNg+K1bBK85MSHmRWAnvFb4z8mr0f8Bvwp5PyyRvcz7y5bOZQ9 t4ov8Pc0HGNe3zE7XRsCNk286mdP5pHJ1SY2q6pYvjwvJFgnZDeULavWIRnn0xvwG66d BLXA== X-Gm-Message-State: AC+VfDwVyUOZLLxoa1Z484DT86+F8NAedNCjwJ2AZy4Fci53TBmZ4G5d Xzl3rMO78wt0rvwNn9qXKm/b/GBI5KI= X-Google-Smtp-Source: ACHHUZ5ErqzNYue3IsR81EKLpgdV2ofXzmHeKkote+bFDE5P/CfrHNwNpRNHxN6f/RnaMXvb7Ljn6yc+ads= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:c03e:d3b7:767a:9467]) (user=surenb job=sendgmr) by 2002:a05:6902:691:b0:ba8:1f20:ff4f with SMTP id i17-20020a056902069100b00ba81f20ff4fmr691767ybt.12.1686271927895; Thu, 08 Jun 2023 17:52:07 -0700 (PDT) Date: Thu, 8 Jun 2023 17:51:54 -0700 In-Reply-To: <20230609005158.2421285-1-surenb@google.com> Mime-Version: 1.0 References: <20230609005158.2421285-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230609005158.2421285-3-surenb@google.com> Subject: [PATCH v2 2/6] mm: handle swap page faults under VMA lock if page is uncontended From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, peterx@redhat.com, ying.huang@intel.com, david@redhat.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, pasha.tatashin@soleen.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E210A140008 X-Stat-Signature: 8ieeyno5xnwy38xheafgw87fii1bdc6g X-Rspam-User: X-HE-Tag: 1686271928-353998 X-HE-Meta: U2FsdGVkX19JHult2fvo25IkKHxEHIrgTCxSSaxwTDk1DQH/Ahv4sFfxBMryJevRrtfwO+V9AQzUM7coUGTqLGS0PWAHY8CqNyZSpGGccEct7ZME5hhH47ztEvlSipbyzc3YEfl4EWUoCixO2EsGlj13VBBXIZ/o+4i1IjyuCVhM2mHj6tJyfpspizuKbSIyNJZ5G3l2KrqhvJ4y6sKDBVaA4q2kiSco1ujdqb0Vu0ZErZLm7pFiPN0PQAPlTIpUPJE40O9zQInL9WwAVxtqalh3RzaKhuWBoa0YA7GuW6IkE6qgA5MBVk+jp6nRBaGz4bVmz2f7S2aIwFiS1R24Lv/JNweo+qoewAM+nd+PxLD+7eatS8ngpDddaVvkZ+8FMK2X85lzcFDpe6/vM1CT1NBXE1QjvUvFVb86+HuOI9ifTVdwY87HqMXH4G4Ftu+02VfifKW1veXrjocfSTuG8aZ4TzSc33Av5S2KWGaZ2Kz3/+UeoS+HxIT6YTDBdvn3L3iROI58svVx8JLcwBzc17qIo8Izm5R+fZKoHQPb3YJm0lshxCHc8wgrk4heFOsK0duKMhFFRJmp8wwRQeeFQmkYS/0wXdQCcuWIYJFpoYmqy9ydzu+yaKpjfLzXMme07NiUOxnMANinBczkfUoGPisnLfgexvGI/ouGRBhEkj4/BF86rpDtHszDll0vkoWyGjRT/vXhq12iPocEOL5azAFFMh/jLqrBqLemdcCfzQvFoPTw4jhB/wZpTJRvUC99krYRqRIMYLZajOfJrHhlG/ikxe/8k+qcroKGP8m/Z8yt0VH/M4Sp3zrARwdjRip/r9T9n+FvO/UfYnjxqQY51Yj33NB7ERjUenYtq09BwQ07E1bF8KApfnRE1msFOjK6jdWW9N5bweM7c86vjUBt6zZCo1zXgqKZ+2B/TPoNEhDTlVGM71HXaT6/ts/ZHyJEr2I83I/LGs7XUbGFWiA /SVdvr2F 7BhFylClz8GYBc73iiUrR/mw43lERrpa4EiCXkkTDGuzdYVSZ/WfT7Qtz0oQYpmUd4ofPKvPR5b1pUBUZ/nHYAbeSizDXAw0eUyfbvkM9TgYc8Xj3yc8XSf6ePlNKjjjeZIRvkBH6lJsk4k/FRNrmfMZdXmKxbQQI6piFlDPSAoRziHBcl3p2fFaN38Jf+HpbijjHJdpJ1KQuoXCeey4EU1t/ej6I1kt5pTkmlkVmJeCvbcwxRo0ow8bAFrZXJi9IF39lpDsP14ADhRecDK/z3fqLKIywgISSA0Q8/y5VHDykWuS6gkEJ9S6Vj9EZGw5yL7UTC7vmvjgFA7oCFR2lYWwBOo29c37COqfwCeQizX5D7HZphG78DqIBkt5J223F/ZGKV8DeH21HYDkxVdl5IWdnRQn8F5/d06G7bn54DkRgZxoNKQ1QtdJSpCvyoNFTXOYtLcVFTz5+hMSvTsbP5cnrvFXjcs8wgXdaIvDZ9D44eRdbWbwl5gtHoivPuRILQ7KtzhxlJigk/vsDgVglLoSPJjGw4wKA2sKS/5+oplcsTm9PBsklSDL5asGTUk5TjiQ6Fj4OkxglVnmzryZU+w6dpUq9pQFFKf0pLTV2AAp8k+w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page fault is handled under VMA lock protection, all swap page faults are retried with mmap_lock because folio_lock_or_retry implementation has to drop and reacquire mmap_lock if folio could not be immediately locked. Instead of retrying all swapped page faults, retry only when folio locking fails. Note that the only time do_swap_page calls synchronous swap_readpage is when SWP_SYNCHRONOUS_IO is set, which is only set for QUEUE_FLAG_SYNCHRONOUS devices: brd, zram and nvdimms (both btt and pmem). Therefore we don't sleep in this path, and there's no need to drop the mmap or per-vma lock. Drivers implementing ops->migrate_to_ram might still rely on mmap_lock, therefore fall back to mmap_lock in this case. Signed-off-by: Suren Baghdasaryan --- mm/filemap.c | 6 ++++++ mm/memory.c | 14 +++++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index b4c9bd368b7e..7cb0a3776a07 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1706,6 +1706,8 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) * mmap_lock has been released (mmap_read_unlock(), unless flags had both * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in * which case mmap_lock is still held. + * If flags had FAULT_FLAG_VMA_LOCK set, meaning the operation is performed + * with VMA lock only, the VMA lock is still held. * * If neither ALLOW_RETRY nor KILLABLE are set, will always return true * with the folio locked and the mmap_lock unperturbed. @@ -1713,6 +1715,10 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) bool __folio_lock_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags) { + /* Can't do this if not holding mmap_lock */ + if (flags & FAULT_FLAG_VMA_LOCK) + return false; + if (fault_flag_allow_retry_first(flags)) { /* * CAUTION! In this case, mmap_lock is not released diff --git a/mm/memory.c b/mm/memory.c index f69fbc251198..41f45819a923 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3711,11 +3711,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - ret = VM_FAULT_RETRY; - goto out; - } - entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { @@ -3725,6 +3720,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page = pfn_swap_entry_to_page(entry); ret = remove_device_exclusive_entry(vmf); } else if (is_device_private_entry(entry)) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + /* + * migrate_to_ram is not yet ready to operate + * under VMA lock. + */ + ret |= VM_FAULT_RETRY; + goto out; + } + vmf->page = pfn_swap_entry_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl);