From patchwork Tue Jun 27 04:23:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13293985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 887A3EB64DD for ; Tue, 27 Jun 2023 04:23:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B6188D000A; Tue, 27 Jun 2023 00:23:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F09C8D0001; Tue, 27 Jun 2023 00:23:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0F1F8D000A; Tue, 27 Jun 2023 00:23:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C66CC8D0001 for ; Tue, 27 Jun 2023 00:23:40 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 92917808F8 for ; Tue, 27 Jun 2023 04:23:40 +0000 (UTC) X-FDA: 80947234200.11.4D567B9 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf26.hostedemail.com (Postfix) with ESMTP id C2B25140002 for ; Tue, 27 Jun 2023 04:23:38 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=SPsLrhIf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3SWSaZAYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3SWSaZAYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687839818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L/K5IT4x+bKsumOYQJ+yi2cT1e5z+UZoZveNoih+cDc=; b=6Q9MYenv/Hz+pf/kTR1btk6Pe3e/CM2zosDQCfwLk8D8guiBNtxntp93C0vGTkaNYxFxfw ugDnEJ8+NlUjoAee/icG7xCwS9qKAqxpnLwSYO+eJT305HmcuftxgoZ5iVlxan7dAM4bES fstWEyyVkYwlJ48YsCkOMR2sYueGK44= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=SPsLrhIf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3SWSaZAYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3SWSaZAYKCKYYaXKTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687839818; a=rsa-sha256; cv=none; b=yAMuP57JzkffNpD1SmCxKytFbEVHjBG1Gf8WH9JH8a54g/5kXfpMYgRXKNkOdwQMTgLEkV Hq7L9Mf0NXw5uh9c9z9gyJAmIDYtWe1i/BRyiVxDNG0Z63HxLFhACl2791UpiNlrPv0LpZ HSi3BARCjEUKD99t4BfOq0IMBY5SzdM= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-bb2a7308f21so5531831276.2 for ; Mon, 26 Jun 2023 21:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687839818; x=1690431818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L/K5IT4x+bKsumOYQJ+yi2cT1e5z+UZoZveNoih+cDc=; b=SPsLrhIf+/FviYu5sa3cDiqhSS9nUO2CjK/9TugBc73fpzLKK7PAw7c+quZSTddlaF vZFybm/r8dGNnIepLjzU4IjRPPNo5Dyn4qcIKvLx1Ra7pU8Cv/EXd8r7DspbiwGRfqHi MU+3kIKCrT0+6JdwkeK19scpvS+xkXgrwyrrwzTvdmNJYgNZXtE3QZuG/vo4kg5AeGI2 bmnmp+IfBgDLtPSxL3LPcMXJgEk0vn02BtNF9KdqeRATk3kahh1G/AEseDofSFqqS0cB IngFLnBfekjj25euZhbjVOsR2M+c4gYV9YLPJZnMZCSFvyWp7HxKSX62DnyRwsvBkXkx 8M7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687839818; x=1690431818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L/K5IT4x+bKsumOYQJ+yi2cT1e5z+UZoZveNoih+cDc=; b=ZMUkXTqvOFF6jMLKmrRBLFm4GfrQue5HE/CwK4YZHPDaokdrhBLYJ+HAYUto2taWWm ZsbnogILHP6EQIUJbZilai6c9KUFBf9K8L3DdZJVG9nRtXQulRtLHIFlb214X7/m7LVB RjL79BraacvT8N9+Jk4si/kvOlGK9FNZ1n8kIzeCuiIUC2OcF6ekuZDPGd5TXlvqbkrT 87MJtXVMTDz3h3iaVd0G/fyWXJ07AuR5P6kN4G4PbXaQ8OOUyV0l49QEUiZ+23TewbES Jw+m2Pro/1qU/CRw7NM3JiyDF62UIoPTANYe7/N71pjHrP4ZEflTVX1E6Yxx7Vrc535e A2mQ== X-Gm-Message-State: AC+VfDw5IReczChjPuJE5UIg81zaWt0ppvGLZXooGSRxQAuhwna858Pe 4Cu9ocxYbfXzaSRoWgCqXbz8UVfG414= X-Google-Smtp-Source: ACHHUZ5LhiNy4/UmnJHGuxwqmDS8M96KwdM5oYuqNif7TJ8z9OhcGrThYwi/uY+L/KRajOJ6sRKDPRQ1ACQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:5075:f38d:ce2f:eb1b]) (user=surenb job=sendgmr) by 2002:a25:e7c2:0:b0:c1d:4fce:452 with SMTP id e185-20020a25e7c2000000b00c1d4fce0452mr3205627ybh.1.1687839817945; Mon, 26 Jun 2023 21:23:37 -0700 (PDT) Date: Mon, 26 Jun 2023 21:23:19 -0700 In-Reply-To: <20230627042321.1763765-1-surenb@google.com> Mime-Version: 1.0 References: <20230627042321.1763765-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230627042321.1763765-7-surenb@google.com> Subject: [PATCH v3 6/8] mm: handle swap page faults under per-VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, peterx@redhat.com, ying.huang@intel.com, david@redhat.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, pasha.tatashin@soleen.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C2B25140002 X-Stat-Signature: x6dk8tqhgbxx6e3ddhfxuibiq3n7ttyp X-Rspam-User: X-HE-Tag: 1687839818-974443 X-HE-Meta: U2FsdGVkX1+rO8Lu3MxpLOKojSaDVjmPKyS3gZnLaNkKEeXS9Oo94+htPq9TKxKed0qRP9C6ujxtFikDeruIJGCx6P3vEj4VWkgbsmn01UBiqYUFZ4ZuK2UcWyWj5zt6McJBdOGKMmxaqiQV18++sLFCpQJ6Tr/frqa2N/j1RrwSmEhnlsjRFv+FmlqCRuIyWSFEgYRKag9CPkog/TK04/niDAuZYUsWb34YJg3HdiZctKLmrJHxItWuhFu3eBrRmDOfHDbdWl/R2llMwir8vHoB9hwaCAgfSOjVYbjofvxa05G0E4OblecvJYeKsvPYsCCQpYIhw+k4fRXaGnENxmmGY6TVVvqGMXY/0g0rYkwoEjkSX+GxE0hwGs6GSPosTxeWBVsGctvwHs29tZjgWXhxa76X6ImGc72cLdcVUCS9onte/uGhqpRE4uF1KFcA8Dz5geowqTkNqjrEc7hzdCsLVMPAnFC4epQ8nvyKPHyQEeaAMmcdVIS9ey8rKk79UhiHcpo0bvoho8zf2DNcy1+IuyLwp5zmhrcLU8/Xp+y7y2fy5yyi/t2QDWVOYWFj/J1BN1UXk+CW2PQahl4nJl4Whj0+Wydlr80NAQcw5vdQ7FJD8FBVtHwJpgpgPGAu+jqD1TCB+GspZRe26i82zR6iTuhzHIkBeZrMF6NGdPMekx4eu7i1SQtdJEEur3JVctt016yEvm8IIxkCEXuOv5GOFzII2a2b3dL4MoE8YfDbxykRfEw84RKGJcyPmaAazL+/7GTjF+2/sWKL1tV40ksOe26sQcSC8e7WHzCzPMh9PWopa2f4VWTLSAEckOu1pIp/eoFQflyWC/a5/0gkzGn/JQaSCZ9Tv5pSxODWakdKlhgcQwTI3zyoYJyc/bXJQ5ltzmTUTSIRzdT00Pyaoh+3ajzHbnVYZJGOBH1/gNKsTU9GUHr3ldOvPv6XDh5THBpXP/92Yt6M1hJNGu5 wbLwaX/x tM5M2tA7ODQnFs2DrBZZ5kwJn3lHw8o3l935vjPlvhdsPzTo6gxvH/r6NKQ6+Qja93/jiC1AJOWQAuBQFe405l0xZIPs6PSuMZfJmUqT2rGyNE0L4+mHPVgpSsRxLWLZRlvstq7i0rPMHXFRQnDWFCT8pAbbkaPIeZWnzhu3/ENZKtp2jou4muSWgAw4tupebCYW+OitaQOvUZsui3iMQQiNjLjswxbNnz8ISLzXf6JHmz/6NNPJQ9d90uK93nr7Zj2SI4Wq0quI8PXQemkca5tAGNQA3K/e5gok8qhgebuMkdFOJmoZEeIBN+gR3m5UuH1BAIZKjOkQ3WtdRWh16OfIwu+4f7kPA0x9FOp0XwnuokMBa1kWiQ/tfP7yb/CtAv2oTt1Zk4RBmUrG9HKg1tEeTRvTXUdtj1tAMn/BfomA7E3KLuCneYXY1lwacEZI7/doMQQzxnjGfplVFv1333IcsgLrun4TE47H+iwfa7txGHda/oKU3a9R8qM7bVtpJ7P+CQeULEsrsYecDQRouK31eWY1HzDpE5NJrf2oNe1pdYGkCZTlABV3HEOYzreTl6rmYrMHfCpgCPzH5i5Dz5Q9FkzPNIwiYVaAyC6LudGAwS50= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When page fault is handled under per-VMA lock protection, all swap page faults are retried with mmap_lock because folio_lock_fault (formerly known as folio_lock_or_retry) had to drop and reacquire mmap_lock if folio could not be immediately locked. Follow the same pattern as mmap_lock to drop per-VMA lock when waiting for folio in folio_lock_fault and retrying once folio is available. With this obstacle removed, enable do_swap_page to operate under per-VMA lock protection. Drivers implementing ops->migrate_to_ram might still rely on mmap_lock, therefore we have to fall back to mmap_lock in that particular case. Note that the only time do_swap_page calls synchronous swap_readpage is when SWP_SYNCHRONOUS_IO is set, which is only set for QUEUE_FLAG_SYNCHRONOUS devices: brd, zram and nvdimms (both btt and pmem). Therefore we don't sleep in this path, and there's no need to drop the mmap or per-VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/filemap.c | 24 ++++++++++++++++-------- mm/memory.c | 21 ++++++++++++++------- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 8ad06d69895b..683f11f244cd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1703,12 +1703,14 @@ static int __folio_lock_async(struct folio *folio, struct wait_page_queue *wait) * Return values: * 0 - folio is locked. * VM_FAULT_RETRY - folio is not locked. - * mmap_lock has been released (mmap_read_unlock(), unless flags had both - * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in - * which case mmap_lock is still held. + * FAULT_FLAG_LOCK_DROPPED bit in vmf flags will be set if mmap_lock or + * per-VMA lock got dropped. mmap_lock/per-VMA lock is dropped when + * function fails to lock the folio, unless flags had both + * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in which case + * the lock is still held. * * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0 - * with the folio locked and the mmap_lock unperturbed. + * with the folio locked and the mmap_lock/per-VMA lock unperturbed. */ vm_fault_t __folio_lock_fault(struct folio *folio, struct vm_fault *vmf) { @@ -1716,13 +1718,16 @@ vm_fault_t __folio_lock_fault(struct folio *folio, struct vm_fault *vmf) if (fault_flag_allow_retry_first(vmf->flags)) { /* - * CAUTION! In this case, mmap_lock is not released - * even though return VM_FAULT_RETRY. + * CAUTION! In this case, mmap_lock/per-VMA lock is not + * released even though returning VM_FAULT_RETRY. */ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT) return VM_FAULT_RETRY; - mmap_read_unlock(mm); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + vma_end_read(vmf->vma); + else + mmap_read_unlock(mm); vmf->flags |= FAULT_FLAG_LOCK_DROPPED; if (vmf->flags & FAULT_FLAG_KILLABLE) folio_wait_locked_killable(folio); @@ -1735,7 +1740,10 @@ vm_fault_t __folio_lock_fault(struct folio *folio, struct vm_fault *vmf) ret = __folio_lock_killable(folio); if (ret) { - mmap_read_unlock(mm); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + vma_end_read(vmf->vma); + else + mmap_read_unlock(mm); vmf->flags |= FAULT_FLAG_LOCK_DROPPED; return VM_FAULT_RETRY; } diff --git a/mm/memory.c b/mm/memory.c index 3c2acafcd7b6..5caaa4c66ea2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3712,11 +3712,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; - if (vmf->flags & FAULT_FLAG_VMA_LOCK) { - ret = VM_FAULT_RETRY; - goto out; - } - entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { @@ -3726,6 +3721,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page = pfn_swap_entry_to_page(entry); ret = remove_device_exclusive_entry(vmf); } else if (is_device_private_entry(entry)) { + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + /* + * migrate_to_ram is not yet ready to operate + * under VMA lock. + */ + ret |= VM_FAULT_RETRY; + goto out; + } + vmf->page = pfn_swap_entry_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); @@ -5089,9 +5093,12 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, /* * In case of VM_FAULT_RETRY or VM_FAULT_COMPLETED we might * be still holding per-VMA lock to keep the vma stable as long - * as possible. Drop it before returning. + * as possible. In this situation vmf.flags has + * FAULT_FLAG_VMA_LOCK set and FAULT_FLAG_LOCK_DROPPED unset. + * Drop the lock before returning when this happens. */ - if (vmf.flags & FAULT_FLAG_VMA_LOCK) + if ((vmf.flags & (FAULT_FLAG_VMA_LOCK | FAULT_FLAG_LOCK_DROPPED)) == + FAULT_FLAG_VMA_LOCK) vma_end_read(vma); } return ret;