From patchwork Tue Jun 27 04:23:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13293979 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EE83EB64DD for ; Tue, 27 Jun 2023 04:23:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E10D28D0003; Tue, 27 Jun 2023 00:23:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D99C48D0001; Tue, 27 Jun 2023 00:23:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C61B18D0003; Tue, 27 Jun 2023 00:23:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B2B468D0001 for ; Tue, 27 Jun 2023 00:23:27 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 821C4A0863 for ; Tue, 27 Jun 2023 04:23:27 +0000 (UTC) X-FDA: 80947233654.30.51A008E Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf07.hostedemail.com (Postfix) with ESMTP id C63A940003 for ; Tue, 27 Jun 2023 04:23:25 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=x2iMPg4X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3PGSaZAYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3PGSaZAYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687839805; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=JWV/DPfsr/X1EdWkyMfUvUzcNXJIr36UfbN56sh3Nho=; b=eN715q6HuTXQ3G/BCP2Xh9z2KdqM4EeUxfCL93VYp3m0capXokj6uo57VPXL6rRALYhXfW dWtMo2Sm9eyTJZnFcd85432p3WSpCBj5xUGX+52OVPkBckGnWXuJL13sbH9ouiR5CEt3F4 19GhKfZean83h5dUG9EheqTR32UvBtc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=x2iMPg4X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3PGSaZAYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3PGSaZAYKCJkLNK7G49HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687839805; a=rsa-sha256; cv=none; b=C62qiZgiXikteUpPdl2SIXv/SZgCjcDIsni4nLUOhoszkW/jy3ioBXrjxLMlSirrO3306A V2f1epNMYAw+GAym8vM2jp6U3pl6XCLkxbcJtEevA8d5W2KwqSBtbmHvy8y5BzuqkE+xcN uuP4JL1eLANPnR6QUe1yBvDqyzffpx4= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-bff1f2780ebso4143176276.1 for ; Mon, 26 Jun 2023 21:23:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687839805; x=1690431805; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=JWV/DPfsr/X1EdWkyMfUvUzcNXJIr36UfbN56sh3Nho=; b=x2iMPg4XRNf8vHFsb9oj4O2ZELGqekijMqu5JGOH1XL1vViWwwWC9MBkqH9HHe4btJ c19XM2XqaHIdWyVgEShFN+FKhTNvVvVMcM9KPGdA2PpIKlKOFfjWIxX/FTRQ+Egzz2Xk iDOHnC+luY1cdNvUBatfG4vVVioTEVkMxkAuGqGmuEGGYxJfP+4hqSMSdMsuJhdGFwBo ha0yY/DE1KDekc0YbbEhrbK/ZfuDdaU+y21gTEGJNm3OiwsK0/m5w1uhZ73NmyB1jFRp VEn2awGEL5+iVwgxI+McDtpysZ6sRcaia6mf8ZdScFiYwaVc51h/mR4Ayui/LnJG+dff b0iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687839805; x=1690431805; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=JWV/DPfsr/X1EdWkyMfUvUzcNXJIr36UfbN56sh3Nho=; b=VhgbLLE6hM565RaIDNPSzv0ADeQqpTBPKIJ0M7NJz4iahboPrf2nGu53Q2q1FrPTLW V++mvETIBRRrPffDQB03Bn+FwaxGfglQ16AqIGmg9MkdlICH6803wS1uEakCi6y3E3A+ OYyEe3mCUh4s86QxDC2wBhBe+JPmHJu2jlv5//i7TCFHq9oIiCSSZSRWfGETHmCxTlqg iaM57jXuiE9NBL1U8z4xMl/mJmRRQc+3mXDh73pWp6REMagsR42XMql/MxYPtPxZN7+w vt4Qlckti3BsWXe/6EmRh2QW8YISX+PkklqVd+9uIgfp3lBjFg/TCilVERDSSnvvIhfK 9npw== X-Gm-Message-State: AC+VfDzIL4YCz3kEt7bj5i8WM5osJGl5Fe3L0qsEvT7qdo3QxuwAEUT7 awErnsMqH9m4Kkj8zYB/FyBDjNbT8Hc= X-Google-Smtp-Source: ACHHUZ4u6ae4giQurDsZzes45mdptlCJHkYuWzppgrpk7yqVJyIQ6870iCQnB4NEEH0e6arLfrBAiMgLNp0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:5075:f38d:ce2f:eb1b]) (user=surenb job=sendgmr) by 2002:a5b:ccd:0:b0:bd1:7934:b4fe with SMTP id e13-20020a5b0ccd000000b00bd17934b4femr13504050ybr.13.1687839804822; Mon, 26 Jun 2023 21:23:24 -0700 (PDT) Date: Mon, 26 Jun 2023 21:23:13 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230627042321.1763765-1-surenb@google.com> Subject: [PATCH v3 0/8] Per-VMA lock support for swap and userfaults From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, hannes@cmpxchg.org, mhocko@suse.com, josef@toxicpanda.com, jack@suse.cz, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, michel@lespinasse.org, liam.howlett@oracle.com, jglisse@google.com, vbabka@suse.cz, minchan@google.com, dave@stgolabs.net, punit.agrawal@bytedance.com, lstoakes@gmail.com, hdanton@sina.com, apopple@nvidia.com, peterx@redhat.com, ying.huang@intel.com, david@redhat.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, pasha.tatashin@soleen.com, surenb@google.com, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C63A940003 X-Stat-Signature: jtuntsq97iduy4xg9bubdyenq4u5wdnz X-Rspam-User: X-HE-Tag: 1687839805-447567 X-HE-Meta: U2FsdGVkX1/aeyqUs2LMMCA17GFa0W1mLJxX4j6MWqqWwK8AEbyxp9Ve+wbWvqZgiVddcITwNrSUZj2IfDmwOQ3VoyvdIqdVJ5CSvj7wYpATZsnBXz1z8okQHjt3HKofqHlzTRSZEWn2CR7tA/bcjr0fusNHE3PjUWafV8C1KAEaNZS5cc5rKszR5c2pGdkG9bK0pI80Ou3YAkeg4rUQqMrYekjdgjqqJsUM92oNHoNhqg3bASBszkJnbKTT1frQLNm5ZXZPqu03+zsiL1lG7+UjQr71M4NlK21Z5B0tA57hqXhV4XEVIDKMYdlGKdzyCc1owgcG61X58pG4nV66VIegLQ4SGKMcZbSZcyAHCR08zxWMMo7+mB7/nPS6dMhQ3h2ukWHH0VxQa+tOWg8vPyd12PUiEHf1CRgAusmtQe9jHBy2rxmxw4uabkMHotGliXN69pGp/7VbaUrF2Lx5OAEEdpXsB8OnoDkOCGFZvjNQWvciUT21HFbPKgPgixpWifEemDYG7ALURmKbH+nwZWA0dQGjFBfffAmAfe8aBXtHfeQh+jThbETiy9Vx2pqnY8bt2Ya5AmqoKabEKy+p9Y9FDc1Qo3M27Q1+SUPJKFR8JgqbqW77zYq/6LPfAjn7ANLCikfInamRP7iu6IdwmYP9AeTBtL6zEm1RaQy5pJZLeH+K7A3MhDBvWTBZIePxbZbnidsGc4t0fXOWPM60xBK/tFrLHT3RAb85iShZ6rDnC4ZrlPTQEgNhEIgS7xQ01j3cWg0GBUaAhq//zos3r9l9MrplLQ6Nj4K2Qk2sU2bgM8cWwzuZMlCivx25K8p/IFcfeQeK1/851trKsiNn6CMMuQdVk4sb9nhqx1xLS1BYtCRT56ghMGcXVHYc5Ci89OX7eHLB7QPTkV+HEc+pt6UZ7zmLmRdvepCvrwDMEp+GnYtyvDyOEi4C+yXiJfvLuW9MJVtQ5tXclu6Liiw 9ofHKmxs ovD+14N6YzkxELgEw3wiCwVr3t94RHxG37+WFeidxbdP6M7MjeJzKqVp1u6qH0Zq7z2JyGg2CNvirJ+wUk5Quzg5ziMJgQMn490+9iX6iRAMO0wVQ7hxlnj8zuHB0N7y3ndhCAkGaYuVo1xsEJMMDakGg9/OA+zJ9lNWrum/e9H7QcRqT0z3EpoymrcEaLa4zSUdsYhHL1O/lYuSHCCLorqqmUmiGp+Ovle3W6+Il/XCuIL247fvckqMyITae6rtLmEgifQG5T7WV92A/IKEiVPVVrmojS77zPaXtAnSXx3aEdrdnidatH+ixR/bfOe7s/RuqIq1qyBWZkdYMl5ORadPcMkr/y7Gkti3J8UUm32M/Fq+ixN6/uCfvcc8CNUe0VEj883bxsWV3Yjscyl/IYp6cv9z2XfmFq8pFn8nUUeNRM6llH9kC/Z1/AasZSc4wML44xHvrlpp7k4JtyO47pR4DT1pqczFiO2zSzzU44RMU08LgE1TvMac95bQnMUWzUZvtX9V/2hhtCBN0Gd74C8kfaCc0IBiOyitocPUuvZZPzfcvi+ryMRdHe0MrDaLGj0I6vkETmox0treuelcx3mdlgkkGCJEXzMgL5iiWxB932UPbRqIbxXYj210a3Nxrv6xf2JW3RWy1xyz1wOltTXF9xwj9xmIQktYwmZWEMZp7RO0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When per-VMA locks were introduced in [1] several types of page faults would still fall back to mmap_lock to keep the patchset simple. Among them are swap and userfault pages. The main reason for skipping those cases was the fact that mmap_lock could be dropped while handling these faults and that required additional logic to be implemented. Implement the mechanism to allow per-VMA locks to be dropped for these cases. First, change handle_mm_fault to drop per-VMA locks when returning VM_FAULT_RETRY or VM_FAULT_COMPLETED to be consistent with the way mmap_lock is handled. Then change folio_lock_or_retry (and rename it to folio_lock_fault) to accept vm_fault, which will be used to indicate mmap_lock/per-VMA lock's state upon exit. Finally allow swap and uffd page faults to be handled under per-VMA locks by dropping per-VMA locks when waiting for a folio, the same way it's done under mmap_lock. Naturally, once VMA lock is dropped that VMA should be assumed unstable and can't be used. Changes since v2 posted at [2] - Moved prerequisite patches to the beginning (first 2 patches) - Added a new patch 3/8 to make per-VMA locks consistent with mmap_locks by dropping it on VM_FAULT_RETRY or VM_FAULT_COMPLETED. - Implemented folio_lock_fault in 4/8, per Matthew Wilcox - Replaced VM_FAULT_VMA_UNLOCKED with FAULT_FLAG_LOCK_DROPPED vmf_flag in 5/8. - Merged swap page fault handling patch with the one implementing wait for a folio into 6/8, per Peter Xu Note: patch 3/8 will cause a trivial merge conflict in arch/arm64/mm/fault.c when applied over mm-unstable branch due to a patch from ARM64 tree [3] which is missing in mm-unstable. [1] https://lore.kernel.org/all/20230227173632.3292573-1-surenb@google.com/ [2] https://lore.kernel.org/all/20230609005158.2421285-1-surenb@google.com/ [3] https://lore.kernel.org/all/20230524131305.2808-1-jszhang@kernel.org/ Suren Baghdasaryan (8): swap: remove remnants of polling from read_swap_cache_async mm: add missing VM_FAULT_RESULT_TRACE name for VM_FAULT_COMPLETED mm: drop per-VMA lock in handle_mm_fault if retrying or when finished mm: replace folio_lock_or_retry with folio_lock_fault mm: make folio_lock_fault indicate the state of mmap_lock upon return mm: handle swap page faults under per-VMA lock mm: drop VMA lock before waiting for migration mm: handle userfaults under VMA lock arch/arm64/mm/fault.c | 3 +- arch/powerpc/mm/fault.c | 3 +- arch/s390/mm/fault.c | 3 +- arch/x86/mm/fault.c | 3 +- fs/userfaultfd.c | 42 +++++++++++++------------ include/linux/mm_types.h | 4 ++- include/linux/pagemap.h | 13 ++++---- mm/filemap.c | 55 +++++++++++++++++++-------------- mm/madvise.c | 4 +-- mm/memory.c | 66 +++++++++++++++++++++++++--------------- mm/swap.h | 1 - mm/swap_state.c | 12 +++----- 12 files changed, 120 insertions(+), 89 deletions(-)