From patchwork Tue Feb 6 01:09:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lokesh Gidra X-Patchwork-Id: 13546496 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D32DDC48260 for ; Tue, 6 Feb 2024 01:09:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 621B56B0072; Mon, 5 Feb 2024 20:09:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D2786B0074; Mon, 5 Feb 2024 20:09:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 499826B0075; Mon, 5 Feb 2024 20:09:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3B0BD6B0072 for ; Mon, 5 Feb 2024 20:09:35 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B1C26A0BA9 for ; Tue, 6 Feb 2024 01:09:34 +0000 (UTC) X-FDA: 81759596268.08.E10C7C0 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf24.hostedemail.com (Postfix) with ESMTP id 11F6218000C for ; Tue, 6 Feb 2024 01:09:32 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Qst5eL4N; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3zIbBZQsKCBU695zD213yCv19916z.x97638FI-775Gvx5.9C1@flex--lokeshgidra.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zIbBZQsKCBU695zD213yCv19916z.x97638FI-775Gvx5.9C1@flex--lokeshgidra.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707181773; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=zFHBvxcNZW2HfjsGIsVhZk87PaG1v642sbVGsClWfxY=; b=4vnYfTMMy58sgOkgx6QNSAsIM3aah6oAHowA2nnYfpJmhFr22KTZlCJ/IFH7WM9d+DIf+Y rxpXvIew9QxJcK9Y+RjVYw9lluyKd72m5sod02uVxSjxSFdbSJfDgPXfFPkVnWrWdPl2EL 0UKd+4ZdEcQZr8XeTJWwYx+BJskcb/M= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Qst5eL4N; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3zIbBZQsKCBU695zD213yCv19916z.x97638FI-775Gvx5.9C1@flex--lokeshgidra.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3zIbBZQsKCBU695zD213yCv19916z.x97638FI-775Gvx5.9C1@flex--lokeshgidra.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707181773; a=rsa-sha256; cv=none; b=YohJW1gCpF+oWcVjfZU5rAwaDHMjwDBV1vMk5CxiSn72WkEipFcf5O37L4UtahfK814X4j tU39toe3AjgTP4A4d5eC9Pld/WFiTybsUmxnOkB1i7g4BdFLmH9qhitYoJTsBVR/pfvseX /43d0bbw/RawxDDXzhvYtlTAPusrABc= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b26845cdso8490245276.3 for ; Mon, 05 Feb 2024 17:09:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707181772; x=1707786572; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=zFHBvxcNZW2HfjsGIsVhZk87PaG1v642sbVGsClWfxY=; b=Qst5eL4NrOSjBY7qHVdHbMO4zJ8GDi61hRqdHglJbo2V5ENR2i/eAHfAFhAzsjzoL2 sbTAgoGmPkG8I+aExxzVtATijbP6okKBae+MaeW0AEtdoXacOEw/CStHHB83hfVpI8EL TGXEkequLqsQqObV1kDcHSzWi6QsNp0gEwHVBm2adbLDJX3lrjUhIFa+nN0OAC7GIBuh gTd2PDed/A2RZD5y+tml04gYN/Gdn+mcuZx/FX/rXBvK6Jl78yPrVX1gPIejGvgTTW9m EqU1eEZuwjJFLrAwS7IZPm1P6b2PCuPea9ic6artFWG4hEkP0peh6tLJ74BcCu74Kh2G puEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707181772; x=1707786572; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zFHBvxcNZW2HfjsGIsVhZk87PaG1v642sbVGsClWfxY=; b=PAKNWyL8GR58jCXL/fJikZGOXF4iGUFtYcFzfu4ISqAFNGO+NvwxiCcqNsCGNd3VNz zBBHMaV4nTnPCYxX3UO8br/i/53SYAtm4QOzP8/EKsqMUf479Sgq0bQZfLdakjNKZ6c2 K0YCviN+/yfyRgnKL/8Qk8q/9RkdRntLP6fDGhBnHQmAL+bnPtJiqZQpKIQOuFsVLtsQ m8ynZ6MPIVmESPiGJCkqhaNVRKEtirdcz1ggB/zRk08Ht6j/XgHksbjCTrw+TjzIxVYg a2W7K6BInDv2GHsoOXBTMdqelPWCedoBC5BrafT9vcBpSTOXh5DznqVmWz6z/FYN18AN fpxw== X-Gm-Message-State: AOJu0Ywerv87UJn88PsQtSgoRoBmNQwX+AkXxxoWyQCMi66cbjXpFj8T SeEqVI9tHlWXENGtJ58/AQcKSYBZ2e+TbpaUzf8Cp6Gw4TDRhUdUZErunklm0mpjBDs8qz7eAZv KNRs06t5Q9dtkywrhiaoVcg== X-Google-Smtp-Source: AGHT+IH1RhvmxdMgiLN/uYEOr6sr5Ko2uXZynT5hRnaY5hwDi5R6HBD+SHyb78d2LA94Xo7Ale0nQeBJWCfcO1Y4yQ== X-Received: from lg.mtv.corp.google.com ([2620:15c:211:202:713:bb2c:e0e8:becb]) (user=lokeshgidra job=sendgmr) by 2002:a05:6902:2842:b0:dc6:519b:5425 with SMTP id ee2-20020a056902284200b00dc6519b5425mr38783ybb.11.1707181772174; Mon, 05 Feb 2024 17:09:32 -0800 (PST) Date: Mon, 5 Feb 2024 17:09:16 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.594.gd9cf4e227d-goog Message-ID: <20240206010919.1109005-1-lokeshgidra@google.com> Subject: [PATCH v3 0/3] per-vma locks in userfaultfd From: Lokesh Gidra To: akpm@linux-foundation.org Cc: lokeshgidra@google.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, selinux@vger.kernel.org, surenb@google.com, kernel-team@android.com, aarcange@redhat.com, peterx@redhat.com, david@redhat.com, axelrasmussen@google.com, bgeffon@google.com, willy@infradead.org, jannh@google.com, kaleshsingh@google.com, ngeoffray@google.com, timmurray@google.com, rppt@kernel.org, Liam.Howlett@oracle.com X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 11F6218000C X-Stat-Signature: itkdfm7z6mho8i8h8riu569ot7rs3txt X-HE-Tag: 1707181772-928974 X-HE-Meta: U2FsdGVkX19rBSHdFBXUwTq/YXxPWUqUzB/4e1FeIDRs3dtekiQGWj+0CPzR0BCjE1nwAyNBkK9Icu4j4XynvnRWYCrS+C+xpfCMUKL4evUJrOxLwu3tl0lU20Cs4ny4Dn+OBPVNexhv8QMkP6rKtFjEHXdZkqvLDQBEFlGkw7t3O2Tt/i9uehX+VHU5xqUGYDZGIEPlmrKEmru6kWi6Yfdi47PkajQP9Ok6VnnqWmYkTthz1xVIafLSnsgFaa9+ejr6ner4ddzYPwK26xWQhZq35yK7v/AFJKkzA0xsFGhLds8uPD6mn/uSwdaIlwHc5af+Hj/89P0k1TUXLgoyYMPHQ5+Pm4oA6ST/as3nR+e7+X48paX/5pENe0nBB5VclWSL6oJS/KH7f/DGuZ8GXDp/rGdn6maJanuZA2GQTLMJJ9a+przIioG+9eYMAiNEXNniuCRbABR415K3hrI1DRrxDi2pTekGoYs9nacIm4rupam/xDHBCUgsc6J14h5ea/WQsyNci3aaGFmTqSKtEoG40tqx4wWnRXW8/1nDLsrMN3ku9MiIz9d5DHliWheuv72jQqs6amDxDvCiPfSwu9rPjnLEOQiSl7G49yKNdAWpQ46B3LKbE4ZS6+IPqS0mcbyAJI/3nHalTUPEIvkv666UsqEKir5EALp4VfpInV6LZWzSdKdgw0J7rdiI3u4ExmEUTjoxrlz9nncxftOE/Dyep9XZDSybjmsEim6veFTj+C8CY3LIUWdB4BWt7YC3k4dzUWjRPWCWuGAkegQJTmtJ+B1wfxIkT7dVYDsCeU/BvqxjCJGiA9PVo8Xjgth6TNbYA8hjlwrbDod5KZTVpl9fze0irmGcNYsEbzo5UMjM/lrWDqhsEDwm8S28SZS391TOxwkDhzN94Yy7WtpkZGq5dYyrmN0Qhh+cduHWsIkzY+qowk+L+TtvI4/6eCoGonRLhZQicS07KEps5Ba HpCabMZA QdgD4yxa0VP7wTr0j87O3MDFReMgg4WWXRKBZ7WLTLZkDty6taHHW85X+cO/nC3ppNlCTI7LhvRggzjx8GY+1HkIGSGOKqzXDSSbL9/m7r26qN6bWKz5O13x1IARCgnrL/BnjxHIt0P+Hc/lILn7mgsgyLCW9xVsYsdBGihaOzsft5C93d/hQVuFIRSriN1ykfZV78S0jgaUHN/S04XiAkMM+WEKIOYtT5XNti6pOehWMB19KOFI+DmqNATmtSyB48+hUGbXICdCVA30RluuvKjEFKKKRomu8s+TOKtM9ILVLDK1LVsZst5na9g2QWiBAd+eatDsP0d6FM7/qK34jjkUPhlHUZzTcXaMVGIobqv+JMH391JIN6WO3DIcBmNlHSqRpzlyv7Dv/1UleKRJJ+XRp0GX6+zuL1OZbiryWYov+Tu+BrdmLPIB/GZ4A/ZJ/YVBULx8y24Q6WSQCPXu/mV5nVyU6FEnKOVesajZu36/noCKWAJ3yJWSdcLl9gp+tP2Jgw2/1S1V3MwlrfF+iFZrXzv1faUvCXfDRttGrqHKBOTH8XBxItLVcp8zGbIrvM9mkKPhHRZK0euFdFh3HTPfyXgXWBvsPdJPtkY96JS8zJHdiZY3xEVYsPiS2jMeAD5dboZLHp09I+se7aPtLqwJCUpCpTCk4lsCIkIcGNXXlIFznuTi7A4gya9X8Xgm09l6n X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Performing userfaultfd operations (like copy/move etc.) in critical section of mmap_lock (read-mode) causes significant contention on the lock when operations requiring the lock in write-mode are taking place concurrently. We can use per-vma locks instead to significantly reduce the contention issue. Android runtime's Garbage Collector uses userfaultfd for concurrent compaction. mmap-lock contention during compaction potentially causes jittery experience for the user. During one such reproducible scenario, we observed the following improvements with this patch-set: - Wall clock time of compaction phase came down from ~3s to <500ms - Uninterruptible sleep time (across all threads in the process) was ~10ms (none in mmap_lock) during compaction, instead of >20s Changes since v2 [2]: - Implement and use lock_vma() which uses mmap_lock critical section to lock the VMA using per-vma lock if lock_vma_under_rcu() fails, per Liam R. Howlett. This helps simplify the code and also avoids performing the entire userfaultfd operation under mmap_lock. Changes since v1 [1]: - rebase patches on 'mm-unstable' branch [1] https://lore.kernel.org/all/20240126182647.2748949-1-lokeshgidra@google.com/ [2] https://lore.kernel.org/all/20240129193512.123145-1-lokeshgidra@google.com/ Lokesh Gidra (3): userfaultfd: move userfaultfd_ctx struct to header file userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx userfaultfd: use per-vma locks in userfaultfd operations fs/userfaultfd.c | 86 +++------- include/linux/mm.h | 16 ++ include/linux/userfaultfd_k.h | 75 +++++++-- mm/memory.c | 48 ++++++ mm/userfaultfd.c | 300 +++++++++++++++++++++------------- 5 files changed, 331 insertions(+), 194 deletions(-)