From patchwork Thu Feb 8 21:22:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lokesh Gidra X-Patchwork-Id: 13550491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C54DC48260 for ; Thu, 8 Feb 2024 21:22:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 385466B0078; Thu, 8 Feb 2024 16:22:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3359B6B007E; Thu, 8 Feb 2024 16:22:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FCD26B0080; Thu, 8 Feb 2024 16:22:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0D86E6B0078 for ; Thu, 8 Feb 2024 16:22:17 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 73C93161003 for ; Thu, 8 Feb 2024 21:22:16 +0000 (UTC) X-FDA: 81769909872.19.A460992 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf21.hostedemail.com (Postfix) with ESMTP id A57431C000F for ; Thu, 8 Feb 2024 21:22:14 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dAfmTVN6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3BUbFZQsKCNoHKGAODCE9N6CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--lokeshgidra.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BUbFZQsKCNoHKGAODCE9N6CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--lokeshgidra.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707427334; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=zI3aS3+a8yyiJmzrNkMYQ1tuNXiDnBYRofv+RFC6d4M=; b=11HuXOMMd25jrt6TgiiXFPk+TnkJiPJ0IqODqPmIMqvQP9Wt31wrb0v9klsNCX0p5cdn9H JITfvUmPGHhc0TXSIqLvfIPGfutAIQGnStDisSXb99vlnVbakYXyMuxsZngEg4x0gMVxH7 cFUEA/GT736b+j03xBmy1vFEOCCYlgw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dAfmTVN6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of 3BUbFZQsKCNoHKGAODCE9N6CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--lokeshgidra.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BUbFZQsKCNoHKGAODCE9N6CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--lokeshgidra.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707427334; a=rsa-sha256; cv=none; b=7t3W1Pn4uBvr4qlpX/7cGjIL5m8x9dq9iTb8pGmiGHDRAyozbuLN4yV3vgvLOz5RW+Bv2/ c0shzjI03OeYWq5UMRr9uWI9mTtYPs54doYFyzBWlZPJ5HfnCzUg+q/IdFNPitdp/6CWIo CqucrhHmbTgGviBRKBF1knleZpMd33w= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc74ac7d015so419600276.0 for ; Thu, 08 Feb 2024 13:22:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707427333; x=1708032133; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=zI3aS3+a8yyiJmzrNkMYQ1tuNXiDnBYRofv+RFC6d4M=; b=dAfmTVN6pXEcM4VFeMnVyHRGE0KnUKCTpyieHatFxB3JlRM/AoDpKkty8z547TFCX3 Y3rAxKqcmYj4LBXikpMn85sNEPKrxui0uCaOT8HZJQP++PDHZCTO6MFIMbgfzWbuLWtt 5y5DIPLWOzmoPBxThsiI53Bj9k+03VNhLj3r04c6iUrIdj4VC0s1Zq4VbGBCWm9iEm14 GXGvqAsEwcPXUt1l3FB0TAoj0cYB+TJ+ag0Ysq872ZtvooogB0D4HQ2IHvaJANeoTRfS 9lZ+yFHNVwPjBuJ32n4x+kvf1jZL1jm1/jz/O4quXyAx5Wwt8hJGKMx3Zzx+HcdO0izG D0Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707427333; x=1708032133; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zI3aS3+a8yyiJmzrNkMYQ1tuNXiDnBYRofv+RFC6d4M=; b=w9ug0Vketq5ueqHUxNoYon26v4h19y4Ph/5cg0S48sOjYP9R2XGzdY6vyfEkE4fyPC BySQORR3ZYBN6BUpzkYhY8MgDzokN38Yem5tDrHfY/9aosbmm6gWwov0iqI+8U0ftzuV SW9WF06zrGITlq++a8m+pXjJiwU1Gds5Pp+s9AjUmG9/of0xpViiMaCK3P3Glzlxlkc0 iQX0m4FdyxLb4Z2cKrPlrDZQzKIMth2VGM2DyTBCHeFy0O5jwlT5d2lnX/3vA00BeP2K +v5BIGweLGE67nJz6XndlQWP0dMrYm6YrV8bX/Deoph/YE1YTELr7yfpx2GI5sSwCK3p VviQ== X-Forwarded-Encrypted: i=1; AJvYcCUtAdcbloL3h0x7JwJ2o3744G2sAgU6Y1DfL6oJEhCslgMg6HhTiT/V2zE7MKVlawrZZHXSwcPUPqkiI7GlxNDagwg= X-Gm-Message-State: AOJu0YwCTNlv0PVOdf1seTfSUXDNtQj7qvgMrMDFFXronvUDWSVHYHcS cPvSh4PablQpTyoqUQ277Hn+c1gl/gYgg132uYSJJgcXnTJY96EbF+pp2ISlClfuB16ZhX8ILg1 fykLIz1sKA3rUcjYyRquO1A== X-Google-Smtp-Source: AGHT+IHmzoTFZnOiD3iDxXCtJzuelRnEq6YZDdMuEI4bfjEAVm3HliYbUIhglRdmNUXogLEp1m8pKD87RvpGH3QiEQ== X-Received: from lg.mtv.corp.google.com ([2620:15c:211:202:e9ba:42a8:6aba:f5d5]) (user=lokeshgidra job=sendgmr) by 2002:a05:6902:1207:b0:dc6:e647:3fae with SMTP id s7-20020a056902120700b00dc6e6473faemr39614ybu.2.1707427333639; Thu, 08 Feb 2024 13:22:13 -0800 (PST) Date: Thu, 8 Feb 2024 13:22:01 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240208212204.2043140-1-lokeshgidra@google.com> Subject: [PATCH v4 0/3] per-vma locks in userfaultfd From: Lokesh Gidra To: akpm@linux-foundation.org Cc: lokeshgidra@google.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, selinux@vger.kernel.org, surenb@google.com, kernel-team@android.com, aarcange@redhat.com, peterx@redhat.com, david@redhat.com, axelrasmussen@google.com, bgeffon@google.com, willy@infradead.org, jannh@google.com, kaleshsingh@google.com, ngeoffray@google.com, timmurray@google.com, rppt@kernel.org, Liam.Howlett@oracle.com X-Rspamd-Queue-Id: A57431C000F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: pb8unxtnqzfnnph7kesz318r7cpzkizy X-HE-Tag: 1707427334-796524 X-HE-Meta: U2FsdGVkX1+36J7X4lgCx5rfWAaJOmIzi3Vl7RxaxTK6pqNlH0CDesbKewfVyh/MqEXUzCixfTYqfr/38wvDhVuHqELzE5AAnD0akDMIjcyvzhs59hkXE0M8UWxYKXDmuXM5WG0N6tBPBMMSV0b+vMCAmra1JById2Zi0MRBtPh4TdJtBF4ETS2e+88+v6N4BsifNIBnwuA/DdpcSFDjrQK11VOK32UwWscCstyDt8sMCOyY+o9sQMMBPUlCkCKf/hQlBDF70pZDKu+pfFSX6nLtHPezvEF8NrYHGRTzS1dTF2Q4X4cSylCONpwvRL6XQqbT4HoZFBMe0+eYAGKvvycmE4IKaRhoAwP6O7IVGr5JQpc9X5U55RRFuvVmBzrg4xqYK/0MEHaGmXWClgtgwTaarsAz8ZTklRo2398zXS+5jHxbNKJWcQCwV5L/IaYwG34+r0hrrA2C/WQaBb6e2vcSmudWe9oEuSWqfFJrJFIdkb799QN+mEvuzwIJKueyZsbgJ5TdxkoumlmnGVI3t0mlo2jjgX/E8pIrEo65FYEmokiDwFuTiy84HZKi+tMW2lU5YtdxmDhRfC2ne3igtez/gsVx9cumXkowSSkgJN1+BFTuHwacnSHuqmJMq7rfHnoG7sQ5r5mcr1CNTtDu5xFfBIIy7g8JZeRrNkxAbbq6hS8Lqs6Ki3VDdrwI+P2LObE26/QzLsPYwk2AKcf2fbAAXm+ub8L1541VQGCNodctvajARa2xzMX27EafyrID5STUOI8/Kst3z30XxhpLLj2Vyx0FyRigTwcOAiYePIeL4FiyAS7WQQmY1L0NTRbHQaiH5ttOPzxzFYqEn0JHo6KSTQSfHDZjQRWAIj9y+pvz080oOMxDUq/JN5byDZ1+jw0SZBp9nUpZbExBzny62V3cu36d5jo8n4iHatdRlcnzZ1+QJu4dVNptYA+31b+P7U4npMQ2ood5vA8xkuO P1cuGGsV HkbxczjmMRk009HxbIIJ6C3OjOvaUjWp1yhbsHmkfUTdx0j9VQCSnGXGKV81jlVbi9U5W3P6UzAuTU915KQhl9I1gLwUYl8h9kkCcVllKBVQ61nvFlwXmzOdt9cCW84PFliMCTuTNyq1vMi+LDUBf1kkr87U1s6koq1wfrhoaBoT/2q8GkTAIJ+bbWqBk0bQjdISDfTTtx1kAKOsZPFH+2XDaHnllmm02gDAkRQR7GERIaKNNYa60WpDwYB6N2u5pDJ+4Kl93vS65yecsYWLl4tcKHvde8BqwtsyZyVb3dqqnf6mCdok/TvRZF4B7qX/OyJHiBiLebmhBLZ1PdopOJVz8pL1xWHt1OogN1YmOB3cWBcjJKQv7T1Lv0Jult1pp/kfAfhq5f0nhxB/oqyRTCG1Z9C0sezWVbaOM+H+C6E0KCagE4ye/8AVdiPmeCyyBkXpOgi0G4vSKjCin3LcM/syUh16OjqaxGvDpjydGXsyN7HQ4nmGLhkOp9j1zHkwlWgaq342pMG8SXdXhD8jHzUBg+eIgdBeSidlD/Wb5Fem8CDOphoHa09WFjR3A0PoH+n9PkV/dypAfRVsvL/++PGOgdzRAkt3F6+Yve++XYxMm7cnuFO6cmupHKiyBjE0nemDwkfdcUMAqp/uy8xz5lyjgRDQswk8GXgXqNyw8ZHCf4+PzR6Uww3E8Qx3AKSe6vQY2VFs2Ecg4crk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001955, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Performing userfaultfd operations (like copy/move etc.) in critical section of mmap_lock (read-mode) causes significant contention on the lock when operations requiring the lock in write-mode are taking place concurrently. We can use per-vma locks instead to significantly reduce the contention issue. Android runtime's Garbage Collector uses userfaultfd for concurrent compaction. mmap-lock contention during compaction potentially causes jittery experience for the user. During one such reproducible scenario, we observed the following improvements with this patch-set: - Wall clock time of compaction phase came down from ~3s to <500ms - Uninterruptible sleep time (across all threads in the process) was ~10ms (none in mmap_lock) during compaction, instead of >20s Changes since v3 [3]: - Rename function names to clearly reflect which lock is being taken, per Liam Howlett. - Have separate functions and abstractions in mm/userfaultfd.c to avoid confusion around which lock is being acquired/released, per Liam Howlett. - Prepare anon_vma for all private vmas, anonymous or file-backed, per Jann Horn. Changes since v2 [2]: - Implement and use lock_vma() which uses mmap_lock critical section to lock the VMA using per-vma lock if lock_vma_under_rcu() fails, per Liam R. Howlett. This helps simplify the code and also avoids performing the entire userfaultfd operation under mmap_lock. Changes since v1 [1]: - rebase patches on 'mm-unstable' branch [1] https://lore.kernel.org/all/20240126182647.2748949-1-lokeshgidra@google.com/ [2] https://lore.kernel.org/all/20240129193512.123145-1-lokeshgidra@google.com/ [3] https://lore.kernel.org/all/20240206010919.1109005-1-lokeshgidra@google.com/ Lokesh Gidra (3): userfaultfd: move userfaultfd_ctx struct to header file userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx userfaultfd: use per-vma locks in userfaultfd operations fs/userfaultfd.c | 86 ++----- include/linux/userfaultfd_k.h | 75 ++++-- mm/userfaultfd.c | 414 +++++++++++++++++++++++++--------- 3 files changed, 384 insertions(+), 191 deletions(-)