From patchwork Tue Feb 13 00:19:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lokesh Gidra X-Patchwork-Id: 13554194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFCD6C4829D for ; Tue, 13 Feb 2024 00:19:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72CDD6B007D; Mon, 12 Feb 2024 19:19:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DCBB6B007E; Mon, 12 Feb 2024 19:19:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CB646B0080; Mon, 12 Feb 2024 19:19:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4B3956B007D for ; Mon, 12 Feb 2024 19:19:46 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DEC14A0ADF for ; Tue, 13 Feb 2024 00:19:45 +0000 (UTC) X-FDA: 81784872330.28.C347C4D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 3EECB40004 for ; Tue, 13 Feb 2024 00:19:44 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JirK+fzA; spf=pass (imf12.hostedemail.com: domain of 3n7XKZQsKCGoTWSMaPOQLZIOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--lokeshgidra.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3n7XKZQsKCGoTWSMaPOQLZIOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--lokeshgidra.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707783584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=wYp0cW/LgWI9ONT8F6R5Ej86mgi7Qlg0RXGsGKtpVus=; b=7+k6sItYTqjSwiIaQEnD2uG6QUkjF3SygtTEF6rbJfqmZScgQI9H2pSHfJVJvRtqggnWom 8a4DnEEsk8qTf5AHMB6iHp+cVtU76KNkWxjlLkD+jdF/X42rJAsXMCoCFg5br/GASC6tqR HAs8GaO6RrLhw6DDcom+/CtjmvAHnMA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707783584; a=rsa-sha256; cv=none; b=ckBSHmFFESLgOlSV0+tazpO1vEJ9O0xtgV/LU0ZBhfVPYbvxQGebjRFdWKNBmQLayUkNcW dTB4FiAVe2eTxiOZovNEAEKYJKrZkYrdzBDX14EnvK1M9NsDUu0gPesMpWrACFwB7REYNr 7gcdoKTqyVE1B5114AEqZt7T+tTDotc= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JirK+fzA; spf=pass (imf12.hostedemail.com: domain of 3n7XKZQsKCGoTWSMaPOQLZIOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--lokeshgidra.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3n7XKZQsKCGoTWSMaPOQLZIOWWOTM.KWUTQVcf-UUSdIKS.WZO@flex--lokeshgidra.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-602dae507caso65680147b3.0 for ; Mon, 12 Feb 2024 16:19:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707783583; x=1708388383; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=wYp0cW/LgWI9ONT8F6R5Ej86mgi7Qlg0RXGsGKtpVus=; b=JirK+fzAkVrRf3jkeFGf3ppHQZWdRAwizHjshtFXa24L7h6vQvmHZDOfBe76iydYYH 3ut1f21Y99syVWkc8p+dm2GbqpvmONDi16d4e7exNVKg65xsRLqXTFnJKyggn3v3NJ5k boIX60JQKM6OtS12+XPQ8MsUGha8q/D1E1GQborKhL8AgwsxzsFRkOg8I3MEETYL9dRB 4YfObP3cOUHRySTZ0cngzpt8+yc5c5pHiqg4sWXJ2Leko2/mRKuSrZP2nvnwsDHwAtAV RNX6dFuR9JbIPFwDEsxkkkwuFlvhzTmfAnHuodDgBEqjvLxDG37VMWEO13r2gHvBeenu nO4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707783583; x=1708388383; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=wYp0cW/LgWI9ONT8F6R5Ej86mgi7Qlg0RXGsGKtpVus=; b=S2AWM8bq5CNs+3XcXpAacSVenX2MEG0Q2GMW+yd0KEtFf9nX65nPEQjUGqwzpZL5kp 1NK+isWxeMCOq52z8IC32acvsFq5DVvn8xI6b2umy3CfyxJcKCOh3NWQ0TnfS2BOUfyU IX2vqxiHVZvS2Tumw0sWptNpk5cdsWmspesrL3XpBD6LR1FbgkH1eRTdzMtipNIWvplW 1wzCj/zmEOKCqJ8NE4jzhjnCHN3amKjwCKXcsG1sU8K7pLNKRgKFZrgKr98K5ZPc343u ts1D0kasNrm3FcUzBIXDjA2I4gdE1/itltNriHgXISJWLD5CmumOj4ERxrZ86JgCK4UW 891A== X-Forwarded-Encrypted: i=1; AJvYcCXWnDfno9u20sv8ownk7AO1Y2X7f0yNYlhmqtcO3kjT0B94m0+yU3hja9VoE4t1HDWZl+4lGxMAy0L/T8TIOot9D/s= X-Gm-Message-State: AOJu0YyayOYq8knaz3Q9H7gKatfCYxw3wLCuXAJZyjyGl+BTHBtsGRLl nJyrVCkYhuMZF7FJqZoyuKaiV327+6YptCE0v+aRosWaf8PqWA/FDC7AIPRTIJ+hLRxzPbRF0Z8 ScMfQnJwWDlGwIJ/DAOHwUg== X-Google-Smtp-Source: AGHT+IETpVKlK4W4ES+O6uH9bAYT9eFgcT5Mcr7AWYf1RXBnBN+OIklD/nT02bdob/BzELYAfud81u94YEZ4Zw50uQ== X-Received: from lg.mtv.corp.google.com ([2620:15c:211:202:fd80:ef0f:6359:fc4c]) (user=lokeshgidra job=sendgmr) by 2002:a81:a107:0:b0:604:228:7e with SMTP id y7-20020a81a107000000b006040228007emr2397004ywg.8.1707783583386; Mon, 12 Feb 2024 16:19:43 -0800 (PST) Date: Mon, 12 Feb 2024 16:19:17 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213001920.3551772-1-lokeshgidra@google.com> Subject: [PATCH v5 0/3] per-vma locks in userfaultfd From: Lokesh Gidra To: akpm@linux-foundation.org Cc: lokeshgidra@google.com, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, selinux@vger.kernel.org, surenb@google.com, kernel-team@android.com, aarcange@redhat.com, peterx@redhat.com, david@redhat.com, axelrasmussen@google.com, bgeffon@google.com, willy@infradead.org, jannh@google.com, kaleshsingh@google.com, ngeoffray@google.com, timmurray@google.com, rppt@kernel.org, Liam.Howlett@oracle.com X-Stat-Signature: ob7ooq3jt4gsbt419naqnkn9o4ek9ez7 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3EECB40004 X-Rspam-User: X-HE-Tag: 1707783584-795489 X-HE-Meta: U2FsdGVkX1+WspqdUR+X3y/hzBk9hSW3ZbUp0Gic7RYtzbvvx3XpHjPjk2cafqEC7bmwmEWkxAD709WDi16j6ZlFlhK71Psm1PBjSud8C7DGLpidD34v7LgboJN0pY8Lf8swIYGPA3t66HDM/Ia/yWxOz4BWFi6Gr7vysgzmAtn1j+8o2pY9XBZOtP0Ehf5VyWmO0RSrtLPi3dDby1Z6krsSzWFvcaY8eueXLio8D1PAnPBWcJkntXbol9BYcnxBc81I18kZOpb4n7T8tRxeRbOJyfHWF/OKagbCPbpKC/8Z5itBIngbUB1KNHnKu4szslBRdig6yUUnDO7ZFxDq0irLX1z4acaljHMfbXHyNGL83Cte4xaFOZR4BYrCH5GvhUIhosR3MLYUgE4nhstydFhcWv5Fymw0s5WljgpB7I+JT+JDnLFGkqAyoWXvlqNTyln0x4vDpnVrJQbALAG+D1DOKpToOhfLWP3t5RH3siS/FqmUmKOaO1C2vjMVGNX5TSHdKPaWWCl047pSMVIXy6CBE7WoVdCpPmL0ZTJedkMuh9WnMTTqoPVKeMXB0qZX4uU31SrX0EqzVPlbCenOtYB/ADvLdKgsqBS8HRXCxfpDvvBShUp1/uqZyPQenPPyaVGy68Fgn+ol5ww3zdhDBNAvruq/XBuKECJEnewbpqmDnihLagoVLx3DuvO4brPb8im13sYiTpIdZHnMS+uvO7RyfEP8Q4d05bx0JJjFEKHh1M0wCu3Y/VNpdemKsEglWCxn8OdO67q6iS4bLbs+RNvTG3yAHW2FHJAn/rDiR2/CvZcc9jWvz030s+RbssDQiRsl73etCxQgHG/OpNjczB/MRhQ2FtDPxauI7IdXho2M26TzFugAxjITph6Z3TbFbvpZGWKuZbJqWYT+YPmeQeCiXw1PQE8rGhb7Ac0xBt6LbOupdf1qrRTsspvxVz+fZ/9sq/5GYuSp8f1rBL+ +zvhGbxd 6ozx6hQNd29BCltwqfNZm95w31xFVhdiaqL/N+EF5PnZDx8ywnx2lBIj8RzPGgBHqwJvDdEcFUVCsUyVFyYhUK4w3oXZzs/hjtVIhUmcAw/dlLWie3SfUciQOplzQ2ZTaack8gouoQEqVLNVnKVartCYp3IWADY25iBd7vcHpXy054DzBNroTDtYhhkpT/EApcipSR5w2nleqCAcSzFlxpkNszUwOZvg0HbxrdfOA/z9ZMW+eO/erPt0hY0jm0CKo0zyTc/dbbPT5ru1qD+u/Qycm6jpbrUyz7fj0bXnsx3r0YXvx0fKCq1Ir6MQQiBzTUPEMHjm8ihcskw+cSpyvnzsanhAKYX4AaNFnuFfT0Je7uo2ZeMhqlbXy/PhMN1wi09GAJV/9iIc7jhk4VUQ9fCv/fYNCb3jzr0AXvXW7oPp8xli7A/2SJajCx9xRoRg/9oECY2uJav4HlUBE4V3Wh9JAZLt3nm4uLaB32gSpDwn27QCeQJMy07ECRdJaVa9IsDb6DuFrPPOqvvHx775PCdSsnHq4nOh+YUTF/3bzzBYhZCo9IzlvTzcQpbgRtC6xHVrXAppvEo8CsTYVaFQN0d/s6Z6rQEmMf7tC1X0dheZHVmMgd837IVU04X+jrfBBkDhEdiVdymDd39LA/eIhaHW8qZlcNhxwVI/K537NINMGj8DUeplhLzCb1ja9+w8aeO9KgMFwI1qUTsw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Performing userfaultfd operations (like copy/move etc.) in critical section of mmap_lock (read-mode) causes significant contention on the lock when operations requiring the lock in write-mode are taking place concurrently. We can use per-vma locks instead to significantly reduce the contention issue. Android runtime's Garbage Collector uses userfaultfd for concurrent compaction. mmap-lock contention during compaction potentially causes jittery experience for the user. During one such reproducible scenario, we observed the following improvements with this patch-set: - Wall clock time of compaction phase came down from ~3s to <500ms - Uninterruptible sleep time (across all threads in the process) was ~10ms (none in mmap_lock) during compaction, instead of >20s Changes since v4 [4]: - Fix possible deadlock in find_and_lock_vmas() which may arise if lock_vma() is used for both src and dst vmas. - Ensure we lock vma only once if src and dst vmas are same. - Fix error handling in move_pages() after successfully locking vmas. - Introduce helper function for finding dst vma and preparing its anon_vma when done in mmap_lock critical section, per Liam Howlett. - Introduce helper function for finding dst and src vmas when done in mmap_lock critical section. Changes since v3 [3]: - Rename function names to clearly reflect which lock is being taken, per Liam Howlett. - Have separate functions and abstractions in mm/userfaultfd.c to avoid confusion around which lock is being acquired/released, per Liam Howlett. - Prepare anon_vma for all private vmas, anonymous or file-backed, per Jann Horn. Changes since v2 [2]: - Implement and use lock_vma() which uses mmap_lock critical section to lock the VMA using per-vma lock if lock_vma_under_rcu() fails, per Liam R. Howlett. This helps simplify the code and also avoids performing the entire userfaultfd operation under mmap_lock. Changes since v1 [1]: - rebase patches on 'mm-unstable' branch [1] https://lore.kernel.org/all/20240126182647.2748949-1-lokeshgidra@google.com/ [2] https://lore.kernel.org/all/20240129193512.123145-1-lokeshgidra@google.com/ [3] https://lore.kernel.org/all/20240206010919.1109005-1-lokeshgidra@google.com/ [4] https://lore.kernel.org/all/20240208212204.2043140-1-lokeshgidra@google.com/ Lokesh Gidra (3): userfaultfd: move userfaultfd_ctx struct to header file userfaultfd: protect mmap_changing with rw_sem in userfaulfd_ctx userfaultfd: use per-vma locks in userfaultfd operations fs/userfaultfd.c | 86 ++----- include/linux/userfaultfd_k.h | 75 ++++-- mm/userfaultfd.c | 450 ++++++++++++++++++++++++++-------- 3 files changed, 421 insertions(+), 190 deletions(-)