From patchwork Sun Nov 17 08:09:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13877795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4D00D68BF2 for ; Sun, 17 Nov 2024 08:09:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8DED56B00C0; Sun, 17 Nov 2024 03:09:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 867306B00C1; Sun, 17 Nov 2024 03:09:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E0AF6B00C2; Sun, 17 Nov 2024 03:09:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4AAA56B00C0 for ; Sun, 17 Nov 2024 03:09:38 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C1B241A0137 for ; Sun, 17 Nov 2024 08:09:37 +0000 (UTC) X-FDA: 82794861366.23.412FFD1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 4BC76180011 for ; Sun, 17 Nov 2024 08:09:02 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tmA8EjES; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3vqQ5ZwYKCC8dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3vqQ5ZwYKCC8dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731830794; a=rsa-sha256; cv=none; b=pom01JZ7GMeAMiy6WwLBVjYQJR4xOHtBvNoHNwPuOKN6aXYycJVj3NQ4xEyfNAtVakUjt2 BGvZyJrlg4JEelwKI1m21RWdbF2thQV/5hv+k8f76kldF3Twk+1MtJMZExCxsdG3Ngwz51 ZAjnxzwARpJJ4M2RXh32xe4jyodIPhw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tmA8EjES; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 3vqQ5ZwYKCC8dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3vqQ5ZwYKCC8dfcPYMRZZRWP.NZXWTYfi-XXVgLNV.ZcR@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731830794; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=T+OFugD+gETNb7vhEAsrG1OgRlzuAtkUoQYa8geci98=; b=kagdGCBRay5x94yzbM7XxyghGxeQ0F4U0t6MgC9Of4gR6QkT2cbRQNOc0jsVPI9Hn9Bd1I yFFCR0zHz9AK8dl3wQU5CpdpajDFZdFB4yTbtNO+ULlBPQeXsnxOUbFCBepMQGtji6xR2U M1Lzt4T6iaCW/p6TFWKWOCRarrHaPHg= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e35e0e88973so3948909276.0 for ; Sun, 17 Nov 2024 00:09:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731830975; x=1732435775; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=T+OFugD+gETNb7vhEAsrG1OgRlzuAtkUoQYa8geci98=; b=tmA8EjESb+lyVTn7RaXvjtiKUMiksbuNJGfeRKRnhvgyuGbEnJwwcMkxlnP7OkFu7k n/xSgbC370GGaie0jlZxn+zDBqMpQOWxeGYHvy9G0ZST07pPc/qEYBxMZ2n8JyyNZXBG 43Yd2HXiH7vC3FfE+Qz27Q4J1DLyYR9TjWqEOx6kCnFPr4kp/qOUTxkpNuh3qu3RBKhv RFCorimKAz9nFciB281rCqsSZ/jndyi3U9I06/rLOdAH6gmuoh0HxaVeoIkBRIKW05J1 hnp0E3q3Qf5SsLi6ylyPC8EHbNLy86SuPjw24QOxZ1bEmM4glrplwBIq0KoNMA5HbKSD 0EuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731830975; x=1732435775; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=T+OFugD+gETNb7vhEAsrG1OgRlzuAtkUoQYa8geci98=; b=SCcTTYdWStg+ASgXTcFpgzCXGczS25j/mN0srzVpKD1jRdIuu8wQUVe8c8/ux9ZIUX 2yHOHrplqnsVy+2b9AkLmgBdkVOkMxEH9p1myd2+khOGVFppB0Uok7HC81vwBB1B3SCq ZeaNYv2JPe1zNEt9sKCetpOJhDNs6z9jG62DaGTWfQ+5AKLHCNn4f/b85RVf4ZtKfgO3 rBqVq+vq3aCZgc14ReXXKUZKPE+BgU291OO3AUUMaHwE0cFrtUzC47r5B4vcAkKoC2VJ pXYXtvjxzhvVZMXkivp9y/TrZgpb/CqoUy0USF9DXlFVBUUau+IyrNbMky8rYEvsssE/ IDFQ== X-Forwarded-Encrypted: i=1; AJvYcCXkRMSgFDeDC73pCuhNgwLbNaej1KzuJIRPS/Ds8l+RKvtorVOZq+Ku/lX7shg9qaBWucH81XWAfw==@kvack.org X-Gm-Message-State: AOJu0YwEuGA8DWTDFE5wzN/FMNgj0X6cbuW7id7jLU4zU2o3KG2bsSI2 Sbt5WC9X+NXdMZI4ed+FqLJvEdU6z6cW+j0mgkzWD33qe2/pP1ImzolxtkUsf4uzagDhHKq08jm Tig== X-Google-Smtp-Source: AGHT+IG4dmb/uQ+GXi1P4l9n6RvmmZk+SVBQpXYBunsXvRdr7iFW+KRUCnhiMQ1KfmEq7maeTeWIGGpGl/E= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:bafc:6633:f766:6415]) (user=surenb job=sendgmr) by 2002:a05:6902:2808:b0:e26:3788:9ea2 with SMTP id 3f1490d57ef6-e3824775f45mr228432276.0.1731830974816; Sun, 17 Nov 2024 00:09:34 -0800 (PST) Date: Sun, 17 Nov 2024 00:09:26 -0800 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.338.g60cca15819-goog Message-ID: <20241117080931.600731-1-surenb@google.com> Subject: [PATCH v3 0/5] move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, corbet@lwn.net, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: xduf7xxrk4ctcqou4o6ya9go81nu1kso X-Rspamd-Queue-Id: 4BC76180011 X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1731830942-42838 X-HE-Meta: U2FsdGVkX1+phy8gDGXtR6mIA32vNENdfjvqA2DitAy0Nc8mnXcMbps0axfhZ6rJXyHeYUks+9dbZs1d4aWu7bYdPmO20m4J5nDeoY2NBym9r0WrDJPF6EUadJ4HSHUQlvC5g8j+Pj5QLcVCsBNqZAI+rZbdWE3ybp3dH04Vp3UoBit7NnpI9PowQ4+VXUas39zZ8M3T2e4yADtIEDlDfPBbp4WawOte92Fui/EJHsxTLJ+/MLO4mcpkC/kY4BdpTY8Cix2jwYM/4fZhS+MRcPoYsLcPqBi3fZ/eNI82kqTMg463Jh53QK7kqjeFppJAtj43mXJVVLX9cZ3SgOcWzg0OBpf8SV0w3fNPmiXqgncfHa61NPpun0Fp6XpoOb/bzkNnO1SiJh9JNrlnX03Px2hgVUtfpu5/WVs559lUh4Udpa4AyenVqUU+eL4vi+qUh84XIHKAo9v4OMENt0SnoTwwpnSotdEUekDa9d49i/+segnAQ5AjSk1kE9RS8GdRfIP+iBF1BikeMiAepLln+T9RorPIWeMgIOdaKJ7SIHG9R6x+YqkBCzhZi/6+mitLBIO1xLUVxk5ViNsemgyzFCDzKOtws0qNmiRSRwjxQ1JWmfHhX+89n5yZs++b138Xo1yX7yEbqIGL5nGAHHyC8Wu6kl0gaoi4uaFh1eH3kIsgd6AlChd0+PBHR/SRa54i3q82gXXu964h6Ji0RJXMpHUlFwx1SsLibn/Jn2WfKarYoVgJ8REPPrN6zVqWvYE16s767GO+G+dPKQO2w8jjRakWzrzdjvul7gsTU94BocBu27gvCWHTNPyMeloqKMpWYny24xfjbgUKOUvuI2pwfOSg7vg/iGIFQlPpSWyLdYHZ54ZxSfUE/mpH2GPDTYFmAnGIb0xDCoRXjbubBNa5cozUNb6KQewMv6Q/VjARNmwifFJhfcOVj8qyJVv03uL3ml0NbwKOuid8FJui2EB o/XbpDui Ftpi+vxlTlFDRrqjQiAVbxcFXPPyvTOuD6gwffg/HemN9VK4nUp0tDY42ZaXCymdgWN1W7sojVBSoKJQNF/P6xwjp3p1EymgLG4gpAZuFEUr17D97GfMKHPSfJVasuiE0GQX5mABZ1V0Ez2E40SPO3YxkbblnnUUhfdmzGkZig5kMp6MC5NTViiOhxnm7UT+P2znWMGWZRRx5fYTzvpbFQiImXysOS7e+Wi7p7suWl9gJruVae4GTCiM87az83ROMfNkO5tboKebOsTZgRmjyJ1BfyMvmY5Lo36HjHoLIKjTr9RC0MyRklnDUdKHgHJIH7dfp82DngU10MIbQ9/Cs/ueKmJWHiux5/MUT6Lu4d5lc6MWTMwqGTVi579h/9/sfhWtPZVeP+u+0OpjPqS9OV7NA1Z6qMXE4N+ddw8Cw0YChkfsyp8UCpqQlux80p9N0E7Ef6C8xf+CZ7YW6giIFTavyybUGmn4DZxbdqUwHrb+fRGJV8QbFP3IxlDS4SL6b/jl4C3DknJ0ixn5XgE7NnceVVVluSmaeqjMes2r4sDwi0tLd+iCFUCHLDEucjsl32qG3rs43EhUUl/MgM7vdl9uc9Nv47drimHqmpn01SK5AQwN5h2CZqtwF8RZhb91+n404 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. This patchset: 1. moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing; 2. changes vm_area_struct initialization to mark new vma as detached until it is inserted into vma tree; 3. changes vm_area_struct cache to SLAB_TYPESAFE_BY_RCU to allow for their reuse and to minimize call_rcu() calls. Pagefault microbenchmarks do not show noticeable performance change. Changes since v2 [4] - Updated vma_start_read_locked and vma_start_read_locked_nested comments, per Lorenzo Stoakes; - Undated patch 2 description to include motivation for moving vm_lock into vm_area_struct, per Lorenzo Stoakes, Vlastimil Babka, Mateusz Guzik; - Undated description to include a note about memory overhead dependency on kernel config options, per Lorenzo Stoakes and Vlastimil Babka; - Updated vma tests; - Introduced vma_mark_attached(); - Moved vma_mark_attached() calls into vma_iter_store{_gfp}, per Lorenzo Stoakes - Moved vma_lock_init() into vm_area_cachep constructor, per Hugh Dickins; - Ensure vm_lock is preserved when clearing or copying vm_area_struct; - Pass mm to vma_start_read() to avoid use of vma->vm_mm, which might be changed by a racing vma_init() if the vma gets reused; - Added vma write-locking in vma_mark_attached() to prevent races with vma->detached check in lock_vma_under_rcu(). Patch applies over mm-unstable [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ [4] https://lore.kernel.org/all/20241112194635.444146-1-surenb@google.com/ Suren Baghdasaryan (5): mm: introduce vma_start_read_locked{_nested} helpers mm: move per-vma lock into vm_area_struct mm: mark vma as detached until it's added into vma tree mm: make vma cache SLAB_TYPESAFE_BY_RCU docs/mm: document latest changes to vm_lock Documentation/mm/process_addrs.rst | 10 ++- include/linux/mm.h | 121 +++++++++++++++++++++++------ include/linux/mm_types.h | 19 ++--- kernel/fork.c | 88 ++++++++------------- mm/memory.c | 9 ++- mm/userfaultfd.c | 22 ++---- mm/vma.c | 8 +- mm/vma.h | 2 + tools/testing/vma/vma_internal.h | 55 ++++++------- 9 files changed, 183 insertions(+), 151 deletions(-) base-commit: 84f7f75fda11c147979984755011c6e2b71e97e1