From patchwork Tue Nov 12 19:46:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13872892 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14E10D32D97 for ; Tue, 12 Nov 2024 19:46:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C9F26B00B6; Tue, 12 Nov 2024 14:46:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 97AEB6B00B7; Tue, 12 Nov 2024 14:46:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81AA96B00B9; Tue, 12 Nov 2024 14:46:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5AAAE6B00B6 for ; Tue, 12 Nov 2024 14:46:46 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D883B1A05C2 for ; Tue, 12 Nov 2024 19:46:45 +0000 (UTC) X-FDA: 82778474772.04.0D58927 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 48C3A18000B for ; Tue, 12 Nov 2024 19:46:02 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TEpaduYC; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3o7AzZwYKCBQCEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3o7AzZwYKCBQCEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731440627; a=rsa-sha256; cv=none; b=4R7ABvZMVY8RxK6h6yTCZ3PhRx8mzNvt2vi5rQxhIMwfkihGfzkMRgtIZ5vi42yfC/aBZe vbdREVja3CL0edT3FXgXbZY0g2rKSW4f5M3gXNcZbyga1n9eONF0JqI/rMpuhEnut3rO5i zt+mMEPpy5nTlf7WxpkNbzPew6wjBmY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TEpaduYC; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3o7AzZwYKCBQCEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3o7AzZwYKCBQCEBy7v08805y.w86527EH-664Fuw4.8B0@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731440627; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c6L2TsxMW3xdptYp+SxjhxiQb9Rn3imOm+Hs2oG6vZc=; b=i69qkW8/NH/8K9MUF+LrYoMEsSwvWNHzHSzHEPx1ak57S8PqfMf2qWLYsgMAmVmy7Hjt0C nvOFfeMitL8zRz5WIhvkEAgutH5+TzqRPOO8Ds++3Ht3OhzypdBotVNM+bgsFaAJoVSK9X wM0ZaDPZIxmZcrOrrSMV9mOyjFB/4X0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6eaae8b12bfso96127557b3.3 for ; Tue, 12 Nov 2024 11:46:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731440803; x=1732045603; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=c6L2TsxMW3xdptYp+SxjhxiQb9Rn3imOm+Hs2oG6vZc=; b=TEpaduYCrZv3qq7pCqlLlBnJmpUTA2+Ml2HINeIzU3tGTguPLsO9GS97GTCRGIVzJm xHYESgJso94Gp4Fyx+2Av7KIjMcvGLxlztYniKiQGxIFnw93mxaGVzPzXWt0KHGCstbt NZSEA03OwEl02w51Np5MJ+i56Z0r0Nb2Jcg6sDfM9fm0xdKfnec6QWrQS/82wxroiiQN bPnaGlB3Ii1wa6q9FX7X6H+EFKZPajiL1CyjgXy2BkG3iYCX58Xg8PCl+6n9pBGFz29b f2Kxvcc3lSlLlQu/n4e5ap/8gfkTAf2TLj9hvzLi9qXR9FBQ7dLLO/5AhoKfK/eEPxSL yX0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731440803; x=1732045603; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=c6L2TsxMW3xdptYp+SxjhxiQb9Rn3imOm+Hs2oG6vZc=; b=fJ1cMYo15i2P7UyrCr6CWeaZ59FMt0+ga6yIPJLRdlXuu7JE4aw6N00GU4RGY6iEQA XRD1HdAhpNFe9DgJDyhTfJ6JvKoXi3PC1vtT75BJ/eNSSyPPsE9Uq0yyYkR/bPCCzO0S W5Os93GRFEWRPU7yorxaSl4RlqLXyaknKAcadfn4Hj1U+Y5NYr0O1nZOH/dU4ErLgKtx OdnqYij30bTEvIG5l/VFbV0VgQspx+d84VSKRUVMJVxTvXaMy4l7q6W7oAKht48pI/6Y 4ubltndpg8tfrApq5nnBMAaS7YxnTkaX1s1KvDWbKt+I/D8+XDJLA3nehltRMEDfBJ8a VDYg== X-Forwarded-Encrypted: i=1; AJvYcCVJe7YrCsw6Fw0GEKL++FriQzcuW5tHg8i1J4IaubwSEqr83ssSPbIG8Nmv/UzUboMCDP4ibrdf0Q==@kvack.org X-Gm-Message-State: AOJu0YwzKwvyfRCEpQJzvhRGqBpcOTNLgu6/1UCOVZr692qWXV6Cb0Pr /+xKsHQARTlWeQZM4v3vkKbG6VBNGn1XuWXqUh0sShbO4S0zg5XlBIgM5OhtkovJXlFVCQSpkjM pnA== X-Google-Smtp-Source: AGHT+IH3ogaKA/6SIpqGiWEzmqcQ3qZ/Nx5QjPItSCKNvMM699jDmgRp92hUZbCEx3dlegBqK4JNX36Bnsw= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:cad5:5dc6:1298:ec42]) (user=surenb job=sendgmr) by 2002:a05:690c:6e88:b0:6e2:371f:4aef with SMTP id 00721157ae682-6ecb344400cmr3697b3.3.1731440803073; Tue, 12 Nov 2024 11:46:43 -0800 (PST) Date: Tue, 12 Nov 2024 11:46:31 -0800 In-Reply-To: <20241112194635.444146-1-surenb@google.com> Mime-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241112194635.444146-2-surenb@google.com> Subject: [PATCH v2 1/5] mm: introduce vma_start_read_locked{_nested} helpers From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 5mdb7tfzkyhujmbd9q3oquooyof7uu14 X-Rspamd-Queue-Id: 48C3A18000B X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1731440762-108786 X-HE-Meta: U2FsdGVkX1/Bp8r7OPX/1M5Xrt8bvQBCzaDOFogZOStgY26yPNptl6dOS8ezYBm6L0TJ5+qBrFK5/Z3Vfhr6nTiiytMTJXWXWbrXXVOT6wZ2/dqng+e6cyPerQF1VN8hcLV1bIyfWiZf8HqS68XIiRXiaHwOBagr0FZdSKTBAEdGLtj3W426+mr2bmFbNRpaQ9f4/Zv0glOq2qCFc8vh6injyc6L/1TbB482wy57+Gjpa/QHEwOwXWGvxKdO649OiH6aFD8PhLofDNEjg6JUgfrKcaaRRAQuo3UQL/LZY3xs+qIpawB9/npEHQ+R/rHAKzbI+0mMyISoqvHMKqIC/0bad82uB1wCH3JaRJzTbzgtFLkyYL1WoslmTzOHcCSZYP3fR9XaBWCXWNtvYMy8IcYNJjUMf26AQj+ADJQMzO6lqsPNqR9aTfNji1aH6vNM7FZsepjgsBTfwih3A0sDf6LSBxo2I+oVpXWr46W48ACIdu9SKuxqq5ksdavKbzVX/s7Rvkm/v8T6kH3EnBscNF81INRLHr1a9lEmKXejVKuNW/CvsQ5cxkHXiCwBdrkLkdt4AmbEKCuyGlRE0/m+3Dk2/mCXxtXW4Dxt8ScGB0rFefiwkxYHdMcw5+WiHZMbC/N54J32+5nuw5MaKRaKAa2X/ZphQUM3KSDS4+z0Wj02wJbEs2T8SkPqFSqZADosDqf2TFn/OmtTGvo3l2O1sZWMRh2aqOiRZTq5tRfkoK556X5YTwCHymRD7ygz0n+FIc+gxmxusp6isa4O58mujWCSwj1HQvJnpDJyuPGo7A5bm2feog1rcfx3vf0/Xex64fDDtHvVV64BCbRvxKsX/fPju/CzmAkROCKBdMrk+PddywkDosLI7+Rk1biIx9YU2pyftlegC+5QkTFPkTKeEOBVReUH3d2s+zuiE49yzS5fGu7Rpo0OZ/tEugstCjfLITcqY338uqkOEspjbaI EvGhD32l b0gMFpDro/fb4WPK4cfm3EVNDGHghGJyf2CdnGPxCK/SJIAri4w81bovQNMsHK8KMn/lRKRmOXLa69qh9YFpWou6Fb0z/nJJYPR3bqc3m073VyQg8xD+YzXCT0DaEqMMIss1Pc3qXzqjjyD3RoiHTcvP+IXswosqmS8njtxA+ofkKC6aexK+DnQOhEa3f+DwXR7JNDng9vdoquKq43hChIB0WK/9qhDDmOwEfmJb5AaPbLS3TtZOnsoANCYv4V4PPsWHudzWIayKZohm9jqD6eHOdW5xhZJ0YVQgnfp8XdXrFlDaZlzphFy9gCoDjubnrlAPzV8eEP5BEEaEdzqFg0Nxa+AI3RgD/g5WgzYj4oBVAox6ckZ6rnlzEftwAwu/UJOlC29JOFeWVp0AVHlGqnruf2rhVXVukM81xJwYvt5Nk6wDCLjssTZa205t7tc0hdmkyR0/Tm4YZSlyqc/gUd48O6E8YS2KSnqfO7h0YQpEAnHKRbHIyRvr7A0m0gcbITnl2wlE682Q93jWGOSHVa9+rnV6rSE8/DJnc52PSG1Lk6Zaip427kS9xNUThMVSB8GM7mYpXYxqulNIVXeF7cpCNNQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 20 ++++++++++++++++++++ mm/userfaultfd.c | 14 ++++++-------- 2 files changed, 26 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fecd47239fa9..01ce619f3d17 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -722,6 +722,26 @@ static inline bool vma_start_read(struct vm_area_struct *vma) return true; } +/* + * Use only while holding mmap_read_lock which guarantees that nobody can lock + * the vma for write (vma_start_write()) from under us. + */ +static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) +{ + mmap_assert_locked(vma->vm_mm); + down_read_nested(&vma->vm_lock->lock, subclass); +} + +/* + * Use only while holding mmap_read_lock which guarantees that nobody can lock + * the vma for write (vma_start_write()) from under us. + */ +static inline void vma_start_read_locked(struct vm_area_struct *vma) +{ + mmap_assert_locked(vma->vm_mm); + down_read(&vma->vm_lock->lock); +} + static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 60a0be33766f..55019c11b5a8 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -86,13 +86,11 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm, vma = find_vma_and_prepare_anon(mm, address); if (!IS_ERR(vma)) { /* + * While holding mmap_lock we can't fail * We cannot use vma_start_read() as it may fail due to - * false locked (see comment in vma_start_read()). We - * can avoid that by directly locking vm_lock under - * mmap_lock, which guarantees that nobody can lock the - * vma for write (vma_start_write()) under us. + * false locked (see comment in vma_start_read()). */ - down_read(&vma->vm_lock->lock); + vma_start_read_locked(vma); } mmap_read_unlock(mm); @@ -1480,10 +1478,10 @@ static int uffd_move_lock(struct mm_struct *mm, * See comment in uffd_lock_vma() as to why not using * vma_start_read() here. */ - down_read(&(*dst_vmap)->vm_lock->lock); + vma_start_read_locked(*dst_vmap); if (*dst_vmap != *src_vmap) - down_read_nested(&(*src_vmap)->vm_lock->lock, - SINGLE_DEPTH_NESTING); + vma_start_read_locked_nested(*src_vmap, + SINGLE_DEPTH_NESTING); } mmap_read_unlock(mm); return err; From patchwork Tue Nov 12 19:46:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13872893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90BB0D32D97 for ; Tue, 12 Nov 2024 19:46:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A0756B00BF; Tue, 12 Nov 2024 14:46:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 24DD26B00C4; Tue, 12 Nov 2024 14:46:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C8BF6B00CF; Tue, 12 Nov 2024 14:46:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DE6B56B00BF for ; Tue, 12 Nov 2024 14:46:50 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 56D6A803E6 for ; Tue, 12 Nov 2024 19:46:50 +0000 (UTC) X-FDA: 82778474772.15.503B01D Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 2C23E20025 for ; Tue, 12 Nov 2024 19:46:04 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2FYPOryb; spf=pass (imf13.hostedemail.com: domain of 3p7AzZwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3p7AzZwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731440635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sF1WO2dwupFGo3hDXc/ZlrjGweMpYlCC+167fV/bC4w=; b=0uX0YCInJjSQlX0nFgJgqi4ULy3fKhfi8bG3neWAMH4Mn/uhUyIqCv7SUmkYGH8npLcKVh f45rxK0WHupJ8HIh8AM/C5PckBR6t0wAdVlFyF1f8sHVh+D8GUKJCRRECV7Yl1fj6ncS9N fz+8wcK0vGsN194EThGezxNquAxaubU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2FYPOryb; spf=pass (imf13.hostedemail.com: domain of 3p7AzZwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3p7AzZwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731440635; a=rsa-sha256; cv=none; b=TjoLaFOnyGr/RuBsvA1Nwa69Ccp9Ptd4gKimACvaFUm66VxfGcf7jwwXBufM5c/mvsJuvR X/o4ZvH6nf4BAjvOhg6Hmav9N9lX3MKeug9rvFnz/UavMjIK/qRK1hXHnbZiEVWc6T43A+ rsGM6TG/0fTB5xODCvzndj1gtXXZw4Y= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e38fabff35so55584017b3.0 for ; Tue, 12 Nov 2024 11:46:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731440807; x=1732045607; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sF1WO2dwupFGo3hDXc/ZlrjGweMpYlCC+167fV/bC4w=; b=2FYPOrybbSdRZdDEOFgGppeAoT5iA4uRG9DetE8MSLpb7IzlPdeDrsHaYR+zPHOa9V rL8uqugvlN7ZfiMJpURi9W1QvdK2G8VLpISd0xF8cdyJ8GtOkyv9ZcjosaJlqaoweoKM QXR3tBc9z+vaW+ZhOPi48bi7wXzBmxIMFKrQJ6MEgTCL0MEj/nkHJYPQQxgtkbPbrKzv znwpkeZriUOl1I7nEooEZtLIRl/24tKfZ6P+nSpC9MjUc7VEsVh2wW19frOtRGmnsfK0 XhfYr6SmpCxxZJraIB4OTpf6tuNZXv5EsbkYB3sNCvzwZfrAPKCicP7TgI1RReKnT1gJ 0rbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731440807; x=1732045607; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sF1WO2dwupFGo3hDXc/ZlrjGweMpYlCC+167fV/bC4w=; b=U+FI6KuzZ1BxjEgSZykZs4zExe5+pfQ6rrOrzoHKrE/z8JLYfCn5101Fj597SuAX7c qFbRXHfBoA1KpGs/5JRXBTUyDZ4wfFuDEH4glGwyJAJtgLQHAQ+mik0uFJPlg6h/vPLH RWs6mOcM2RohPl34C2nbbFk0DvBnp5sXjCfGwUsGGrotQt0Q5gT7OiCdhHtY1MowBGEv 5QzfyNDi9vkUkHWqfrhUPCkdaXdDwK8wlW7ilkTyeW3z5adcrBYXkj+72ckLFnXnZxo0 /SOQFLgvIdlfTzC0PRkbv9WdkfWRkWZ5ty3nspxiLBewrhrA2L1IrZLABVmVI04ynY6D 5siA== X-Forwarded-Encrypted: i=1; AJvYcCXv+JNeXhU4dQ9GGA/0CSetKVAbv1VFakydp+V9Au2aGXrI32Xly/UyJP2Ad2EzXU+DviNlXbL7yQ==@kvack.org X-Gm-Message-State: AOJu0Yxo5otMc7AZuHGSmqLcV5xWiCKBQJOLng2KUG2ULGvtPePL6mlE PTFX8rin6TG2YaWx/qui4x9BIWpBC3Zu28C+paAwnev+sV/YrLDG2PMcfdmm6Xp3HJh5U45FRp1 fuw== X-Google-Smtp-Source: AGHT+IGT7bUbztUxwofDze+pdbzWP41mkYNy17hbQqt2HDfFo+siQLTPoStIHl62MnUeX+5yshC7tcTeIN8= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:cad5:5dc6:1298:ec42]) (user=surenb job=sendgmr) by 2002:a25:3302:0:b0:e28:e4a7:3206 with SMTP id 3f1490d57ef6-e337f8e44f5mr69931276.8.1731440807574; Tue, 12 Nov 2024 11:46:47 -0800 (PST) Date: Tue, 12 Nov 2024 11:46:32 -0800 In-Reply-To: <20241112194635.444146-1-surenb@google.com> Mime-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241112194635.444146-3-surenb@google.com> Subject: [PATCH v2 2/5] mm: move per-vma lock into vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 2C23E20025 X-Stat-Signature: w1whw356weumuyyydq7ikq4n9rgzsfmo X-Rspam-User: X-HE-Tag: 1731440764-14944 X-HE-Meta: U2FsdGVkX1/p2aYzntA1vLjAStstdWSe9+fjr5z6EEwcuty351/kqdmTdHde4hKiNJ2bJWK/XfiYW9s1Rx5CvsZiHbNBtwro+znI9UKILRksNYroPgyLkA2lKYm6pw6ZB7KHbwEfG1jAgUCnXYHAVlCkqa6mcTGRoOuaa+Q88kwJj9prCxUwz5sa8HZr4ZcL9fXy9qGH4aj18PzTgcAK4rYix6RRA6dYTHneYb3J7Kpgmk8w9HHdcFz0EOIYbxcDUlMjjHxbxBb8SwE6G5rWnuhCuEOiVkyIsna7i00LIpzPgNudRoBUl5u/bL0grbDUzDpiZA4c0gHgX71oCY6yzE/FpwWlCSPLRJCiiCEopz1k5j1nIGSZO+cBwZzcUECj0k5xa4XG6mz6rARPWHTEjvYlgsDmONkB85k4oREgCRS8Z0RiV8pnvKXSDw7OsHrGLjwmIfwc/VIgI/slV6dXnzQNKY3SQsOu47KVriLUmfXG6yh8oU6v9EXXXGdQVcf2wZlttIhcbl9KwETbcbkA1fg1bTr0w+cX+mmR+QV24LBhrwGFZHqd4RtPuqY42E+W7Zom0e7Fye65XV2TIxqaLUfrK/DG1FgUSuU3f7196W5dmN1xVaDGRynToOSJX8WU3tiybbWFPXTP5aWcMaRC6NKE7qGGUjOIbU59etTuck/YvKqHVGvFZrsW2UyDC+ig72YFZ6DFCudG/n9vdmuNaVCC71lH9koHdT1HWsVKxJ7UHN+bDHo9lzsoaPa5PXkIEK4Gj6LRdkveLPMMwXxNxCwJCZk72DJY0ZNRG3GHRja39nzTDlHo8MzBjf2CgEvZ+yiDnSZbOD1xiZ/pdr4/cLWOP8bqivxcNJMYsrce/v5g80vmfz1W3MxJFIVdtLuMq9qrfbV9GIZvEejgiSDatwH97tHLLc20S+pdPGwGgKao8hMJa0rtaXHnr4W6g11srlaj9GdtZ2OY4ibhvCC 0c4dXyVG OJxzmDJ/62inL+LgUYKttKmnMUM3Qk/wIuCqagmtavghdXERcgebr5eQvK5ivwcRK+jFiCbUnbJlUxeR8IJBLosODAKQPr6/gED/Tb1O97kXafoXIZKbb+nFrGIsQRH/y0QdZpqOY5UGwrZVv6zKnvDdQoOqwndC3t4LFYBpYIsX1R2Q0NXw4lFKrTCoAB1yXWlCW53j+rmkU0iEIVf43V6nT1BomDeAhtl5/Oumx+SVnrwhpx+3RsFtwELnECzJAsB8iswMQmN/ZWppA1bE4cHmjeXtbo9I+4kxgNip/Wn4qZUnyn7/6QOJ1bcelHv0afSsV/QDOmXejzPn3aQs2rkTVMIQ91owLJgk2jmlgtSeNbz0iaP9cltOxuCWyE8bbzuT8FsbKuTYWJe3lXr0fg6nuIWutxWotLBl3JYZmj7PeJzJwq362fO3vJanNQdc9ACxClFZ2MN3u+gdj7IGcfirffreZw7LeuGnQ5Eps2Xt0D2E2EmzGTSXR1F0F6dD+kwhDLYhCGBpnka1h9qI3ECSbc4lR1dH/nWWFPOKG9bQ/8rDxKtXHstGgxvcxySeIg9gY88iCQLFF9Q0oV1rBgItKytVcH9L+vx2g0LVDzTrtV6dIxukkNnKAZmiT6J5oXDyj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3]. This patchset moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cache-aligned as well. This causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes: slabinfo before: ... : ... vma_lock ... 40 102 1 : ... vm_area_struct ... 160 51 2 : ... slabinfo after moving vm_lock: ... : ... vm_area_struct ... 256 32 2 : ... Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. This memory consumption growth can be addressed later by optimizing the vm_lock. [1] https://lore.kernel.org/all/20230227173632.3292573-34-surenb@google.com/ [2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/ [3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/ Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 28 +++++++++++++---------- include/linux/mm_types.h | 6 +++-- kernel/fork.c | 49 ++++------------------------------------ 3 files changed, 25 insertions(+), 58 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 01ce619f3d17..a5eb0be3e351 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -684,6 +684,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {} #endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK +static inline void vma_lock_init(struct vm_area_struct *vma) +{ + init_rwsem(&vma->vm_lock.lock); + vma->vm_lock_seq = UINT_MAX; +} + /* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -701,7 +707,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0)) return false; /* @@ -716,7 +722,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) * This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); return false; } return true; @@ -729,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma) static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass) { mmap_assert_locked(vma->vm_mm); - down_read_nested(&vma->vm_lock->lock, subclass); + down_read_nested(&vma->vm_lock.lock, subclass); } /* @@ -739,13 +745,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int static inline void vma_start_read_locked(struct vm_area_struct *vma) { mmap_assert_locked(vma->vm_mm); - down_read(&vma->vm_lock->lock); + down_read(&vma->vm_lock.lock); } static inline void vma_end_read(struct vm_area_struct *vma) { rcu_read_lock(); /* keeps vma alive till the end of up_read */ - up_read(&vma->vm_lock->lock); + up_read(&vma->vm_lock.lock); rcu_read_unlock(); } @@ -774,7 +780,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) if (__is_vma_write_locked(vma, &mm_lock_seq)) return; - down_write(&vma->vm_lock->lock); + down_write(&vma->vm_lock.lock); /* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read(). @@ -782,7 +788,7 @@ static inline void vma_start_write(struct vm_area_struct *vma) * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq); - up_write(&vma->vm_lock->lock); + up_write(&vma->vm_lock.lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -794,7 +800,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_locked(struct vm_area_struct *vma) { - if (!rwsem_is_locked(&vma->vm_lock->lock)) + if (!rwsem_is_locked(&vma->vm_lock.lock)) vma_assert_write_locked(vma); } @@ -827,6 +833,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ +static inline void vma_lock_init(struct vm_area_struct *vma) {} static inline bool vma_start_read(struct vm_area_struct *vma) { return false; } static inline void vma_end_read(struct vm_area_struct *vma) {} @@ -861,10 +868,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf) extern const struct vm_operations_struct vma_dummy_vm_ops; -/* - * WARNING: vma_init does not initialize vma->vm_lock. - * Use vm_area_alloc()/vm_area_free() if vma needs locking. - */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { memset(vma, 0, sizeof(*vma)); @@ -873,6 +876,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); vma_numab_state_init(vma); + vma_lock_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 80fef38d9d64..5c4bfdcfac72 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -716,8 +716,6 @@ struct vm_area_struct { * slowpath. */ unsigned int vm_lock_seq; - /* Unstable RCU readers are allowed to read this. */ - struct vma_lock *vm_lock; #endif /* @@ -770,6 +768,10 @@ struct vm_area_struct { struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK + /* Unstable RCU readers are allowed to read this. */ + struct vma_lock vm_lock ____cacheline_aligned_in_smp; +#endif } __randomize_layout; #ifdef CONFIG_NUMA diff --git a/kernel/fork.c b/kernel/fork.c index 0061cf2450ef..7823797e31d2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -436,35 +436,6 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -#ifdef CONFIG_PER_VMA_LOCK - -/* SLAB cache for vm_area_struct.lock */ -static struct kmem_cache *vma_lock_cachep; - -static bool vma_lock_alloc(struct vm_area_struct *vma) -{ - vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); - if (!vma->vm_lock) - return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = UINT_MAX; - - return true; -} - -static inline void vma_lock_free(struct vm_area_struct *vma) -{ - kmem_cache_free(vma_lock_cachep, vma->vm_lock); -} - -#else /* CONFIG_PER_VMA_LOCK */ - -static inline bool vma_lock_alloc(struct vm_area_struct *vma) { return true; } -static inline void vma_lock_free(struct vm_area_struct *vma) {} - -#endif /* CONFIG_PER_VMA_LOCK */ - struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) { struct vm_area_struct *vma; @@ -474,10 +445,6 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) return NULL; vma_init(vma, mm); - if (!vma_lock_alloc(vma)) { - kmem_cache_free(vm_area_cachep, vma); - return NULL; - } return vma; } @@ -496,10 +463,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) * will be reinitialized. */ data_race(memcpy(new, orig, sizeof(*new))); - if (!vma_lock_alloc(new)) { - kmem_cache_free(vm_area_cachep, new); - return NULL; - } + vma_lock_init(new); INIT_LIST_HEAD(&new->anon_vma_chain); vma_numab_state_init(new); dup_anon_vma_name(orig, new); @@ -511,7 +475,6 @@ void __vm_area_free(struct vm_area_struct *vma) { vma_numab_state_free(vma); free_anon_vma_name(vma); - vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); } @@ -522,7 +485,7 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) vm_rcu); /* The vma should not be locked while being destroyed. */ - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock), vma); + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); __vm_area_free(vma); } #endif @@ -3168,11 +3131,9 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - - vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); -#ifdef CONFIG_PER_VMA_LOCK - vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); -#endif + vm_area_cachep = KMEM_CACHE(vm_area_struct, + SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + SLAB_ACCOUNT); mmap_init(); nsproxy_cache_init(); } From patchwork Tue Nov 12 19:46:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13872894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDE30D32D97 for ; Tue, 12 Nov 2024 19:46:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 538066B00C4; Tue, 12 Nov 2024 14:46:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E5606B00CF; Tue, 12 Nov 2024 14:46:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 360286B00D0; Tue, 12 Nov 2024 14:46:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 13E6C6B00C4 for ; Tue, 12 Nov 2024 14:46:55 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BCCA8ABC06 for ; Tue, 12 Nov 2024 19:46:54 +0000 (UTC) X-FDA: 82778475066.18.BF1F1B4 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 94FA7A000B for ; Tue, 12 Nov 2024 19:46:22 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ou2jJscd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3q7AzZwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3q7AzZwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731440684; a=rsa-sha256; cv=none; b=llxEujPbpfMWsWepoSFI9b3IkLB0r8EB+wHopVK5wBQP69r+gP39Fo06C9e4xGOwBBD0yJ d61WfjWqwSkpU4CbUvnhROj4TbHIlRZM1QIP2aKRb/1/4K5FunNscC7SUDvCUF0soJcKRS GgYceW163KMmGLmrNA4PLoxU9CmOMPk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ou2jJscd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3q7AzZwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3q7AzZwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731440684; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z1CCbcAxwVITlach3ohAL8Fs+Rkv4PQfEREgrvS8KzA=; b=KIwVWzrdsqar45UzRW7CHaHfhAdQb38zgwUPJJMNxbEsrOCBonWXl1FruaZgJH3kULwfeG A8pIUMItzZrgtX+FStqs6JkWK2m97+t99RSW79sn7FfBYwqOTerMn5iWO5MpbxAIBv+g0B P9jlYBYUono27L6y5OMbyw/X5VJCoyk= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e35e9d36da3so848505276.1 for ; Tue, 12 Nov 2024 11:46:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731440812; x=1732045612; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Z1CCbcAxwVITlach3ohAL8Fs+Rkv4PQfEREgrvS8KzA=; b=Ou2jJscdR3h6yFgjyrZPnjjCco/JBuP0OG0jFKyH2ccvZTFM/F6kYqqaygqtKlW00U PbNK+Mvk0sqP5WYzehSi0xgSS8iVk1McJ34cQY0TTdTRvj0VMqtNjxL1yiXAFtX1K/mn vhtdaftmTyGLhtMv7itgSow3HxZNL13uqWZF6AKoDdI/b6GKGfrf+I4ClzRRLV3f3vKc ZnIIx3nS4xNRF2+suBRhTEF8hMrHY0hZ8eSxyeV21w8+Bdf2zW5RBpdL2/OVu2x9cdKA KsMSycJ1vcYyK8EjiH5i82seYAvG28Wmmb9kb947SBkIik5DKito/Jt6oMcsdKQm0XhS gTlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731440812; x=1732045612; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Z1CCbcAxwVITlach3ohAL8Fs+Rkv4PQfEREgrvS8KzA=; b=D0o7W90jj0Ef1ka8dKHAvzGyqq/QTkzEDbH8q4SNKO4/gES9i9h/f5tQ7o9zbrNhOz Gx6oPXEV5Pc4Gyo3FPFpUlKNEMDkcYWLIT/sHqflH5qy7eLPWwPcBEEIQiuk2JIQMY6A i0gmefKgr6xWVW6eZXqpAOAeyWfjYtL8PCEI3g3URLJjqGXki6ylr9hGcGLoC0aTcT/e tWAtHXoZ4MRIzon3sjo2L/yiqyKc3MHcxZuYOWyRU18JL6uyG3u3FRNCCle7J8vH4l1i HJ27sBkD1pBjr2txpSZ0KkNUCYzDl+nXp7+A7vmIG9f0ijTDcOBwXJk6YsA3J+2tA6h5 SDrg== X-Forwarded-Encrypted: i=1; AJvYcCVq5XaVuuY3YHusZcyX1zOMd9DYsi02tzJZCEV89ilgqNxda0L3s4MYcUzsT5ZA0jMB8ZGRN2uPeA==@kvack.org X-Gm-Message-State: AOJu0Yw3lvce8t46rot3hmMzoeRvX/O4ZjT5Ki1ZgJsfGX8HWzghXXl1 kqFMUh7zZUDOrlFOsIwIi/yAwTYCtVPfTatyScb2gPLzK59bStQ1n5sV6Na9HhYv0Vx+eT6MNwF Ihw== X-Google-Smtp-Source: AGHT+IEq0LcdS3gaxVGqCSmSAjHrghcWgLVyn1pnLLC1ahL+sX7JpJH17PWyfqB3xOS3eIrVCs2OBxnyQWY= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:cad5:5dc6:1298:ec42]) (user=surenb job=sendgmr) by 2002:a25:dc92:0:b0:e28:eaba:356a with SMTP id 3f1490d57ef6-e337f8f04e4mr71200276.9.1731440811893; Tue, 12 Nov 2024 11:46:51 -0800 (PST) Date: Tue, 12 Nov 2024 11:46:33 -0800 In-Reply-To: <20241112194635.444146-1-surenb@google.com> Mime-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241112194635.444146-4-surenb@google.com> Subject: [PATCH v2 3/5] mm: mark vma as detached until it's added into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 94FA7A000B X-Stat-Signature: bxjt5qdbgz9cq8pgx6ox183z3faodqcb X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1731440782-781226 X-HE-Meta: U2FsdGVkX18PfvJmSYk8GXPhNp0k79B0UkPfyf3JgGVl68H3MKAW80j1F6qWfi0lDpK9R4RrepLlFkl1sWjOCpFyiP7CkUl7WBcTmRjUQ6/eM+LJZdz6ooTGNrAHcE4/2fhugDQ7IGUxDog5CvkX63/4A5dJNTCNN42sQyvXDErJlTRMOM4S0zUP4iRu3eU5FhJm1mu+Qp0AIosa8HyF3LMfKUqXWg8mVJXJHm0UtvOzyuJ36xugzZPYYmNDUQQY43aZs7sppErEwoSz2TSrxFMa+Wf+/EciskqDFTOlqOlLK8xCc9I3XzEZ75r6Pq5fprYAHZE8sOY39acLgPAF/RYLMBDNR00vDlkelixX+VHFHGOQlwLgBTpeOfoDV8Wno/GC/rIiP9YU57uY6LfHjgtm33y9oNG512CjhY8GIavbk6vooS3KjgFcWrTjA0sn9LTclpgbJFaVlAz/uh3lM2UcdWkyVyqL9Aq6mTXB4ne8we+5vEHqAxVvrHPmfmXMy5gqUZ0lRcrcndcEv4LbYILe/xPjUx+1hFwnVPNS/bwzwO15OCTN9Gp0B8nA8rnS4GEBH1U/mwVYpqC39ZNwFCp5se9QeON2SM069bPJPPBTExvzaPWrPjKsHsBZxwWII6ey8AjrK7BtdiQdoFYrcTmBZZaswix4D3dIfzZAuaHkqLg1nin1+OJF/1IlCl2hHFGuy5k3sD/qbowtys16qAfW/ubD+P44cvJfKoGJwYwLi3D2PU15vLpWfsngmLqwZ77mK3D4mDpmwBGa7zYCcEE63EtMGxxewz6PUQYoubQ18AV9+pXtgQOj2qwzlfdvSZoyMxt15sm/aheCvrvnKMZjTkc+O4yRCj9eoNRX/c58iPeUWs/4eTrtNbo8Tk+us+A+bcG7zagrIYD/afwGOKubuuXCh9PHKnvNXjYlBD+EYme95w9UaDrdv8Ulb7sWgos0oZCBSMvAnYwGAi2 DBgKL+aG n1AZ1vFUnnaFic67EsuFzJUEZsR4dWzRWT/60J/1c1226tsuwuAPZafiDJ/gNx7FLWN1iMKY6ZxzHxcAJp3zgBv2L42vwOYV4Rd2mOQ6VEEVKboaQTJUBqnLV5UIriuqvKRp5Gq21LSyVCKOlgziV2tgQ37hCpTJMT40dTUhIxOvKGUUbZY8oP1srNUgaqSni20sRYVfWZwGTFUvTzA3vUPRypv6e4FOPnXlEPzW23K32eK7WvgoUHHJSDIjdhVvLJumlvsnEhNR3k2yKTGMEZHeZglPJjV8JUoxlQXCcoFeaht74TeiBfvGcqaVACuC7v9uNfVFUBzDLP3RoFa9a1D7/PQXawfdc8d4I8mCEg6EaNJMWzPksE8UMAHncIaIzW9d26i3nE3FdisLzc1sW2aSL17KCTpopUBfn8rGwRIZcCobsevgHgShO/NWeNPo8XJWFWwZTektX2vEVLIP6Dr63AFq1fpCWpmXKkvQ1Hy+xZ7K5cgyXMW0pl8vmeIHlelHUPUFZYu+CU9A6z75B1xHWCnddJxcKopdPkYSF4FbBkWk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- include/linux/mm.h | 10 +++++++++- mm/memory.c | 2 +- mm/mmap.c | 2 ++ mm/nommu.c | 2 ++ mm/vma.c | 3 +++ tools/testing/vma/vma_internal.h | 3 ++- 6 files changed, 19 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a5eb0be3e351..245a85caf4c3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -812,6 +812,11 @@ static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) vma->detached = detached; } +static inline bool is_vma_detached(struct vm_area_struct *vma) +{ + return vma->detached; +} + static inline void release_fault_lock(struct vm_fault *vmf) { if (vmf->flags & FAULT_FLAG_VMA_LOCK) @@ -874,7 +879,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); +#ifdef CONFIG_PER_VMA_LOCK + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init(vma); vma_lock_init(vma); } diff --git a/mm/memory.c b/mm/memory.c index 209885a4134f..d0197a0c0996 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6279,7 +6279,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (vma->detached) { + if (is_vma_detached(vma)) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ diff --git a/mm/mmap.c b/mm/mmap.c index 386429f7db5a..1295c4cedaf4 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1570,6 +1570,7 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_store_gfp(vmi, vma, GFP_KERNEL)) goto mas_store_fail; + vma_mark_detached(vma, false); mm->map_count++; validate_mm(mm); ksm_add_vma(vma); @@ -1890,6 +1891,7 @@ static struct vm_area_struct *__install_special_mapping( if (ret) goto out; + vma_mark_detached(vma, false); vm_stat_account(mm, vma->vm_flags, len >> PAGE_SHIFT); perf_event_mmap(vma); diff --git a/mm/nommu.c b/mm/nommu.c index 9cb6e99215e2..6afd5c2bd97d 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1192,6 +1192,7 @@ unsigned long do_mmap(struct file *file, current->mm->map_count++; /* add the VMA to the tree */ vma_iter_store(&vmi, vma); + vma_mark_detached(vma, false); /* we flush the region from the icache only when the first executable * mapping of it is made */ @@ -1357,6 +1358,7 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, setup_vma_to_mm(vma, mm); setup_vma_to_mm(new, mm); vma_iter_store(vmi, new); + vma_mark_detached(new, false); mm->map_count++; return 0; diff --git a/mm/vma.c b/mm/vma.c index 8a454a7bbc80..1426871fa6e0 100644 --- a/mm/vma.c +++ b/mm/vma.c @@ -275,6 +275,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi, * (it may either follow vma or precede it). */ vma_iter_store(vmi, vp->insert); + vma_mark_detached(vp->insert, false); mm->map_count++; } @@ -1690,6 +1691,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) vma_start_write(vma); vma_iter_store(&vmi, vma); + vma_mark_detached(vma, false); vma_link_file(vma); mm->map_count++; validate_mm(mm); @@ -2369,6 +2371,7 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap) /* Lock the VMA since it is modified after insertion into VMA tree */ vma_start_write(vma); vma_iter_store(vmi, vma); + vma_mark_detached(vma, false); map->mm->map_count++; vma_link_file(vma); diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h index 1d9fc97b8e80..fdb60978821f 100644 --- a/tools/testing/vma/vma_internal.h +++ b/tools/testing/vma/vma_internal.h @@ -438,7 +438,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_mark_detached(vma, false); + /* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; } static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) From patchwork Tue Nov 12 19:46:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13872895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0959FD32D97 for ; Tue, 12 Nov 2024 19:47:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92FF66B00D0; Tue, 12 Nov 2024 14:46:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DE536B00D1; Tue, 12 Nov 2024 14:46:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 732A56B00D2; Tue, 12 Nov 2024 14:46:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 520F96B00D0 for ; Tue, 12 Nov 2024 14:46:59 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0CB72804AA for ; Tue, 12 Nov 2024 19:46:59 +0000 (UTC) X-FDA: 82778473974.07.BDF670B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 7F28B1A0025 for ; Tue, 12 Nov 2024 19:46:04 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="P2/ycWK8"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731440730; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+O3HT96xjOpyZyOfTy1XSWvtoYhqZSjPLYRvhqOFff8=; b=rPR/TKu8gJmsK3154VSzYJJKFIzTlEyISeuCppEPKA5FegeI0AUAoOi8nuuwDQ/vUTTT2f Ym1fU8YtTOWdKtNHlDG4/GnhP248Iy2j+puduJTPBxdE+XN9R5MfVN8xIJNZIXa74pgwX7 tIQmVkeZu1I15+7xKX3WFe9Y8whwmho= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="P2/ycWK8"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731440730; a=rsa-sha256; cv=none; b=KQSoCXbzP+1oH/nf6jo6QjV03zTSz/iTdtEoFttzECEIARVXjx6TUIL7fqStuqvC0JHSi1 OAgJT5VQyY7n977WX/NR+VEtJBdpLjPeN69cxo2Z/ufkrBoM8DmzwGlqdcEZQDHahSakDz L/HLGhvWqc/xM0L+OF0M7xLHNcIY9Zk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e59dc7df64so109707b3.1 for ; Tue, 12 Nov 2024 11:46:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731440816; x=1732045616; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+O3HT96xjOpyZyOfTy1XSWvtoYhqZSjPLYRvhqOFff8=; b=P2/ycWK8hMd6cpq2w2OPn5qWX0IGVNOSrdePFi/BL8Q37w2dSPATdFmY3Oad9yZkoC bPMQXyehThg8X2ueC/fIo2tR/DCRkzQOyFigBGFIMr8JOlInRnI3kaKqUoFtN1ZlN0ES 2ZVJbSlwuqhtvc0H73gATZtb8n1+wOq7jmXUCbjCtCrk7HHBkjjc6Mp7M2UpVHwl8l74 jOGQP3FmnTluJIiYsnMoA1slqSC7LlJCUpo8MHBOOWJIT5G0xnDU2dEKBoyLwiy8eCFn o19fzP3v8z0Wi9J3CyWi38PGA/NUxDXmvyUnL+nM5Ls55LQHuzuUOU3qTdheKY3E5TUa YkqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731440816; x=1732045616; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+O3HT96xjOpyZyOfTy1XSWvtoYhqZSjPLYRvhqOFff8=; b=nEPi/p6Patb37pzk6fWWOfV2INq5JELx5gyNRMBqIfv8pFgxLUaHzGzHBtDCE+qi6C aNtM/1u6VL0Z1ZGsWskgYuGdhg0aYV2t5GNxAoPQtUzgRGIWRjC4G3QtFVyJVf5eobB7 zO+7ONdLn/QWdEeX3Oj91Fw/ffasi+hLhYD4TOfqu513osPlfV/Mm4qv1ad/F5btgq2u RpscJ7nxkZPp0XaE9jT8d6VNFXqAE86JeiRuM68cq06bquhuW17fi52g+C4MGL+9IapM QEKQcgqH/PALxcF1K2klR09gS2kGhWxKPyW7MG9liHcmZDhoIAEYj1Yj+Jx03dpDBSTS 1aEw== X-Forwarded-Encrypted: i=1; AJvYcCWJZ20GqyPKyHpuW9aLcpiUAoHegG9sXOYVimL9s7a8nZ6VGyl9DRDReDuAFAGj+qWAfabFtekvxw==@kvack.org X-Gm-Message-State: AOJu0Ywd1rzcqkxMTn5Zva2ZWDzGuj5pYndT/zouESJfDPvtj+GQyXjq tuPY+kZCPjeLrmcCiQ+b1vypa6KbeLRhUpO5pbsc8IJ8pKy1lwcOjoZzp0ndKYQ59JUz0+AXRCI SbA== X-Google-Smtp-Source: AGHT+IHOEk82i1GQQA1lcKwQVafHXzM7QcBOoBI0xQdYABDX0Xu7GKWAPVcRoNYH7FkpwH/Uu3J2wrcQokM= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:cad5:5dc6:1298:ec42]) (user=surenb job=sendgmr) by 2002:a05:690c:8008:b0:6ec:b10d:5745 with SMTP id 00721157ae682-6ecb10d5a0bmr28207b3.3.1731440816377; Tue, 12 Nov 2024 11:46:56 -0800 (PST) Date: Tue, 12 Nov 2024 11:46:34 -0800 In-Reply-To: <20241112194635.444146-1-surenb@google.com> Mime-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241112194635.444146-5-surenb@google.com> Subject: [PATCH v2 4/5] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7F28B1A0025 X-Stat-Signature: yo61j6s178maz3mskoi54qu8gh3zrzcm X-HE-Tag: 1731440764-601242 X-HE-Meta: U2FsdGVkX1/UXUxWFTGpKm8iNYyKkqCTy2aWNYJGdkL7tnwC0hEs5TFPQr/FuxSXhQpKJJmVJb9uO3n9PbTva92SiPD+7Ds76j3B9Kdso92PasjutDMDg6xcRAFsFMFRGXj/4AqQsjpUjfE8JLxVelRvMUR8ufOHTv3VjaCLKj3CeNBi4P/P1wojoS7aHDn10Ehn6My+76XKaOxzT53lMhvkOY6NFP1iJ53dHMwbmB07hIvhIhPJ3DqeuX63CvsCB1BZPk0mL0tesL0snUpNwLiiLv39LSSsbnJ1wJLDB5nAfGrXBgLWFmOnxT4TQ0s67UWRE/Gb+wj7WfCjSJgHhKXXVMbzchE6hZWHgi2ViDrlALUmdq2GoVAbExeqs2z3UVOnJWZPSokBppJ+9UOckSfV/HXHMrHkUY2Za/VNTEEJvQ6SNIpUmhN3CwIfnrWRkTSpt7P+20L9JjfPP7Hgp3C+DTbGYx1hLjP7mdqP5Wv7D3mIn1eI6coK+N+r/jetB2x1ikeHd9wOXwV2KY9vKXJ4hbyxmu9TOr4embWUQhtoHl5S5NhqSX2prvppcOoTVlJOmAHFRwZzoqUlZJ+sWL1gfbopn8xgX2Jf4Kck2vmMv7omG+8amZTc3WvhWhS2BGT+xh7rOELC4LC4wpqSbVSgidcgYk1tI62tDjVLw+OINhjvjyVI0rfUR8Gsix2y4ZV4A1/sOAWmc5ieS0ZDiQkWvsrhVi4MJ/juWEb6zvBRx3sqQenCRD/26OxQjIgqy84s4HuGH5tCPWqUuK4QfiwDYIyvUNC9NMHsaU8vqkPiTlLYiW1KbBOoMbUSBV7G9uHSR+eUOmShxrI/0incKqyc2Hkpde1R3vrikAnPPsc5daxPp2SU3IvFzhnuyuUkEN860XaSNzx5p6N/rRXBehlOUAgp5Sffk2wnscs/llXMroJOrUM6kQDeE07y8Yw9h2x8pOa1cdYuB8WKz0J Nd+1Fnal FDjnxGTw1byx9SfijNCN98p+tTSRg2TEP0yubg4uvVYmKGPK5yNAu2c9uj1TKzCyYE58XFsmZmgGAqsHQ3EAXeiHQCYXYk/GwyvDAjhEnI9dz+odQ4fcwyzE9yHgEuTDh2PBaEsjfLsCbpZvf4o7YGQdaiwhy6LSlY/XEo9kEDdDngPhrFZ1eWBN2khoYKuDXfzUSLWpy6YYbTAZrOU1astfTwzOu9xQPAFjPDbeIuhvac0+Zj9AynpQO49Qxt9ivBavGCutXbpUfuv9UxrJm3/yzLDZctlOddMZF+kG/e7DEwGwOs4qlVnZmGlmpupPyRf8JNlm3SGS4nSw6TLC1wx3RbnAZivCKYg3UV+0q/zRNaFCjWM28B6u9LNJ5MgGIh7VeiupohRApygQIdFbs4P65pcZVvw7IrV73HOl16MvF3sFiTwUG5yzohy1jx0QjRKZFI6zhnsll1zlLkMXSS0nS+gvXUEUmxCUdqW8WlO6p8j4CVKeyAprXSMeYVRwKflqp5fkJCmbvVqsqnaxOyebA63WAAE/lJjMr4S3YkA89yr7QrChd8eMsHkqqdHquAn9tilsJZUUIOWNt1VIA+EWNkw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu(). lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it. Vma reuse introduces a possibility that in between those events of finding and locking the vma, it can get detached, reused, added into a tree and be marked as attached. Current checks will help detecting cases when: - vma was reused but not yet added into the tree (detached check) - vma was reused at a different address range (address check) If vma is covering a new address range which still includes the address we were looking for, it's not a problem unless the reused vma was added into a different address space. Therefore checking that vma->vm_mm is still the same is the the only missing check to detect vma reuse. Add this missing check into lock_vma_under_rcu() and change vma cache to include SLAB_TYPESAFE_BY_RCU. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Adding vm_freeptr into vm_area_struct avoids bloating that structure. lock_vma_under_rcu() checks of the detached flag guarantees that vma is valid and attached to a tree, therefore unioning vm_freeptr with vm_start/vm_end is not an issue even though lock_vma_under_rcu() is using them. As part of this change freeptr_t declaration is moved into mm_types.h to avoid circular dependencies between mm_types.h and slab.h. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 10 +++++++--- include/linux/slab.h | 6 ------ kernel/fork.c | 29 +++++++++++++---------------- mm/memory.c | 2 +- 4 files changed, 21 insertions(+), 26 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5c4bfdcfac72..37580cc7bec0 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -32,6 +32,12 @@ struct address_space; struct mem_cgroup; +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the @@ -673,9 +679,7 @@ struct vm_area_struct { unsigned long vm_start; unsigned long vm_end; }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */ }; /* diff --git a/include/linux/slab.h b/include/linux/slab.h index b35e2db7eb0e..cb45db2402ac 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -212,12 +212,6 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/kernel/fork.c b/kernel/fork.c index 7823797e31d2..946c3f9a9342 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -478,25 +478,15 @@ void __vm_area_free(struct vm_area_struct *vma) kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) +void vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - +#ifdef CONFIG_PER_VMA_LOCK + /* The vma should be detached while being destroyed. */ + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); /* The vma should not be locked while being destroyed. */ VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); - __vm_area_free(vma); -} #endif - -void vm_area_free(struct vm_area_struct *vma) -{ -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else __vm_area_free(vma); -#endif } static void account_kernel_stack(struct task_struct *tsk, int account) @@ -3115,6 +3105,11 @@ void __init mm_cache_init(void) void __init proc_caches_init(void) { + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr), + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| @@ -3131,9 +3126,11 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), &args, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| SLAB_ACCOUNT); + mmap_init(); nsproxy_cache_init(); } diff --git a/mm/memory.c b/mm/memory.c index d0197a0c0996..9c414c81f14a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6279,7 +6279,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (is_vma_detached(vma)) { + if (is_vma_detached(vma) || vma->vm_mm != mm) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */ From patchwork Tue Nov 12 19:46:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13872896 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5AF9D32D97 for ; Tue, 12 Nov 2024 19:47:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 584676B00D2; Tue, 12 Nov 2024 14:47:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 534BB6B00D3; Tue, 12 Nov 2024 14:47:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D69C6B00D4; Tue, 12 Nov 2024 14:47:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1D7886B00D2 for ; Tue, 12 Nov 2024 14:47:04 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9CA4FC0161 for ; Tue, 12 Nov 2024 19:47:03 +0000 (UTC) X-FDA: 82778475066.30.D47B39B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 299FDC0008 for ; Tue, 12 Nov 2024 19:46:43 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vzQsPCWF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3tLAzZwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3tLAzZwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731440759; a=rsa-sha256; cv=none; b=DoNafKy2W/vpVetqzDDCV6Jw2pERCRwveCNLCILa3A7ZqntytKDoi40PDMm5XA3McKzRSH rXBVHUgjiPAbxDwaEydHlc8uJh2Q4SJ/1POSPWmoTMX2u3CkW4b6VDRdvRTEZofF2b1JyU koKvSuVE4eYohlUCQ14SIUKPp5u+uFo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vzQsPCWF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3tLAzZwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3tLAzZwYKCCUTVSFOCHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731440759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wUwaf5ZgB/1FQH1yPsvN8djw+8+0TDY9z5rOH6D3QU0=; b=4Q6FJzBwGT/vn0R87qwfOzTruKYEagyzYoFOKzxc6AgZ9+n/em8x9q9MAatyInOi2nAjc4 0HEwQkSSlgC74FBIfWnGisVcRdHOIvbCjqheXVxzdLxzp4tfH5//0QJbuXkUFZA2dh2fDd 7EvbPIfueQ9v9hVus2pqziUJnWxXhcw= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6ea33140094so106910187b3.1 for ; Tue, 12 Nov 2024 11:47:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731440821; x=1732045621; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wUwaf5ZgB/1FQH1yPsvN8djw+8+0TDY9z5rOH6D3QU0=; b=vzQsPCWFaaIO5MQ3FMvCDNfYFfXV7odskEeh7ilR5TN9ixn4MR+7p0xLvdLV2ap8Gs s+2LCleAnZ3wyB4eGOTJhm3/sijF2HW1KGsshCKYlN1rXIuxQ6wpOwqIWiR+ws154joA s5eJdOoar7WH882A4nrZFxdynlIb1F41RPaTUVCg6AJKQ9rgHmPO7tY62htSDiSk2XiP MdZQ1HGkjtLPGekageY/CM0N4WZ2DrDiRO5Fuyr3eZpYOpi8WhJ9LnKHIxymO6B9moaw EqYQjhVt0S9d0j97X627138uin9Agap80V1vNsTBSQMMgJ7a52/4NNwO3spV8ZYE+7ET uB9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731440821; x=1732045621; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wUwaf5ZgB/1FQH1yPsvN8djw+8+0TDY9z5rOH6D3QU0=; b=LYHi9KrHNiMeCdTrjOW28kfQ4oUWx70e4XjALQEOmY/tI/+m+GnAc77eko+pVeZWNq rrBzA4SUI88FG8GH8pDkEYmWmX6YDsjilE316gRyqloc6Mv8eNcpRn8PS3Eqp/IOyDaT rM2rIlOqeL5sc0wtKJFHEWkIc0+9xVqlyGPL77yIEgKuZ842BKb5NvYtUPHrsensAYBN xBY/GeZUVNImQxCIUaV7JoxivXt2aJp928rQZS0SURnkGBFNP1gb3j7aXYXzADRj77/k 6vqsgT8NwjUiLtMBjaE42Mkwmnq+tyrt7KirpZFZgEKnmUuV7mSgImIZqkSo538CSaba cghQ== X-Forwarded-Encrypted: i=1; AJvYcCW4kEKoL2C3+7vKPg5mmJfTiSFupIWnewEoJU7nzrb3mADAqQrCtQ4gGTEkYitlvaaWVNZ8X9KJ3g==@kvack.org X-Gm-Message-State: AOJu0YwLaCZP48fvgKLkOQQjgvxJGsNTF1864c8QV5NZA7++/fZyAJB8 qc2o2eJHD9hxKIEq2PTw0yHZihh49A616RjAr74GKNXwX+J3Vmg9bkN5KT2WK28Em87/M6HkCoU Cdw== X-Google-Smtp-Source: AGHT+IGaC6PuAhu3pWt6DipZ12u32jxx/tWme6oiVJjTtPaAPq0Tli46qIKCkEtg3rNyPGo2HA9JrDSEFng= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:cad5:5dc6:1298:ec42]) (user=surenb job=sendgmr) by 2002:a05:690c:6608:b0:6de:19f:34d7 with SMTP id 00721157ae682-6ecb31b51f7mr2737b3.2.1731440820935; Tue, 12 Nov 2024 11:47:00 -0800 (PST) Date: Tue, 12 Nov 2024 11:46:35 -0800 In-Reply-To: <20241112194635.444146-1-surenb@google.com> Mime-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241112194635.444146-6-surenb@google.com> Subject: [PATCH v2 5/5] docs/mm: document latest changes to vm_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 299FDC0008 X-Rspamd-Server: rspam01 X-Stat-Signature: ghn1drwum375f5fd1g9mtb3x7pp9em76 X-HE-Tag: 1731440803-454046 X-HE-Meta: U2FsdGVkX1+Yv1UpptaS4qvSO8xZq//kQRKjiomtO37qh31UYimEtoUEnfFLaSAkTE3EJ8mcabvlRGn7ozBelVrsxZRz0In+5EOYJWSj7slnd1TE+uMh/XQ9np+GZ5qw6EJv/xlfh+C6DG7+JAnXNsbdZyve3TtQzUJ6KczpAWGkG8TP8wCXiBYFR+I2vjQTL9JX89NTrrrbvXriXhYU6xm754OTzKvCI18e8VWBTvymfdVoM6lu3YQphW8/9VPjtZtfXk9x1ke4f/wExl7f3tAeahStoJ1Uf+ilhVTUxF/IUHBC56mr8xmHMiF+HBhvrBNK1Of0Pu7qJqD5wTpHqV6J1dvqgIOMe9+gNt2FBfQqhksUFqFcbi2V923YJPTV2PrnvqmfcSGFI94X1QgXF7yAJdWlLGXKdGaVYeuFibSYY0wJyfMIkIR0drLc3bsC0g7JV7zfayz7R7brwEJ/rF77+CcdzvmInQtva7uIYBjDMb8NWJbojopQyrYT/HnyIBuehZ1cuJGQYBisVIw3AOnTwBipTUAhdcxv/+kVn98R9Kaok4CQV+u3PeuIXh1JLSrS4nlGAWRz3t65j+x/Ao5WjuK7mwvO66sTDnuPoZMCcDZO1T0NqOkZlcAIkDN1DDrqvNsXtg5heZu+wv6v7w5k6ENTRAriYTXnOjPzV7smMfK1JXhu7qoEZlGtWpEo3wb+JbBU5S4BBZ3d+5HRRPtY3IwREUUIu351cwYV0qFNEaOBhHTFhV+pohcgEctf8jlNXVZsKIgkGarJB2XUUPVqto5hddEAVHX9Hb5DRr7SUoYR1UFh7orzRjDT4ci5Rlz4r2veiwsj9OJSBI/UsaKaDx2oGpd5stO+4ulfgrVXwN6cwWzaaJ9oVvJLpSUFkEyHmOX1sh8DFMrFUblTsEgGrqEcckMMc0RKLn+yf7PDTk3nbAh/ICkc4ZFkro/OwTLA7QEWHp3jvAAyvlz xquUrzLh REHSyMe4Ar2FBO08AymW7o8TQtuTDiq6tA2I3Bn9VToANg/tDznT6i0gd2VOnqItJNejy2jemcPInDDmpigezTiXOiF66bVsH3tEssvPF/0rIfIocH/RfQTMoXOWSG0sMATtXx+fN7PHAYKMp3mWXJqrH+9sAphtJWXQOPxDljChvrZy8yHsuKpT00YM9I6BxBn68w/gegid99RmuI3U/GjggVqvo0dhYhCzEY2rJlqW7CuUOmf8kD8hjyC8VjZq1m1B/efcx8D7krHdAmAwjMZ2q6LrSxLfkP62ngGV7ZHFQy6Xc4kBPZp0F6N5bVtaNMqbj48RjV4WqQf0cbWyizsMB9WlSUmnA2RUflk4b0xSQtIUjDqNI5Hb2GNDj3mHjOjRsjk9yBGms2LrXvPeQB7xyKPvP1LKcQNEH7JUUhGXpgZGp5rAW/VikwIbwSZK2TEETsJycGzSJatAjM7QOaA2JdkEb8TG2/mBpRArP/Dl8VM8IaYY5TNc7Rbf9AsGozwWwOTrBVovo/KsB/y6OjkXb5G+8Msv+bAqiVC8CELlfGABDKd3D1b9RuIa+gRpAiNM+N/ybIFxtH4SMAMAzUJlzlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000085, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Change the documentation to reflect that vm_lock is integrated into vma. Document newly introduced vma_start_read_locked{_nested} functions. Signed-off-by: Suren Baghdasaryan Reviewed-by: Lorenzo Stoakes --- Documentation/mm/process_addrs.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/process_addrs.rst b/Documentation/mm/process_addrs.rst index ed74685ffbf2..c8935509173e 100644 --- a/Documentation/mm/process_addrs.rst +++ b/Documentation/mm/process_addrs.rst @@ -675,7 +675,11 @@ RCU critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`, before releasing the RCU lock via :c:func:`!rcu_read_unlock`. -VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for +In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked` +and :c:func:`!vma_start_read_locked_nested` can be used. These functions always +succeed in acquiring VMA read lock. + +VMA read locks hold the read lock on the :c:member:`!vma.vm_lock` semaphore for their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it via :c:func:`!vma_end_read`. @@ -739,7 +743,7 @@ keep VMAs locked across entirely separate write operations. It also maintains correct lock ordering. Each time a VMA read lock is acquired, we acquire a read lock on the -:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that +:c:member:`!vma.vm_lock` read/write semaphore and hold it, while checking that the sequence count of the VMA does not match that of the mm. If it does, the read lock fails. If it does not, we hold the lock, excluding @@ -749,7 +753,7 @@ Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu` are also RCU safe, so the whole read lock operation is guaranteed to function correctly. -On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock` +On the write side, we acquire a write lock on the :c:member:`!vma.vm_lock` read/write semaphore, before setting the VMA's sequence number under this lock, also simultaneously holding the mmap write lock.