From patchwork Tue Nov 12 19:46:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13872895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0959FD32D97 for ; Tue, 12 Nov 2024 19:47:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92FF66B00D0; Tue, 12 Nov 2024 14:46:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DE536B00D1; Tue, 12 Nov 2024 14:46:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 732A56B00D2; Tue, 12 Nov 2024 14:46:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 520F96B00D0 for ; Tue, 12 Nov 2024 14:46:59 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0CB72804AA for ; Tue, 12 Nov 2024 19:46:59 +0000 (UTC) X-FDA: 82778473974.07.BDF670B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 7F28B1A0025 for ; Tue, 12 Nov 2024 19:46:04 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="P2/ycWK8"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731440730; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+O3HT96xjOpyZyOfTy1XSWvtoYhqZSjPLYRvhqOFff8=; b=rPR/TKu8gJmsK3154VSzYJJKFIzTlEyISeuCppEPKA5FegeI0AUAoOi8nuuwDQ/vUTTT2f Ym1fU8YtTOWdKtNHlDG4/GnhP248Iy2j+puduJTPBxdE+XN9R5MfVN8xIJNZIXa74pgwX7 tIQmVkeZu1I15+7xKX3WFe9Y8whwmho= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="P2/ycWK8"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3sLAzZwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731440730; a=rsa-sha256; cv=none; b=KQSoCXbzP+1oH/nf6jo6QjV03zTSz/iTdtEoFttzECEIARVXjx6TUIL7fqStuqvC0JHSi1 OAgJT5VQyY7n977WX/NR+VEtJBdpLjPeN69cxo2Z/ufkrBoM8DmzwGlqdcEZQDHahSakDz L/HLGhvWqc/xM0L+OF0M7xLHNcIY9Zk= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e59dc7df64so109707b3.1 for ; Tue, 12 Nov 2024 11:46:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731440816; x=1732045616; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+O3HT96xjOpyZyOfTy1XSWvtoYhqZSjPLYRvhqOFff8=; b=P2/ycWK8hMd6cpq2w2OPn5qWX0IGVNOSrdePFi/BL8Q37w2dSPATdFmY3Oad9yZkoC bPMQXyehThg8X2ueC/fIo2tR/DCRkzQOyFigBGFIMr8JOlInRnI3kaKqUoFtN1ZlN0ES 2ZVJbSlwuqhtvc0H73gATZtb8n1+wOq7jmXUCbjCtCrk7HHBkjjc6Mp7M2UpVHwl8l74 jOGQP3FmnTluJIiYsnMoA1slqSC7LlJCUpo8MHBOOWJIT5G0xnDU2dEKBoyLwiy8eCFn o19fzP3v8z0Wi9J3CyWi38PGA/NUxDXmvyUnL+nM5Ls55LQHuzuUOU3qTdheKY3E5TUa YkqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731440816; x=1732045616; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+O3HT96xjOpyZyOfTy1XSWvtoYhqZSjPLYRvhqOFff8=; b=nEPi/p6Patb37pzk6fWWOfV2INq5JELx5gyNRMBqIfv8pFgxLUaHzGzHBtDCE+qi6C aNtM/1u6VL0Z1ZGsWskgYuGdhg0aYV2t5GNxAoPQtUzgRGIWRjC4G3QtFVyJVf5eobB7 zO+7ONdLn/QWdEeX3Oj91Fw/ffasi+hLhYD4TOfqu513osPlfV/Mm4qv1ad/F5btgq2u RpscJ7nxkZPp0XaE9jT8d6VNFXqAE86JeiRuM68cq06bquhuW17fi52g+C4MGL+9IapM QEKQcgqH/PALxcF1K2klR09gS2kGhWxKPyW7MG9liHcmZDhoIAEYj1Yj+Jx03dpDBSTS 1aEw== X-Forwarded-Encrypted: i=1; AJvYcCWJZ20GqyPKyHpuW9aLcpiUAoHegG9sXOYVimL9s7a8nZ6VGyl9DRDReDuAFAGj+qWAfabFtekvxw==@kvack.org X-Gm-Message-State: AOJu0Ywd1rzcqkxMTn5Zva2ZWDzGuj5pYndT/zouESJfDPvtj+GQyXjq tuPY+kZCPjeLrmcCiQ+b1vypa6KbeLRhUpO5pbsc8IJ8pKy1lwcOjoZzp0ndKYQ59JUz0+AXRCI SbA== X-Google-Smtp-Source: AGHT+IHOEk82i1GQQA1lcKwQVafHXzM7QcBOoBI0xQdYABDX0Xu7GKWAPVcRoNYH7FkpwH/Uu3J2wrcQokM= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:cad5:5dc6:1298:ec42]) (user=surenb job=sendgmr) by 2002:a05:690c:8008:b0:6ec:b10d:5745 with SMTP id 00721157ae682-6ecb10d5a0bmr28207b3.3.1731440816377; Tue, 12 Nov 2024 11:46:56 -0800 (PST) Date: Tue, 12 Nov 2024 11:46:34 -0800 In-Reply-To: <20241112194635.444146-1-surenb@google.com> Mime-Version: 1.0 References: <20241112194635.444146-1-surenb@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241112194635.444146-5-surenb@google.com> Subject: [PATCH v2 4/5] mm: make vma cache SLAB_TYPESAFE_BY_RCU From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: willy@infradead.org, liam.howlett@oracle.com, lorenzo.stoakes@oracle.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mjguzik@gmail.com, oliver.sang@intel.com, mgorman@techsingularity.net, david@redhat.com, peterx@redhat.com, oleg@redhat.com, dave@stgolabs.net, paulmck@kernel.org, brauner@kernel.org, dhowells@redhat.com, hdanton@sina.com, hughd@google.com, minchan@google.com, jannh@google.com, shakeel.butt@linux.dev, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7F28B1A0025 X-Stat-Signature: yo61j6s178maz3mskoi54qu8gh3zrzcm X-HE-Tag: 1731440764-601242 X-HE-Meta: U2FsdGVkX1/UXUxWFTGpKm8iNYyKkqCTy2aWNYJGdkL7tnwC0hEs5TFPQr/FuxSXhQpKJJmVJb9uO3n9PbTva92SiPD+7Ds76j3B9Kdso92PasjutDMDg6xcRAFsFMFRGXj/4AqQsjpUjfE8JLxVelRvMUR8ufOHTv3VjaCLKj3CeNBi4P/P1wojoS7aHDn10Ehn6My+76XKaOxzT53lMhvkOY6NFP1iJ53dHMwbmB07hIvhIhPJ3DqeuX63CvsCB1BZPk0mL0tesL0snUpNwLiiLv39LSSsbnJ1wJLDB5nAfGrXBgLWFmOnxT4TQ0s67UWRE/Gb+wj7WfCjSJgHhKXXVMbzchE6hZWHgi2ViDrlALUmdq2GoVAbExeqs2z3UVOnJWZPSokBppJ+9UOckSfV/HXHMrHkUY2Za/VNTEEJvQ6SNIpUmhN3CwIfnrWRkTSpt7P+20L9JjfPP7Hgp3C+DTbGYx1hLjP7mdqP5Wv7D3mIn1eI6coK+N+r/jetB2x1ikeHd9wOXwV2KY9vKXJ4hbyxmu9TOr4embWUQhtoHl5S5NhqSX2prvppcOoTVlJOmAHFRwZzoqUlZJ+sWL1gfbopn8xgX2Jf4Kck2vmMv7omG+8amZTc3WvhWhS2BGT+xh7rOELC4LC4wpqSbVSgidcgYk1tI62tDjVLw+OINhjvjyVI0rfUR8Gsix2y4ZV4A1/sOAWmc5ieS0ZDiQkWvsrhVi4MJ/juWEb6zvBRx3sqQenCRD/26OxQjIgqy84s4HuGH5tCPWqUuK4QfiwDYIyvUNC9NMHsaU8vqkPiTlLYiW1KbBOoMbUSBV7G9uHSR+eUOmShxrI/0incKqyc2Hkpde1R3vrikAnPPsc5daxPp2SU3IvFzhnuyuUkEN860XaSNzx5p6N/rRXBehlOUAgp5Sffk2wnscs/llXMroJOrUM6kQDeE07y8Yw9h2x8pOa1cdYuB8WKz0J Nd+1Fnal FDjnxGTw1byx9SfijNCN98p+tTSRg2TEP0yubg4uvVYmKGPK5yNAu2c9uj1TKzCyYE58XFsmZmgGAqsHQ3EAXeiHQCYXYk/GwyvDAjhEnI9dz+odQ4fcwyzE9yHgEuTDh2PBaEsjfLsCbpZvf4o7YGQdaiwhy6LSlY/XEo9kEDdDngPhrFZ1eWBN2khoYKuDXfzUSLWpy6YYbTAZrOU1astfTwzOu9xQPAFjPDbeIuhvac0+Zj9AynpQO49Qxt9ivBavGCutXbpUfuv9UxrJm3/yzLDZctlOddMZF+kG/e7DEwGwOs4qlVnZmGlmpupPyRf8JNlm3SGS4nSw6TLC1wx3RbnAZivCKYg3UV+0q/zRNaFCjWM28B6u9LNJ5MgGIh7VeiupohRApygQIdFbs4P65pcZVvw7IrV73HOl16MvF3sFiTwUG5yzohy1jx0QjRKZFI6zhnsll1zlLkMXSS0nS+gvXUEUmxCUdqW8WlO6p8j4CVKeyAprXSMeYVRwKflqp5fkJCmbvVqsqnaxOyebA63WAAE/lJjMr4S3YkA89yr7QrChd8eMsHkqqdHquAn9tilsJZUUIOWNt1VIA+EWNkw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu(). lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it. Vma reuse introduces a possibility that in between those events of finding and locking the vma, it can get detached, reused, added into a tree and be marked as attached. Current checks will help detecting cases when: - vma was reused but not yet added into the tree (detached check) - vma was reused at a different address range (address check) If vma is covering a new address range which still includes the address we were looking for, it's not a problem unless the reused vma was added into a different address space. Therefore checking that vma->vm_mm is still the same is the the only missing check to detect vma reuse. Add this missing check into lock_vma_under_rcu() and change vma cache to include SLAB_TYPESAFE_BY_RCU. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls. Adding vm_freeptr into vm_area_struct avoids bloating that structure. lock_vma_under_rcu() checks of the detached flag guarantees that vma is valid and attached to a tree, therefore unioning vm_freeptr with vm_start/vm_end is not an issue even though lock_vma_under_rcu() is using them. As part of this change freeptr_t declaration is moved into mm_types.h to avoid circular dependencies between mm_types.h and slab.h. Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 10 +++++++--- include/linux/slab.h | 6 ------ kernel/fork.c | 29 +++++++++++++---------------- mm/memory.c | 2 +- 4 files changed, 21 insertions(+), 26 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5c4bfdcfac72..37580cc7bec0 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -32,6 +32,12 @@ struct address_space; struct mem_cgroup; +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * Each physical page in the system has a struct page associated with * it to keep track of whatever it is we are using the page for at the @@ -673,9 +679,7 @@ struct vm_area_struct { unsigned long vm_start; unsigned long vm_end; }; -#ifdef CONFIG_PER_VMA_LOCK - struct rcu_head vm_rcu; /* Used for deferred freeing. */ -#endif + freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */ }; /* diff --git a/include/linux/slab.h b/include/linux/slab.h index b35e2db7eb0e..cb45db2402ac 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -212,12 +212,6 @@ enum _slab_flag_bits { #define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED #endif -/* - * freeptr_t represents a SLUB freelist pointer, which might be encoded - * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. - */ -typedef struct { unsigned long v; } freeptr_t; - /* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. * diff --git a/kernel/fork.c b/kernel/fork.c index 7823797e31d2..946c3f9a9342 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -478,25 +478,15 @@ void __vm_area_free(struct vm_area_struct *vma) kmem_cache_free(vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb(struct rcu_head *head) +void vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - +#ifdef CONFIG_PER_VMA_LOCK + /* The vma should be detached while being destroyed. */ + VM_BUG_ON_VMA(!is_vma_detached(vma), vma); /* The vma should not be locked while being destroyed. */ VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma); - __vm_area_free(vma); -} #endif - -void vm_area_free(struct vm_area_struct *vma) -{ -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); -#else __vm_area_free(vma); -#endif } static void account_kernel_stack(struct task_struct *tsk, int account) @@ -3115,6 +3105,11 @@ void __init mm_cache_init(void) void __init proc_caches_init(void) { + struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr), + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| @@ -3131,9 +3126,11 @@ void __init proc_caches_init(void) sizeof(struct fs_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL); - vm_area_cachep = KMEM_CACHE(vm_area_struct, - SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC| + vm_area_cachep = kmem_cache_create("vm_area_struct", + sizeof(struct vm_area_struct), &args, + SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU| SLAB_ACCOUNT); + mmap_init(); nsproxy_cache_init(); } diff --git a/mm/memory.c b/mm/memory.c index d0197a0c0996..9c414c81f14a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6279,7 +6279,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval; /* Check if the VMA got isolated after we found it */ - if (is_vma_detached(vma)) { + if (is_vma_detached(vma) || vma->vm_mm != mm) { vma_end_read(vma); count_vm_vma_lock_event(VMA_LOCK_MISS); /* The area was replaced with another one */