From patchwork Thu Feb 16 05:17:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13142503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C90D6C61DA4 for ; Thu, 16 Feb 2023 05:18:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6752F6B0088; Thu, 16 Feb 2023 00:18:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FDF46B0089; Thu, 16 Feb 2023 00:18:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 452416B008A; Thu, 16 Feb 2023 00:18:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2FA966B0088 for ; Thu, 16 Feb 2023 00:18:22 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0B645407F8 for ; Thu, 16 Feb 2023 05:18:22 +0000 (UTC) X-FDA: 80471999244.13.46DE3EA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf10.hostedemail.com (Postfix) with ESMTP id 41C2FC0007 for ; Thu, 16 Feb 2023 05:18:20 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KnhAt7qu; spf=pass (imf10.hostedemail.com: domain of 3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676524700; a=rsa-sha256; cv=none; b=GvV++fVV5ezZcjVnweTs9GJebM/QAuYUZ65HQgvyF+YFjv28GLKRKiTwrsEhG+uvxB6wyJ gehjmddWcZ8UdKQljFsD3umvH8pH/6LvC9G/FvoXpVkuMNsq5PmrzzIS7haqIfVJQIROsg mdfCvlVQbav/0jWwA5qcRkLM7fxOI0M= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=KnhAt7qu; spf=pass (imf10.hostedemail.com: domain of 3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3m7ztYwYKCPAkmjWfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676524700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=4kD9dKn6hJEZvJZVtiuvyEMAdYFNYr0a2SKNPlTQzdLN98ZSKZVGf79JLGh86yfeyJ1ITp QItVmxZBgyi/f5pwudqD1TCsvM19+TJhchw6hiWN5PPXFa+u0qZRXW6n1X0XEyXT35gSnb lA6djHvuFUI3IZULA6IF/Rn4AUyKjYg= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-52ec2c6b694so8316627b3.19 for ; Wed, 15 Feb 2023 21:18:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=KnhAt7quYBwXQRhfuSYaJDek4wttI7RDXOOACIL/F63Nz/58eVv03No3Zqh2UIwNdl TkKGisyfzRrXImR2jOkU2wcQekf+kt9/XxXIsRoXlqskuIeX6hOhHTNZXgdfaMplle7K SAATQyrMS3HXKQy7j0lc9Kc3dK3KueDsfgI8ryy5OQhnNBROeaCbU2Hmg2SLUQOwNbgt XwBV7zK7XX+1ClOK24Y2b0Ex8amo9ljLG0ZokYkKzEniw9mpgT07P+84GOQVR154zkPK 4YJcp2AEGd79KcxB1dsy4RuRGjRlYt9yxaLz/0Wa2WtK7Y2fqnfOErlG1kh3RHBuBBJK 58yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nnxUMMQI6vT0M/Q5sa1IAaTl/WTbc27CfHmxL3/FviM=; b=FQKJEO12CVRsb4Tycx441XXUl0sAOSJv+rs9EAnK98iRiN5k0shfubvYj5atdDw3dU u/KbZdmg1m7gcszDhmn3Ptyu3VkjwzNtNwmM2eLJfBzwaVN+qzirLMvH6k2tsRaCIil3 V9BqCwfbpbv7K5ERX8603zb5XfQgHr/ekDjmFH5BKTM/t/5i1rJVW8MxIU5hcXwT7A2w FmfGojb9akOEXRc4qqygJH/z5hj/Bjq7zZ/TFJhpqvVOTv40s5rP2Lxvd5pNdldmYy9E BpbruFCUdcHjaZPxLFuM8Lr2QLpqtkRdgfVGysqB9hUSiHCkf0NFxMydxSnoAyjnvCE7 VBBw== X-Gm-Message-State: AO0yUKWPImTxOS4XanYxt/el36nUZpwYc8VuOtxH0Rcnc5bp6MvcwDzq cV6LnFscZ6+A3e4XY8FW7v3u89jr17k= X-Google-Smtp-Source: AK7set+tlzm8noztFtNGPsWUqf3Z7zno+mYjAZ9UmWznIF5F5gJ5LrgX6pBfJhOwGepp15lWUy19j/IuAtc= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:f781:d5ed:1806:6ebb]) (user=surenb job=sendgmr) by 2002:a81:a20c:0:b0:52e:f79e:b639 with SMTP id w12-20020a81a20c000000b0052ef79eb639mr389199ywg.283.1676524699450; Wed, 15 Feb 2023 21:18:19 -0800 (PST) Date: Wed, 15 Feb 2023 21:17:25 -0800 In-Reply-To: <20230216051750.3125598-1-surenb@google.com> Mime-Version: 1.0 References: <20230216051750.3125598-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.581.gbfd45094c4-goog Message-ID: <20230216051750.3125598-11-surenb@google.com> Subject: [PATCH v3 10/35] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Rspam-User: X-Rspamd-Queue-Id: 41C2FC0007 X-Rspamd-Server: rspam01 X-Stat-Signature: 8p5eqyrgrna845p5nsiujfoj7oxr1ori X-HE-Tag: 1676524700-528578 X-HE-Meta: U2FsdGVkX19MaQ3BNlC8P/MnGDtHDYzLcGEuLCfo541+NKm++mj0RsgWKD211FGP7+5sRuup0VpkCm/YHKLM8Uf3s1OLMH9ky3KWhNdJwBOQNFTFYQwcpLffd/7/Y1KyITsclpFH7GaVjdtV1QBZwo9kXRW+KcHaEeyyPP1ziYRTGzbB4RodvU6q4zorwK640fPZec5Sl0pH1MSN2k8ZUnEpScWjCIQmqKUmcWJifzqwU5WpJ5f45v0SZQ3mpSgky8N9VKXXMTouAGr/jVrb1IgR+RhfRwGbVdOnJVb9MhWjq9qf1qf0rixmZ2TNA3906ll5VJ9rtoGjq+31JqQLi+nA31Xin43xku/RLFg6R94jMN4uNH2gaNOAY6ssfcIviS3ZekGrhSyVf+D5e5TYx7fecud5pWviBCYKRmonEutsIftonupPfGVvBx9IElJszNflbDalp/oaj2X7H73EhbxwdzzhiFtHgTY1LvoXWqi5q1K9cGxSC2lzCLJeDXH5DUlVTMYEGIlN3eSPGdt2yJY4a4gIbsqNYD5PGQWoSewoo3kX2giqPwWc0CSNvk0+lX2YstvNpGwp0TKPXjdKbOufmwTkcmvq9Pd+LtM30GACsMEf2umwggccSIUt0YYvcxhrGTW//NkNi/0BEctMrIi/rwdjYrHsOay4LcJhNimUzCi9Q/IasJQLmlsNhG8lSIWivdFHJ6WselNZAOaRURZhf/jhms0GnzCffrfEJPmkk9M0PbC6biyC+2T34sg2zRcOX4nbKdI3NVCwdLJwyMdxAXwlJsF4Fl6vnNErSVEGAqkHuX24Zq8tVpea5E2xiLP1GjkXe/AIeKkBFq0GtVsOdVqqRPkD1sXMfcSQmg0RhBcR9FNHHtBgc4hJb1sKXf/hiTybn2pNZYdQVyfRhA0XUU/TSpyfWuvAUcHNigQ41XdZLTWFf2FqxDmxLFs3P/94pjiUJVFI5Nd/zPx UWzDyjwo g/gSYC9RLt6juy2U0tqRLU431DFIXMVPKcnE3i1Pj7QF3t3K2RDRa01YbqAQE5Ob1cSgMMwD4VXK7Cn2QtZ8lHGie0b+SBaZJxtpwF+mXDyMForndTXmHUCScHC8Y5cJYtPkoSsFsc3tPX3dFIlR8M9IBI78wHBIdFXVRW/KpC+YZiQWxZOm9JSrSWotmznSmrcURYjvlxwXY266oa5N++CBkJIk0SPu4TQQ1Cd03qWfewtUmqxgLyn8fuRZwq8cwacpwTzKOdN+N4rTsvfmgU2gtcKEpr23iELfrkLUx1ktl8rW6nn62Uy9mImSxIWMuYYiu6kK6+yzZNns4viWhOer04VqNQEBouyxhK9xo+xIVyxh0Ywxe5mZ2RUj6ZIgRA1rLTod14GomyRhGIaMv2UL2NNwLlfvD5XMLEliKDc3zjw5jrq+xAWDTtBCiwsKuHa2KE5CaRiyQ/XTgL5eksyZw8slkvpr+fUr+RoK6C73IOiaa43RaN8vCXXIQUo7dtZMmUOT/j56N7HwD3G27dvgz4bIMvjp/813CiP29hgl10C2DEPHytj6Q6i8XjAXGoo6mRVd++gLWsecrYcgck41d2AxTefTfbWMKQYawRs/AvwT3baJG1TJZCFQXcvhF8QNYBLzgZ7n4+IHtMUDDp4NL0mt8nfhyCNO+ZnXn6WogKTezzPag7PMqWg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 20 +++++++++++++++++++- 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3fd5305dbbf9..fb4e2afad787 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -480,9 +480,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 5f23d5e03362..314d51eb91da 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,12 +479,30 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -void vm_area_free(struct vm_area_struct *vma) +static void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } +#ifdef CONFIG_PER_VMA_LOCK +static void vm_area_free_rcu_cb(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + __vm_area_free(vma); +} +#endif + +void vm_area_free(struct vm_area_struct *vma) +{ +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb); +#else + __vm_area_free(vma); +#endif +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) {