From patchwork Mon Feb 27 17:36:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13153995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 683D6C7EE2E for ; Mon, 27 Feb 2023 17:37:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2F7128000C; Mon, 27 Feb 2023 12:37:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96AA9280001; Mon, 27 Feb 2023 12:37:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80B4628000C; Mon, 27 Feb 2023 12:37:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 63C41280001 for ; Mon, 27 Feb 2023 12:37:56 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4C20B120408 for ; Mon, 27 Feb 2023 17:37:56 +0000 (UTC) X-FDA: 80513779752.08.5A36DEC Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 7C3DC4001C for ; Mon, 27 Feb 2023 17:37:54 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AIld2qfh; spf=pass (imf04.hostedemail.com: domain of 3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677519474; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F3L0KDnzHjfWSabmEfm0QwnSpYouz37Z4kQXdx4wyok=; b=P9g1GR+q9Ob7BzbtQodHdZeQfujPzQueZrgt+4y2eF64RTK5U8DfMyNuT8J+avQnyH0a2c OhJQVwiAdyT0LZBJ+3tYrlDquLkja7hIPCepzvPSJdSyLImvm1HWbeLI4s19v3B8Yzk2lw VpufjPhWrjmUrfFTgy2Ra+vEMzL8PCs= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AIld2qfh; spf=pass (imf04.hostedemail.com: domain of 3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3cer8YwYKCGAQSPCL9EMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677519474; a=rsa-sha256; cv=none; b=eVxrfFjGO6kHJy3cRbfwiR1AYNDu8ytWSEaNpOzKHuuYRnwipAiz+dak/pmmCKSnbZy1DF vt7aB6u7ckFJeXjH/Fj8TAtnREaoglhAQK+BXNf6ifiWNGDCooNP3cmVMvfvECvl3fWECB RMRoRiZDMGFIiubO/y4ofV1ObzAfzrU= Received: by mail-pj1-f73.google.com with SMTP id i6-20020a17090a650600b002372da4819eso1871270pjj.0 for ; Mon, 27 Feb 2023 09:37:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=F3L0KDnzHjfWSabmEfm0QwnSpYouz37Z4kQXdx4wyok=; b=AIld2qfh1hJ81pjBHIB2k6hHEBq4B9cQ7unRZ/ajU5zus4NStrM4vKSoC+D3xf5i2V I3lWxazmPww6pY3JnOd6ILpcq/3gMzUiBxySkT7gq+qA5zIjvIc98kXlCsV1T3OkgNoC wkB+0CjdoEsCW0SjfCSG1j8AzXTBtq+xLwxdRi9trKrWP6pzQ/z8RPgzdw1aTSBFWEFp tuWqf3fIoKXV2B5eA1r5q/mXlFY1b1mqVSiItBYfUpJVFKPT1XTPlAbMteQM7dFl9Yt7 rI/aJtYFzH6GTH9qBEbz2dvU/L39seWUdch/8Eafr+7mGpiemwVm0woR+UJD5ik1qt28 1vdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F3L0KDnzHjfWSabmEfm0QwnSpYouz37Z4kQXdx4wyok=; b=OOpHVuxZsgWUfB+3/hm/e2lIYtbJuRzqBskWy5wF4nnVBtjm9N1kHZCgTxz7LizWJN uHrZc497jfKfroevGTWG20T85m5Kij+MGNkmdk1Ibp7wYPGGTacl7tqpcLwrK39rUBmz /Mtk88G/qKmCjuL0APdPR88JD7JSCSAJT0rv4XlkdWup9TP23AcdxTa4LH1Qa0o2yEC3 usU1f41hnXFBWcCb2ac9SJdfuFKChuG7ClHlyQJ4nWBmWX8zKrYAdTciuYVnlDQM6ofL VR85EDffl16WX1FdHA3PlheW/uqZpN4EckRuHerYwtmSldRoGGFvliYVEtczcZd1YPmc PIAA== X-Gm-Message-State: AO0yUKUmOOKOcvDVkb3/7rOfSm0y69IXmbijaa1njvCX+dOKWqGYmFSi tbzI4pYIbASTCTzwbd4L+gkvH16Gp2U= X-Google-Smtp-Source: AK7set/h2/2/QuyVN0yF0KgPZgsVR0g2a/0YsQFw2JhIAgW+zKPw0n5+assM7s0mQSWtQbHur04pQimjpmw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:e1f6:21d1:eead:3897]) (user=surenb job=sendgmr) by 2002:a17:90a:b388:b0:234:ba6f:c97a with SMTP id e8-20020a17090ab38800b00234ba6fc97amr24135pjr.3.1677519473325; Mon, 27 Feb 2023 09:37:53 -0800 (PST) Date: Mon, 27 Feb 2023 09:36:31 -0800 In-Reply-To: <20230227173632.3292573-1-surenb@google.com> Mime-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> X-Mailer: git-send-email 2.39.2.722.g9855ee24e9-goog Message-ID: <20230227173632.3292573-33-surenb@google.com> Subject: [PATCH v4 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, Suren Baghdasaryan X-Stat-Signature: 3awegoiagdegh699w8xg1smsrkejgpz4 X-Rspam-User: X-Rspamd-Queue-Id: 7C3DC4001C X-Rspamd-Server: rspam06 X-HE-Tag: 1677519474-133506 X-HE-Meta: U2FsdGVkX18XBIwYen47h3ZvomU8YCb0O3W7oV974l9cLYr7X6hIhYDnPJdT5XnPMxI4csTB2H7qLg1cXDELNYtOZjHWmSPuIehOWHY4GOisAD263X4+5UQbugVItfYJupz/JAw9yfHkF2pIi11szbLG0tk9QtdLfz9zlL+CuLII9lZeMyxebhe2143bO1IAyBA1KDrREEU2PfDGCC1DyAlXZ+WlqWwJejMuPN3VQinfAHTQo/2W1JdXuzOMq1+KivGkP8vGmM6KTVMNR57nRzop1Dbo1HfU3NXjdCKAm+jh553Z0QZohnr5iDTAotdZ6NiDIY73t6/hm73qgi67TeNhDABqOIcaiqOL7uTtVVZ5ysTEmFsisW/bVOlw91paligopER1OYtH5aN+qnj/1OIj9CORzAN/fhNQQpDFPJT2YC5hMkg5OXgF8B24RUPvBhdpGRy0xURGcP/vl7FK/Zvcd+FZWsOwI62n3f2QXMEw3difXTNknbYB4jKp0XsNXnygee2R+YFJTVvv7hpSQn8RDB6k38FBOyw1DysSJUksGcf/2RtuAbxpmc7Xvmxizw3oNR/qXqjhZFPJDtPW36s0DEwhmZ4f2SqKWmbhP3GGIFHqW2sHuZ80k3Lqy7I8gdcs46+DfWXWYDxxmbW1Va/BoibZi6dVpwBra4+00H/erFZDJwjj7VjOqt6TacXFpUGfgW8tMpQNWTLrAmyUCaUULYM4hVSbsRDAWFPxHUOHIw5RE/LhZtnsiPsxT9uYei0FmEThm2Cb+uOfGFvuVP4zsTPayhMCWUKEl4ubotLeWFGwoWS9k91qQthJg9bEGgWlAE9ojWFvyr/QEegrApY4JJ0q+WFtCCBLnam9qN1toK8RU0L1rEFRzT/fioXSacgEEmBoWQQeD8b6qGsNeqkwT+NLe7lEhuW8NXVWk+8mhR6/5uDOOZkjCuMuArAMrimfXFE0a7CutwFlT0G S/mnnBq8 SaxgMkGLF9TRBVXm8M3k9qPe7ZlpmoY4S+wJgaK2dXkOQ/TdTW9yGuhWd7539hQFEnvQqppFU6I+KGG5zOTa/OtXIUrJwneazttwEAoqCsKJZGLCNCz7ycTYc8yYJFS4HCoovDGTFkYYG27qSMlzvfzHti5Z9FvVMb6NmLiEBv/lIj+eSmazK90RjwTSH9leYD0v0lC4I53X6tsDSZBomV5NjtMk+2RSF6v1mcoEk5hXxBBwd5QRu2CAu48slwjyfjjIC8/4PM17wjaXeTjrRqS92LnUuupNBr9cyYnoqwecPukIXHzeD5yYdZ6gBDgSkEDB6htFlDK3zEoTDfifyg9zNHd9UH/HdTPIszUInzfA05W3pbqL7pOVvGUkLP8bCUUsPakfqOM0XcD5zwg+NuaryH3ngyXS9lQ/Zo3nJXPAHlBIX7S4HXqzdHHcNEtmamBLnlIFAnwoQqrje3oFlhPFJn3qfBytQyYU94rLgpxks8Uayi4v1avcGHiFwqLwbEayU7pymTPF2xtV+BEXfPgcCstTMSoL8K84o5wEvSqQ11CNwmLUpC3lFCc1xW56otSWmkc1gP0gFnrrS3fRhv+00MwaHA+Iv/tdyxNIhTCk96VPuQ4UayBYMyaz8uLzUZ0C8Bx5ZLTUyfJZLER0l1NgmylEaRaPHvlZ2H9vC2tYbyZU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d07ac923333f..5e142bfe7a58 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -256,6 +256,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index bdb55f25895d..ad37f1d0c5ab 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index df13c33498db..0cd3714c2182 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2130,7 +2133,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3070,7 +3073,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);