From patchwork Tue Aug 22 18:41:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mateusz Guzik X-Patchwork-Id: 13361324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DE98EE49AB for ; Tue, 22 Aug 2023 18:42:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60D4A280058; Tue, 22 Aug 2023 14:42:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 56DB4280040; Tue, 22 Aug 2023 14:42:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E813280058; Tue, 22 Aug 2023 14:42:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2A176280040 for ; Tue, 22 Aug 2023 14:42:04 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 062111C9394 for ; Tue, 22 Aug 2023 18:42:04 +0000 (UTC) X-FDA: 81152610168.21.6D83F66 Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by imf21.hostedemail.com (Postfix) with ESMTP id 2DD471C0008 for ; Tue, 22 Aug 2023 18:42:01 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=QfVGsVPw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.167.41 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692729722; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mEOToIeoViTRF8c8rAS7lPuWoT5IGl18T901p8s8lTc=; b=TONMYC6RLl/kYAfwpWlOUHcsB3t3WsqjcasgcfruUzyMoKeckVkw4jjarn5sA+FBRXziKz 8dTCP8p8QZKd/ecTohiOSYBuKVspOv+mwQTCiVgt4tygkPQxNlD2fiBsYfQTI8urlBXmEt oJE3seMok+HlcdO7+swgcQf3ppQeq9s= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=QfVGsVPw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of mjguzik@gmail.com designates 209.85.167.41 as permitted sender) smtp.mailfrom=mjguzik@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692729722; a=rsa-sha256; cv=none; b=R3qt4SvZYCPP0f88eIoN1Nklvxqlj3AIIv9Rqlk0svVx7Evb5AMZAww74RHYNhWBE9paB1 5BOip3jTWiHDwe1yzBW9y7mRKnX1myAPft6XFA4ld4WQl8ATshvlwajzcKUu/fRi4Z2Nmk t+hQ52dNMtKtAB+5n4GntfcCtVJCL8w= Received: by mail-lf1-f41.google.com with SMTP id 2adb3069b0e04-5007abb15e9so4450416e87.0 for ; Tue, 22 Aug 2023 11:42:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692729720; x=1693334520; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mEOToIeoViTRF8c8rAS7lPuWoT5IGl18T901p8s8lTc=; b=QfVGsVPwABxhEcYrvXbcmGiUuRvcJXymMpVHwTotunc6jGK3pxIitnK0pOSZRFmEuL oXQs8Q+9oKQP1g8HsM+D9ok35AXs4Q8BZUmGNdnBekmT+68LHd+hhZPCs9wHNZ3jhRgI Vt3/bJlJuCKxjpWBm1O+QxCqEFTn1KPle9ZPbnJa0Z+fOom8Xqlq3gvp71cs9gA3QHHB ginjoZv3MmDg3zGF/e+ejtBgdIEqcgUcmM2ZFdFCZ3XMR7dmlwZ6qhx7HaIW6rc0zrX9 CjO/UNJsvzEP05/0XLKfCrEX730ap91i8SeofwitpKE0UWFbkwYzwE72YWNVtvqJbPPB CkbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692729720; x=1693334520; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mEOToIeoViTRF8c8rAS7lPuWoT5IGl18T901p8s8lTc=; b=WbVtkUxA1oowJodLmOZVBWy9B2veaVZ63OwM1mRCVtmVFTm9sRIpx5e7V+6rnsnxza 4kAcULWd7APKopPMFVtdyGqvgH88lQaXuy9u1AnRzNLVFVQeXZe6n97JtyKI0m8uNH8m dOmQ8hg0XCfq69UR5DHRBXWYbsycmJGf8TZhADOIJtR+LsPP64jaNnhjLuCT7dmq7JwV JNQlZIKnWumRVwri0CGhHlxx4QCwmV7kfm7rJdBNQGGVGAKvgW7yuWRcIYgCJ956/WYR M9gyukwbMdPdADAgmalJle2Lp2T8GJR+rJuVoVbFbX6lpbSCFHAMCG/gmheW7Dec6LDQ SSAw== X-Gm-Message-State: AOJu0Yy24/G0JG+419Gf5NUk8O+UZEX5ARf4RnyNLlnCl2L3pgA6sTDi EzrE4mFT7pwbMRXCsrMVS0E= X-Google-Smtp-Source: AGHT+IEL+C5O6TMeHs29lajaMbWo+0Cbc3ZEq9kla6k40ShytD8+zBpbctMDQ6f76ngrJSIhW5Btog== X-Received: by 2002:a05:6512:39ca:b0:4fe:82a7:814d with SMTP id k10-20020a05651239ca00b004fe82a7814dmr8200504lfu.48.1692729720458; Tue, 22 Aug 2023 11:42:00 -0700 (PDT) Received: from f.. (cst-prg-85-121.cust.vodafone.cz. [46.135.85.121]) by smtp.gmail.com with ESMTPSA id w9-20020a056402070900b00525683f9b2fsm7945317edx.5.2023.08.22.11.41.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 11:42:00 -0700 (PDT) From: Mateusz Guzik To: linux-kernel@vger.kernel.org Cc: dennis@kernel.org, tj@kernel.org, cl@linux.com, akpm@linux-foundation.org, shakeelb@google.com, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH v2 2/2] fork: group allocation of per-cpu counters for mm struct Date: Tue, 22 Aug 2023 20:41:52 +0200 Message-Id: <20230822184152.2194558-3-mjguzik@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230822184152.2194558-1-mjguzik@gmail.com> References: <20230822184152.2194558-1-mjguzik@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2DD471C0008 X-Stat-Signature: pz98o4hq6jbt5ysifzxfwjoo5qeinn5w X-HE-Tag: 1692729721-241946 X-HE-Meta: U2FsdGVkX1+jgJCHe5f8TchWcEPQ4o/LE6bpmLHC5BMb7B+4f2l1eQfU9RecuPBZnJWG5W1vrEKYj/8fPCjyJX8UgVqWYKi9qcLQCv+cE+/pZLvQos0u0i6zvKr1m7g4QLzdntjnOsfZLJBCUOgo3Egu8TJf+KuJJCD5lui3l+jYDwm4MtzJEl9uHGQYNXzNUk7hyWRHuIuUHhyrS/up2RNlQMYgDYYps+B+iwzUscyUdAhZlm5Bp6IpcG5NNwQYdZlAgOXKakv1YzlETnSIKegYw73QpyLzwg0dY89zaf/wjLha/mtUuFI0b01N2/6nMQk4NT+kLUG3a8Woy+PuQXQCJU6ogPQ5rKTho6fCtAxZBdNa2XIaByIXa1d8NKZZ+9JNQlMyMqhWMpEXg90/enBEnzPOGlAYrHZ4COqExjiRWTPXcYEvfWNSAfKAHAmWd6UoGKGj1ZwVernloGkfuVjd9C5HE3cgoooNmRXYi9B9Sr156fdNntaJCD1Bbg+83/1iYAMsFjL1Rfq146lZpg6qFdCLYyh6zp2DUSZLewnrGH0CbS+9WKKph56L51FijIVdlyE7D3MVceJYvRXBn2Rx6qyLPXbXaU4WuBwIkMKM5TOHUF4iIG4tcTC+1qVSroLK6QoGKv95LpN8w/e8s3kIssWm5nq1Ybtrq8s1VHdKSnAb8zb1rdZhI9Azg+4inDN9SDodjf0rSDiuqH7ZN6k6BTfXEGU3Bt4xTvkp6h3v9vnNCMOraOlx4BNLiWIvmgrNr1snEtGeMJE3KJiXNIaQjajU5cLTDN5PjqaEf+FlYZInPFq61opAQNGCkb/0n2K/FIZMTV0yP8SK72ZCHV2f1lHRsgHo5XKIIPT84D7bOU4YomUJCoWBPhmxakcQOT/MIyMMSWEg5tbK32Lod8v9nli3ZHd5UD5YYrJDbGE6RhOiKmNPWGHf+pcBIozVlJPwETAD/oxxyIAPvVe TiFPWPK8 IURMamXqSTsVjwjwgJGWSyZVq16HmzIrnjbnOEHL7EYlOez9znnzCTuZyjEnScCKo81lwJkpiEou7wkT2SGfiQccKWWDTkmncAc/kisNLGfhFOkPd15JfngHH4m6RueHMsYNtPPNdWoh4iXtkOHgRDLJ89txyQwJyqPb4weaNYKt0/J96TkXuoseuCOCSnMuuJv1+AjKeF1LQPrgZx7CIk86CKhZONOTCy+p1vPBWFSR/AISgWNiCYTqgR13XVcn0qklTL3eKBaxi5gU/dLef3xaksF2aGaAmlYFsBVMcP8ZSSmJxnc+x9JBPQ/wDbO4m77AiA7p1nM0XZnz8BNFrcslJ2q2qe2yo5tempz/lbuHgUEfy/bufXOpwLWhoZnwfUccGDoN9aXuM9ClhUWxFW3pFb9IUk2ecAU/LtkMEuzJujP22wx4m8AGK4Q+th/BqC3Bn/dVaF99tUHHMXmKO7O5mZ2t2rNkGt4lnspEfYfaI6tL59RQT0Mui8A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A trivial execve scalability test which tries to be very friendly (statically linked binaries, all separate) is predominantly bottlenecked by back-to-back per-cpu counter allocations which serialize on global locks. Ease the pain by allocating and freeing them in one go. Bench can be found here: http://apollo.backplane.com/DFlyMisc/doexec.c $ cc -static -O2 -o static-doexec doexec.c $ ./static-doexec $(nproc) Even at a very modest scale of 26 cores (ops/s): before: 133543.63 after: 186061.81 (+39%) While with the patch these allocations remain a significant problem, the primary bottleneck shifts to: __pv_queued_spin_lock_slowpath+1 _raw_spin_lock_irqsave+57 folio_lruvec_lock_irqsave+91 release_pages+590 tlb_batch_pages_flush+61 tlb_finish_mmu+101 exit_mmap+327 __mmput+61 begin_new_exec+1245 load_elf_binary+712 bprm_execve+644 do_execveat_common.isra.0+429 __x64_sys_execve+50 do_syscall_64+46 entry_SYSCALL_64_after_hwframe+110 Signed-off-by: Mateusz Guzik --- kernel/fork.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index d2e12b6d2b18..4f0ada33457e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -909,8 +909,6 @@ static void cleanup_lazy_tlbs(struct mm_struct *mm) */ void __mmdrop(struct mm_struct *mm) { - int i; - BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); @@ -925,9 +923,8 @@ void __mmdrop(struct mm_struct *mm) put_user_ns(mm->user_ns); mm_pasid_drop(mm); mm_destroy_cid(mm); + percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS); - for (i = 0; i < NR_MM_COUNTERS; i++) - percpu_counter_destroy(&mm->rss_stat[i]); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); @@ -1252,8 +1249,6 @@ static void mm_init_uprobes_state(struct mm_struct *mm) static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, struct user_namespace *user_ns) { - int i; - mt_init_flags(&mm->mm_mt, MM_MT_FLAGS); mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock); atomic_set(&mm->mm_users, 1); @@ -1301,17 +1296,14 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, if (mm_alloc_cid(mm)) goto fail_cid; - for (i = 0; i < NR_MM_COUNTERS; i++) - if (percpu_counter_init(&mm->rss_stat[i], 0, GFP_KERNEL_ACCOUNT)) - goto fail_pcpu; + if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT, NR_MM_COUNTERS)) + goto fail_pcpu; mm->user_ns = get_user_ns(user_ns); lru_gen_init_mm(mm); return mm; fail_pcpu: - while (i > 0) - percpu_counter_destroy(&mm->rss_stat[--i]); mm_destroy_cid(mm); fail_cid: destroy_context(mm);