From patchwork Fri Oct 30 01:38:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wonhyuk Yang X-Patchwork-Id: 11868191 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9542B61C for ; Fri, 30 Oct 2020 01:41:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 20AD720719 for ; Fri, 30 Oct 2020 01:40:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rzABeJjb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 20AD720719 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id ECB4D6B005C; Thu, 29 Oct 2020 21:40:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E54636B005D; Thu, 29 Oct 2020 21:40:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF4866B0062; Thu, 29 Oct 2020 21:40:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id 99E5C6B005C for ; Thu, 29 Oct 2020 21:40:58 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 34C14180AD807 for ; Fri, 30 Oct 2020 01:40:58 +0000 (UTC) X-FDA: 77426888196.09.nail46_2f0809027292 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 19301180AD806 for ; Fri, 30 Oct 2020 01:40:58 +0000 (UTC) X-Spam-Summary: 1,0,0,2a5912a5a2f01c2e,d41d8cd98f00b204,vvghjk1234@gmail.com,,RULES_HIT:41:69:355:379:541:560:800:960:973:988:989:1260:1345:1437:1534:1542:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3165:3354:3865:3866:3867:3868:3870:3871:3874:4321:5007:6261:6653:7514:7576:7903:9413:10004:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12683:12895:13141:13161:13229:13230:14181:14394:14687:14721:21080:21324:21444:21451:21611:21627:21666:30034:30054,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04ygdzw9etjdqw4hgyatcbm3f4wqoyciyjghs9p7w4ccc4a71wsjeot5qbh839e.9ueafxeqzjbo5zkdn5ja5pfe3gum9zay9oot395i6n1qejytn91fs9fejqd9iem.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:67,LUA_SUMMARY:none X-HE-Tag: nail46_2f0809027292 X-Filterd-Recvd-Size: 4964 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Fri, 30 Oct 2020 01:40:57 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id h6so3847624pgk.4 for ; Thu, 29 Oct 2020 18:40:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=AZcSyo5g/nEswTTuCvZSdfCYIDYbgJRtMj8/IIpSBag=; b=rzABeJjbRo85BMF1iHu5Af6UBaJtXgDcgSwaXigKAQ4lNEqTgPog+0tRfpW0H/3aWE ominm6dXgKO1Pv3fmjlbvg9lxEp+N0aYXGIb36DtbkgldPJbSOnDnsEr5SacDx/hE/sN PX6SdY3nRNxkfJ9MGX1Z1NxhCRWl2VwFc7DyjqVT5m0S+z+A17z7LMEN/J2Ud3qGFO51 h44mUG1r5i1Vk1r4iTbm3/Ph8asrLxJzoyCJ5RZIIQn5WIyYl/aQdA1CyuquPe50YjIt 9YxQ2cDUkdEFNMaEx2Aj+6wqxnCV5CWjTTrwf1LJ2MnzdBGSUojNCMkpBfOjGunQbHpp qb0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=AZcSyo5g/nEswTTuCvZSdfCYIDYbgJRtMj8/IIpSBag=; b=RGFB/97LASV8e/kzZW4Ji2Inx3VzMG5t9Wr1abAgqP+UtvFaUBm23Ul3eUuZv12O7l hR5VaYdcShySmwlaAn75h6UWqUqcwLP3UbPRZ/QF5iqKhlewbPl3vArGyg12r4HQOSrU VHgRNF2lVEQ6pqUytQ4Qrrn4Eh5l47qv2/rKDJmPB4/EP2kXqSfUf+RfmAs1q6EjG/Jg c/+ZC2mrFYY4Gc8s0uVyJGPn2sJv/a4Dz/Vm485V7rc4Y0RI9rb9JiFQtX/4tonh8ETW FSLRhMhErJcUYSjY3EJW3IyGkjcfe0KQ2YZY79tre66lZ4rs/sTsD+E9jDb3FxVoQJ1O u0Ww== X-Gm-Message-State: AOAM533qI5+NvjB/3UT0tqfXeHrdSc9aAKh/jfOAxa2dsbyhysN2t26G E68sMseR8lVK052PimkmgPE= X-Google-Smtp-Source: ABdhPJzciEVdFyqCCQpYDCSDHtcfSVYvsDop12mcyH58dhOaN7wurQOghKpm69j5uRM7acH9QwPoLw== X-Received: by 2002:a17:90a:e998:: with SMTP id v24mr2056935pjy.5.1604022056659; Thu, 29 Oct 2020 18:40:56 -0700 (PDT) Received: from DESKTOP-P2JGRFE.localdomain ([1.234.114.36]) by smtp.gmail.com with ESMTPSA id r8sm1548358pga.33.2020.10.29.18.40.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Oct 2020 18:40:55 -0700 (PDT) From: Wonhuyk Yang To: Dennis Zhou Cc: Tejun Heo , Christoph Lameter , linux-mm@kvack.org, Wonhyuk Yang Subject: [PATCH] percpu: Reduce the number of cpu distance comparisions Date: Fri, 30 Oct 2020 10:38:20 +0900 Message-Id: <20201030013820.29758-1-vvghjk1234@gmail.com> X-Mailer: git-send-email 2.17.1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000038, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Wonhyuk Yang To build group_map[] and group_cnt[], we find out which group CPUs belong to by comparing the distance of the cpu. However, this includes cases where comparisons are not required. This patch uses a bitmap to record CPUs that is not classified in the group. CPUs that we know which group they belong to should be cleared from the bitmap. In result, we can reduce the number of unnecessary comparisons. Signed-off-by: Wonhyuk Yang --- mm/percpu.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 66a93f096394..d19ca484eee4 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -2669,6 +2669,7 @@ static struct pcpu_alloc_info * __init pcpu_build_alloc_info( { static int group_map[NR_CPUS] __initdata; static int group_cnt[NR_CPUS] __initdata; + static struct cpumask mask __initdata; const size_t static_size = __per_cpu_end - __per_cpu_start; int nr_groups = 1, nr_units = 0; size_t size_sum, min_unit_size, alloc_size; @@ -2702,24 +2703,27 @@ static struct pcpu_alloc_info * __init pcpu_build_alloc_info( upa--; max_upa = upa; + cpumask_copy(&mask, cpu_possible_mask); + /* group cpus according to their proximity */ - for_each_possible_cpu(cpu) { - group = 0; - next_group: - for_each_possible_cpu(tcpu) { - if (cpu == tcpu) - break; - if (group_map[tcpu] == group && cpu_distance_fn && - (cpu_distance_fn(cpu, tcpu) > LOCAL_DISTANCE || - cpu_distance_fn(tcpu, cpu) > LOCAL_DISTANCE)) { - group++; - nr_groups = max(nr_groups, group + 1); - goto next_group; - } - } + for (group = 0; !cpumask_empty(&mask); group++) { + /* pop the group's first cpu */ + cpu = cpumask_first(&mask); group_map[cpu] = group; group_cnt[group]++; + cpumask_clear_cpu(cpu, &mask); + + for_each_cpu(tcpu, &mask) { + if (!cpu_distance_fn || + (cpu_distance_fn(cpu, tcpu) == LOCAL_DISTANCE && + cpu_distance_fn(tcpu, cpu) == LOCAL_DISTANCE)) { + group_map[tcpu] = group; + group_cnt[group]++; + cpumask_clear_cpu(tcpu, &mask); + } + } } + nr_groups = group; /* * Wasted space is caused by a ratio imbalance of upa to group_cnt.