From patchwork Thu Apr 8 03:57:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 12189911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 729FCC433B4 for ; Thu, 8 Apr 2021 03:57:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EDCAC610FA for ; Thu, 8 Apr 2021 03:57:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EDCAC610FA Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BEF266B0083; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9E1E6B0080; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B34B6B0083; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4F5F46B0080 for ; Wed, 7 Apr 2021 23:57:51 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0F2015DDC for ; Thu, 8 Apr 2021 03:57:51 +0000 (UTC) X-FDA: 78007841142.26.EF5987E Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf10.hostedemail.com (Postfix) with ESMTP id 6336B40002CD for ; Thu, 8 Apr 2021 03:57:47 +0000 (UTC) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 1383u7Wl004488 for ; Wed, 7 Apr 2021 20:57:50 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : content-type : content-transfer-encoding : mime-version; s=facebook; bh=1S0iKb3KcPWfTp4X6pJq8tXj56UgxHzPrVzYK3Diahw=; b=mLcloA3JvLMDNz1/RIn8JVf2z+K4k+NSwBus6JviYOGr+2vzRNAVwvkoWTw3ZdEsBWTf IzuymoSBRoMhX7rqB/mcOpqIEOSrgxzJ8exkQAzkafWbJpcsurylVqA/rvxIdVF1X96F RZ5/WYvQSMvFmRJjrsbg+6Z/yHnLRTTQQIk= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com with ESMTP id 37sh2nk8bm-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 07 Apr 2021 20:57:50 -0700 Received: from intmgw003.48.prn1.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Wed, 7 Apr 2021 20:57:48 -0700 Received: by devvm3388.prn0.facebook.com (Postfix, from userid 111017) id 55E2C602B00A; Wed, 7 Apr 2021 20:57:41 -0700 (PDT) From: Roman Gushchin To: Dennis Zhou CC: Tejun Heo , Christoph Lameter , Andrew Morton , Vlastimil Babka , , , Roman Gushchin Subject: [PATCH v3 0/6] percpu: partial chunk depopulation Date: Wed, 7 Apr 2021 20:57:30 -0700 Message-ID: <20210408035736.883861-1-guro@fb.com> X-Mailer: git-send-email 2.30.2 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Lag9t85RWaljyUog5SdXK_uy_WY9D4A1 X-Proofpoint-GUID: Lag9t85RWaljyUog5SdXK_uy_WY9D4A1 X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.761 definitions=2021-04-08_01:2021-04-07,2021-04-08 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 impostorscore=0 bulkscore=0 spamscore=0 priorityscore=1501 mlxscore=0 mlxlogscore=999 phishscore=0 adultscore=0 suspectscore=0 lowpriorityscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104060000 definitions=main-2104080023 X-FB-Internal: deliver X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 6336B40002CD X-Stat-Signature: 8bmouz1eyapo81a3bzijjfjukdxg7xn9 Received-SPF: none (fb.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=mx0b-00082601.pphosted.com; client-ip=67.231.153.30 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617854267-659681 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In our production experience the percpu memory allocator is sometimes struggling with returning the memory to the system. A typical example is a creation of several thousands memory cgroups (each has several chunks of the percpu data used for vmstats, vmevents, ref counters etc). Deletion and complete releasing of these cgroups doesn't always lead to a shrinkage of the percpu memory, so that sometimes there are several GB's of memory wasted. The underlying problem is the fragmentation: to release an underlying chunk all percpu allocations should be released first. The percpu allocator tends to top up chunks to improve the utilization. It means new small-ish allocations (e.g. percpu ref counters) are placed onto almost filled old-ish chunks, effectively pinning them in memory. This patchset solves this problem by implementing a partial depopulation of percpu chunks: chunks with many empty pages are being asynchronously depopulated and the pages are returned to the system. To illustrate the problem the following script can be used: Signed-off-by: Roman Gushchin --- #!/bin/bash cd /sys/fs/cgroup mkdir percpu_test echo "+memory" > percpu_test/cgroup.subtree_control cat /proc/meminfo | grep Percpu for i in `seq 1 1000`; do mkdir percpu_test/cg_"${i}" for j in `seq 1 10`; do mkdir percpu_test/cg_"${i}"_"${j}" done done cat /proc/meminfo | grep Percpu for i in `seq 1 1000`; do for j in `seq 1 10`; do rmdir percpu_test/cg_"${i}"_"${j}" done done sleep 10 cat /proc/meminfo | grep Percpu for i in `seq 1 1000`; do rmdir percpu_test/cg_"${i}" done rmdir percpu_test -- It creates 11000 memory cgroups and removes every 10 out of 11. It prints the initial size of the percpu memory, the size after creating all cgroups and the size after deleting most of them. Results: vanilla: ./percpu_test.sh Percpu: 7488 kB Percpu: 481152 kB Percpu: 481152 kB with this patchset applied: ./percpu_test.sh Percpu: 7488 kB Percpu: 481408 kB Percpu: 135552 kB So the total size of the percpu memory was reduced by more than 3.5 times. v3: - introduced pcpu_check_chunk_hint() - fixed a bug related to the hint check - minor cosmetic changes - s/pretends/fixes (cc Vlastimil) v2: - depopulated chunks are sidelined - depopulation happens in the reverse order - depopulate list made per-chunk type - better results due to better heuristics v1: - depopulation heuristics changed and optimized - chunks are put into a separate list, depopulation scan this list - chunk->isolated is introduced, chunk->depopulate is dropped - rearranged patches a bit - fixed a panic discovered by krobot - made pcpu_nr_empty_pop_pages per chunk type - minor fixes rfc: https://lwn.net/Articles/850508/ Roman Gushchin (6): percpu: fix a comment about the chunks ordering percpu: split __pcpu_balance_workfn() percpu: make pcpu_nr_empty_pop_pages per chunk type percpu: generalize pcpu_balance_populated() percpu: factor out pcpu_check_chunk_hint() percpu: implement partial chunk depopulation mm/percpu-internal.h | 4 +- mm/percpu-stats.c | 9 +- mm/percpu.c | 306 +++++++++++++++++++++++++++++++++++-------- 3 files changed, 261 insertions(+), 58 deletions(-)