From patchwork Tue Jul 2 08:44:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huan Yang X-Patchwork-Id: 13719116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4544BC3064D for ; Tue, 2 Jul 2024 08:47:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D72836B0095; Tue, 2 Jul 2024 04:47:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CFA636B0096; Tue, 2 Jul 2024 04:47:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AFF226B0098; Tue, 2 Jul 2024 04:47:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 888746B0095 for ; Tue, 2 Jul 2024 04:47:57 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 49ED9A19C3 for ; Tue, 2 Jul 2024 08:47:57 +0000 (UTC) X-FDA: 82294184994.30.F164F32 Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2051.outbound.protection.outlook.com [40.107.117.51]) by imf05.hostedemail.com (Postfix) with ESMTP id 4405B10000A for ; Tue, 2 Jul 2024 08:47:53 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=Uleg6qOv; spf=pass (imf05.hostedemail.com: domain of link@vivo.com designates 40.107.117.51 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719910057; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HdNNIIt5i57KrDH2G5inTTWNfKhy1MKf1/9NPo25L0U=; b=Scu0Ji0dsi2aX4spTDvvq7zyzZn166GgxGR5CRKXA6Qmdxlj5fOoAttuJLbrIr9WfrSyvc EhRD2KazwkrEgDswbbDmvT5sFYEouZGWbOjDnjyiiv56NZlGUJz4She3N1AH2ROUpRJ8Lg P2DPKn21Vx+kO84PrTamQpWwIhL/g6k= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=Uleg6qOv; spf=pass (imf05.hostedemail.com: domain of link@vivo.com designates 40.107.117.51 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1719910057; a=rsa-sha256; cv=pass; b=C1UEG2J177Bp3W2WHE4uOY6Np984F0C7GKAVzlOXGmTnOPUJxI5tvEEVOSE334GBjE8jqV 1ROcT3tkzAv2iO6UXykj5zYnspcwrb+6WuR5sUkNvn6rhbeDdPFEqkkg64L9aZSHjDkF9o 6twgdUEq0vjWb0CduZbx3Om+DqH4iH4= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PArtcsHgRAOSoH4wlQV0BTnJwYMYtPxo9TNyjMxlu+Io0MD6KyPqi640mghyMjGo2MhaESfmWTiFGAP3zFvPZvVhnhSbxOszrRKQpJ+WoUFnm+DmLJcwqc0Ut3wxe5+6B/SXC6WqZAvAan4tnspWxMiznXHbwWr23D/f3LimnOU9gHMwlyEp92zD+ZnkcM6+YbbuVtloeAB7KmO6e3uGVpnkWdyLuPMBzmq93mckW11oLse4pdXgIjLz5OTbzYugTaWqaisa205A2VSG2GERD/xMqJABcTnTyyYA1mD6T1FUYeiX+yeJ9IIwyZCk4DjOsYJdb4E7lkDiYMFMHeeaAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HdNNIIt5i57KrDH2G5inTTWNfKhy1MKf1/9NPo25L0U=; b=ZyE/rjfdNJvt1dGaNUOndoRB4Z3p7ZGQPywyq5lCS1muakE8fnop1g5OJpxtlWwFjlAOT5k1OwAq3jCXTuvCZlD4lxntLbfe/0gU+Y9F/R9u7+L9H6NcP+mHmG5HT3MPvLMGOnZE69lXo9kni0LaXA5quJScAwHPP0wr3qLDAiqs/94/mKc5z979BNV8CRY7Zbs77D6uWFJmfYiNRaC7LkJr7UFDcQUa1auIaBqDMgxvK+cUBU1u1mK5iRKKh2fNinbrvQpMNSR27Z7rYxlts9IVm/legA9w4M51x1Ry//O5C9X1lW9Hp4mRf+1CdKC+WbYZwuDKgzWTVbfVbHDurg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HdNNIIt5i57KrDH2G5inTTWNfKhy1MKf1/9NPo25L0U=; b=Uleg6qOvvtBmqW/bniZ1yZvAX/tErIBcHdw7gJV6hKmLW1fQrnwPboNr2XAIdhL35QqJoJ6d0G6wHcQunSRsrZdclBX7MPPO6lQOyvT65Y1ox07rY69CkadkOuzUXL+BhhVac08YoHfq5L2KeCWDQVGLP4lMAaQjAX5FVECqxui+/2vqkkQuSfP3mNmEjhldD/RELjPfjYchPRh2SFugas4ZQWrtbqCXHV2iTSnWGcNWJ4e3qYhiCkIdZfLeaINF2cFOc+PzJiKJ6QLmL40by/5aL/Gck3j3UPOr7NOg2aEw9mv0H/+04uOQlimQamPRNbyAc29U0eulwOXbv0A0BQ== Received: from PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) by SEYPR06MB5790.apcprd06.prod.outlook.com (2603:1096:101:b9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.23; Tue, 2 Jul 2024 08:46:05 +0000 Received: from PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f]) by PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f%6]) with mapi id 15.20.7719.028; Tue, 2 Jul 2024 08:46:05 +0000 From: Huan Yang To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , Ryan Roberts , Chris Li , Dan Schatzberg , Huan Yang , Kairui Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christian Brauner Cc: opensource.kernel@vivo.com Subject: [RFC PATCH 1/4] mm: memcg: pmc framework Date: Tue, 2 Jul 2024 16:44:04 +0800 Message-ID: <20240702084423.1717904-2-link@vivo.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240702084423.1717904-1-link@vivo.com> References: <20240702084423.1717904-1-link@vivo.com> X-ClientProxiedBy: SG2PR01CA0148.apcprd01.prod.exchangelabs.com (2603:1096:4:8f::28) To PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PUZPR06MB5676:EE_|SEYPR06MB5790:EE_ X-MS-Office365-Filtering-Correlation-Id: 84119b26-1c3e-469d-ac40-08dc9a736905 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|7416014|376014|366016|921020|38350700014; X-Microsoft-Antispam-Message-Info: YMOCAppgicIGMw9JCerPdCkLRRDdwAiX/jpU7zGN5fSPeyM2dr7gvAh2gb+rJQUTogBf8Qu4Lg58CjBNUiZqFjjhZ9wsRHzLo4a2GaEiZW/aW/THCANTGtHdUyuQPOoQ54TNsRLXci3Ysv0vJjZuNY4nz+wNlBIwo9JXBYBoJ71xORkQwRzdZ/7y4Z0BhkbCcFTHoprPE0bjnBnjKGTZcVntGQSAwemHESM6zQxQm1F8QS/AdHbObv6J4oujjaHQC81XiURMmR6O3tw0/yszKQ2aTFUTKUMWTLXX9pC5RX3hAG+zyJQBWjIUqkhu9AFmubFIRPPArBuiz7b24Z1xiXzVX9/K5iTCV9Tq6dSRv55chyoW8F+3WRRVheZaMY5hE38enCer91bEipc1jYHDwpFTZKYvzNF0V0YUisOUI54XiTkKYo/q7wOmzl6bvNHC61bBIyO3PAEssYiRZkHvaKoSVDaW3taoLoqlf7FexSV5jWFE8q9lBhjJvNlyrXccPs4eXxZouAzgVG6wiVHEJK+EKsEleVmXBsWH9AzytpQScUI1qkaDzGXZ6mtz5aVQgzyN2P188R0bsyv1pjSQvBkdS8xQnfqddxkCQd/TsNtBwW43zbytHnTWFwC6xH2CPCs8fQSJeuWZsAuf2G/NVtU1sgyCEk9Cjcekm+TLWDcq8SAth/D3SBP3YKoeOE13nO1zwZho66vUzbBGc/VfUGABv0RkQix06lckJ1Whc6cfGCchZBRDBrRhh6q3qn60r4Z34EvdXvYHUnokPiXC/WLKq1O/mbT2YbYVa0iolF4CyNxLaQGEa69Jb47EFJfwlw7q/ffzOQMwkEONygG5JBPuRPsV/1R2vTRWsLI/s4hyOH/l6VTE/zqUwmQUe1W6yR8X2xVW8EsnvQBLHS/G9CU7z321yWW/lZl4JBxKCNDdKejdDWNAK8YfHAmalIe48HAY5CYaiSI2W17Vk8bu4sU3o0IBKkxidMlWZTAo5uuQNKkc5o4pOXdWM4MquGDLNaxep8xLCBvRajDZTNTBGZwH1fj7PBlDUwwiAXPdZCiNlSuP0UBZwEPMs/wkdSxWUK/DAw7bQ4URDjJqlpVyrY0uXUj/05MopO0w5XN8vfIrMZdEcvHOyUZPU1uU9451UmcuoYeCxvBgILQ5k2glpu3j1IWr/xsnbzamrKDtrGWMscbOk0GlmXIHH3jgu6GmtC1GLpnxjjsxiCxw4YQas1HhazlbB8LtU3M3lJqz9Ius59+x0D0ZeQ1UQ9BSyAyO4nRI4ZRxbzqMjzI5H7HeuCVY5u67xCbGl1tUvO+JBtb2f+2mou5lJO0qi6V5ezQ4ppZPBbJPd66NjRPlSaLYvgNs3hrBYNLLB4GKqHqjV/dPSPzHb56H4w2EwUsX2k6vnpM95HdMGQzvQ6B4OwDCgE+R/gf2p7b66qznO7v0dyI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PUZPR06MB5676.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(7416014)(376014)(366016)(921020)(38350700014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rMuw7FBY94xkJq2Gp8p50HP7cnuz+T1pVbvGAQeSpi/dSPjeGNCWmnUa9kp4YAoBdGOCXnqcQECybIG/4nLJmki/AgeEny7/DmHre4+0E/OUYNxcxG0sbQkFRDmN4xHmWHICms4rdYQX4opsULKFiCAAy8HntHF6ERbxCbvaAHZeFKQO9GVWQxToq7pXfp37+KvcTS5PJTrbblJ79Qopi+LTf4Hf8bfjKRG2ugNIOVXK1E8u3YIY53iXdXQcFeW97grJRqXemV3b1F4sVEBY5arw+iywX6zLNXEghOcp1mqkgNGWzMBUaG2ARKsxUw3f9/NBvqkpZq2nhOmyIpApK3jcBB56Tmr/+avQY/BoQ5kZY+aZQ5MaOfNLAnFoQ1gegXZcQ1RhHVGRMRQuT2CMPRMxCyJIurIhPSeJUNKuOxZqzDc8E5kSPW47poBzOf086PLkJvvfRQ5hIhMpzcX7VlF+Tv0R16bgGHzCLvq0/gHRLvOb0rK94CA2ummYLW4QQr11fqEB42ZmoIS/vT7nLfZCBKP6bNb6cLcTQuG1IEmtZcNx462srUyqZq5pEuOtbCaDoJuh7+3DzkaOyyIZS3Kb235fTJ6ALHnvFf3u72bnjKoO/8hrq50xMyCuOSiqE8WDqgWEpv2gz9sSMqXuzqel5GG2TCRALWIDGDqlyE70wr/W05V2uG37F7w/xMbI9HxzmQMktU/aazSn0kCnUyRgV6m/p5LbBuADxhUs2MC+T8Ow8XfMYHxwImky1JElXXOHoIDrmyvxn4pw8Q1eYg5orXFTm6avKxb4KX0mxzf34g2zBevOI3hO2AnTXH2TcwPLV9q6ZhkffIsKc88Ny9r4OZ3sNWQ+wIHNGxoLLLkKrxXEmSgwDTqxlgcQes64yMyZpuBRKNy+WICLJpvvUqKpSe3Xx8rAEFViGPzfZSoZ62LwXkPJTxZkCRzvhRrCjkCBcuFP09gl90WwJiRAW2nMMdYIHxpxdSJ4s/ecS3oDsVhJfTHf1IHZLU2/gMaXJv9ni9lVURo1kCjin6lUjZjBkuITyR7LYWK7A0+r8/aMBJNQ/aaulWKBbT3lgjJjud+NBdidOPK1ryjXjmqrofvihhgItMZCpHPjh7RCwS3917pE8lLMWLwttgZaiar9L5jGGErl6Rzdm1u3QuH/iRI9+ZO4bJP/h3Ti9tFNBADvSTenyvBD2mMaNoQzE65vU4osPACP7jWOa0hg+2ifDFg03wJ36PxIXn9mF3pepXKLytCYf/snEvh9HNJD55Et58dveKPSzSO8dHb7qRjgDbtdYh+fdbxPuJvHwoCwHSiB516nTriR+S8mx9Iwsn8bzrYTkUAl72FzgbE44VYBu7fKpYhgbpAQeheT5+Maym+lYBwLJRtmvvjgL/sGvfxdsNlEgR4MQ3yRnlma5X4uARkB1hAZFKsiKzN4KtVjthzYqgYyvZ90VmGaFSXTN8j6ucnayRJmMtLhvoIpdKKrAnc2Un516jIy3faEKkBpFeGXk0p2I1mfeZcbtNkxmwh8zWcVcSJi3KNAD9rw008N6xUVBhuig6T8YTvLhkmIdcpZte0SlEfdOZOD/T/a+cE2 X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 84119b26-1c3e-469d-ac40-08dc9a736905 X-MS-Exchange-CrossTenant-AuthSource: PUZPR06MB5676.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2024 08:46:05.5322 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: E1kgVzoS6lowAWmEcHfN2rn34YWoi5y3uowseeJVne2cYk0TPp806UqtaQ7TgD+uM1NR3Kp7oeBa6MM+OXaptg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYPR06MB5790 X-Stat-Signature: jhm3zf3guxrop4zrczjdngt9heod9k5a X-Rspam-User: X-Rspamd-Queue-Id: 4405B10000A X-Rspamd-Server: rspam02 X-HE-Tag: 1719910073-512725 X-HE-Meta: U2FsdGVkX1/kNaAeMhy7HKV8WfowEre+lnr7dbtstDPbD1iPocNd6rXfW6yErrsjIAzQALG92+rHJt4l43zfaxHHDkJmeMMEW4H7SMsDjU9p63iIk8e3VN/y/et4r+7AEcruH0HJmYnvMeenl6MTCNP5TOOHSxkLar54EnkPjyER6RKatQPUfrlwZ/HfkOU6v8LTwg2l135h9F/aT4mXJlZw4XtU4pUpPwequU1KFcT7zwqUKLmZ7PP26pFD/N72uAO3yCxn+HNf8ELMFggd+PKVV7MGvwOjneyFldyO6oJE9fjiZ00h1vTfTN+eOZ41OB9Li8kxrCp/EMkdNdMsBW+mMqDyIsEb3XxN9VO1aTA9CoFx1YboJTduBDuduS1JLVD/H+hfb0V4oQe3v+SUk0X0TUa+rKkIzEHBnrWpFHGrDy9jGnt28l59es5opsMfd1ykgDsBfxP74V4hDckXIlwyiYvYRjDcGuttUdmdn49NN4eVIT7/Q+JawpXGg3GzFqb23h/XsnvgiE+6Jj4UP13UltSv+QosoOVk+2a7SeSoxRszD13OLFB08ocLgByoTkBvVU1vCQN/gpqvfJ77atXgVeokn/jJvoyIaai7JN7GEgWg5fS6EMFdSMKL2vKu/e943LMynq9KmrjKBed/0ry7ToaB/gHH8H1SaXvOcLw8itPUA72R6Efk8yI8Lin2LLCyGhIpul1u8/X+Ikrrq9QRa5L09DtKAZAp3aQKpGP0lQaeGWmUsp+nBuFEXqQ0ox02qAN+W5ApCorvZW90ZXM5llS1NVIjcm9L8HQqfTjWGxqSuHVzX+J3iuZV8pcGiNB1md5bJHjS3x1Gpx/LbKnAkmoJ20EHmBO8Lpnn+1PqE1YBexbezohWkIdy4/KbQVbpWH4kumEIBHIaFDPMEwyP7Hbs8wEF9wsh2HbTqB4urKtu9XvxijYywgxcChxy2fARfH/8ZZxYA+ipd5q XnosuuFR vDnmOpAD94O19XjM7elfpxxXzo/ajc810F4arv56nQmq8bHT+X2xEMcPM4K4SaRDaM9ZZNxMok7iKEIhk2+k1/WHOoa6pwGleaRi5+aWDwOOjW0zRldBLNABIignV6Z5ezz4ySyTzL1OaewwH2zhhWTntCqXqHsRtXHah419NJK1Ev76tVdpK4YsL2l/YZ7t25GhklIZF8VZ4JU6Wuk2SOXo3BrGkrTjKwNKAjotD4O9BxsZleweC1E1y7MljKsceuvxxgiLLJdX1fMdzyOwOIYNS5XVNxOsa3lGD8dBglBQsy5LzmUN3megcE+OHZgIKeybg0+W8BkR6PP6HaUs25WzIPpYeaa6v1DjFNBAv94H8zK0hERICTYbzvA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: pmc - per memcg cache This patch add a feature pmc in each memcg unless root memcg. User can enable pmc in a target memcg, so all task in this memcg will share a cache pool, the alloc/free order 0 page will high priority turn in this cache pool. Signed-off-by: Huan Yang --- include/linux/memcontrol.h | 41 +++++++ include/linux/mmzone.h | 25 ++++ include/linux/swap.h | 1 + mm/memcontrol.c | 237 +++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 146 +++++++++++++++++++++++ 5 files changed, 450 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8f332b4ae84c..5ec4c64bc515 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -130,6 +130,7 @@ struct mem_cgroup_per_node { bool on_tree; struct mem_cgroup *memcg; /* Back pointer, we cannot */ /* use container_of */ + struct mem_cgroup_per_node_cache *cachep; }; struct mem_cgroup_threshold { @@ -336,6 +337,8 @@ struct mem_cgroup { struct lru_gen_mm_list mm_list; #endif + bool cache_enabled; + struct mem_cgroup_per_node *nodeinfo[]; }; @@ -557,6 +560,8 @@ static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgroup *ob return memcg; } +extern struct static_key_true pmc_key; + #ifdef CONFIG_MEMCG_KMEM /* * folio_memcg_kmem - Check if the folio has the memcg_kmem flag set. @@ -1185,6 +1190,25 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, gfp_t gfp_mask, unsigned long *total_scanned); +static inline bool pmc_disabled(void) +{ + return static_branch_likely(&pmc_key); +} + +static inline bool mem_cgroup_cache_disabled(struct mem_cgroup *memcg) +{ + return !READ_ONCE(memcg->cache_enabled); +} + + +static inline struct mem_cgroup_per_node_cache * +mem_cgroup_get_node_cachep(struct mem_cgroup *memcg, int nid) +{ + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + + return nodeinfo->cachep; +} + #else /* CONFIG_MEMCG */ #define MEM_CGROUP_ID_SHIFT 0 @@ -1648,6 +1672,23 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, { return 0; } + +static inline bool pmc_disabled(void) +{ + return true; +} + +static inline bool mem_cgroup_cache_disabled(struct mem_cgroup *memcg) +{ + return true; +} + + +static inline struct mem_cgroup_per_node_cache * +mem_cgroup_get_node_cachep(struct mem_cgroup *memcg, int nid) +{ + return NULL; +} #endif /* CONFIG_MEMCG */ /* diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c11b7cde81ef..773b89e214c9 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -603,6 +603,31 @@ static inline void lru_gen_soft_reclaim(struct mem_cgroup *memcg, int nid) #endif /* CONFIG_LRU_GEN */ +struct mem_cgroup_zone_cache { + /* cache pages, current only hold order 0 */ + struct list_head pages; + spinlock_t pages_lock; + atomic_t nr_pages; + atomic_t nr_alloced; +}; + +struct mem_cgroup_per_node_cache { + /* per zone cache */ + struct mem_cgroup_zone_cache zone_cachep[MAX_NR_ZONES]; + struct mem_cgroup *memcg; + + /* max number to hold page, unit page, default 100MB */ +#define DEFAULT_PMC_HOLD_LIMIX ((100 << 20) >> PAGE_SHIFT) + unsigned int hold_limit; + +#define DEFAULT_PMC_GAP_WATERMARK ((50 << 20) >> PAGE_SHIFT) + /* + * Only when zone free pages above high+allow watermark, can hold cache, + * unit page, default 50MB + */ + unsigned int allow_watermark; +}; + struct lruvec { struct list_head lists[NR_LRU_LISTS]; /* per lruvec lru_lock for memcg */ diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..d7b5e0a8317c 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -420,6 +420,7 @@ extern unsigned long mem_cgroup_shrink_node(struct mem_cgroup *mem, extern unsigned long shrink_all_memory(unsigned long nr_pages); extern int vm_swappiness; long remove_mapping(struct address_space *mapping, struct folio *folio); +extern int mem_cgroup_release_cache(struct mem_cgroup_per_node_cache *fc); #ifdef CONFIG_NUMA extern int node_reclaim_mode; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1b3c3394a2ba..404fcb96bf68 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -95,6 +95,15 @@ static bool cgroup_memory_nokmem __ro_after_init; /* BPF memory accounting disabled? */ static bool cgroup_memory_nobpf __ro_after_init; +/* + * How many memcg enabled cache? If none, static branch will enable + * so none task free/alloc will into PMC path. + * Else, hold/free cache in target memcg, disable static branch. + */ +static atomic_t pmc_nr_enabled; +DEFINE_STATIC_KEY_TRUE(pmc_key); + + #ifdef CONFIG_CGROUP_WRITEBACK static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); #endif @@ -5738,6 +5747,8 @@ static void mem_cgroup_css_released(struct cgroup_subsys_state *css) lru_gen_release_memcg(memcg); } +static int __disable_mem_cgroup_cache(struct mem_cgroup *memcg); + static void mem_cgroup_css_free(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -5762,6 +5773,8 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) cancel_work_sync(&memcg->high_work); mem_cgroup_remove_from_trees(memcg); free_shrinker_info(memcg); + if (READ_ONCE(memcg->cache_enabled)) + __disable_mem_cgroup_cache(memcg); mem_cgroup_free(memcg); } @@ -7088,6 +7101,223 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, return nbytes; } +static int __enable_mem_cgroup_cache(struct mem_cgroup *memcg) +{ + int nid, idx; + + if (!mem_cgroup_cache_disabled(memcg)) + return -EINVAL; + + for_each_node(nid) { + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + struct mem_cgroup_per_node_cache *p = kvzalloc_node( + sizeof(struct mem_cgroup_per_node_cache), + GFP_KERNEL, nid); + + if (unlikely(!p)) + goto fail; + + nodeinfo->cachep = p; + } + + for_each_node(nid) { + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + pg_data_t *pgdat = NODE_DATA(nid); + struct mem_cgroup_per_node_cache *p = nodeinfo->cachep; + + for (idx = 0; idx < MAX_NR_ZONES; idx++) { + struct zone *z = &pgdat->node_zones[idx]; + struct mem_cgroup_zone_cache *zc; + + if (!populated_zone(z)) + continue; + + zc = &p->zone_cachep[idx]; + + INIT_LIST_HEAD(&zc->pages); + spin_lock_init(&zc->pages_lock); + } + + p->memcg = memcg; + p->hold_limit = DEFAULT_PMC_HOLD_LIMIX; + p->allow_watermark = DEFAULT_PMC_GAP_WATERMARK; + + atomic_inc(&pmc_nr_enabled); + } + + if (static_branch_likely(&pmc_key)) + static_branch_disable(&pmc_key); + + //online + smp_wmb(); + WRITE_ONCE(memcg->cache_enabled, true); + atomic_inc(&pmc_nr_enabled); + + return 0; + +fail: + for_each_node(nid) { + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + + if (nodeinfo->cachep) { + kvfree(nodeinfo->cachep); + nodeinfo->cachep = NULL; + } + } + + return -ENOMEM; +} + +static int __disable_mem_cgroup_cache(struct mem_cgroup *memcg) +{ + int nid; + + if (unlikely(mem_cgroup_cache_disabled(memcg))) + return -EINVAL; + + //offline + WRITE_ONCE(memcg->cache_enabled, false); + + for_each_node(nid) { + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + struct mem_cgroup_per_node_cache *p; + + p = nodeinfo->cachep; + + mem_cgroup_release_cache(p); + + kfree(p); + } + + if (atomic_dec_and_test(&pmc_nr_enabled)) + static_branch_enable(&pmc_key); + + return 0; +} + +static int mem_cgroup_cache_show(struct seq_file *m, void *v) +{ + struct mem_cgroup *memcg; + int nid; + + if (static_branch_likely(&pmc_key)) + return -EINVAL; + + memcg = mem_cgroup_from_seq(m); + if (!READ_ONCE(memcg->cache_enabled)) + return -EINVAL; + + seq_printf(m, "%4s %16s %16s\n", "NODE", "WATERMARK", "HOLD_LIMIT"); + for_each_online_node(nid) { + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + struct mem_cgroup_per_node_cache *p; + + p = nodeinfo->cachep; + if (!p) + continue; + + seq_printf(m, "%4d %14uKB %14uKB\n", nid, + (READ_ONCE(p->allow_watermark) << (PAGE_SHIFT - 10)), + (READ_ONCE(p->hold_limit) << (PAGE_SHIFT - 10))); + } + + seq_puts(m, "===========\n"); + seq_printf(m, "%4s %16s %16s %16s\n", "NODE", "ZONE", "CACHE", "HIT"); + + for_each_online_node(nid) { + struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; + struct mem_cgroup_per_node_cache *p; + pg_data_t *pgdat = NODE_DATA(nid); + int idx; + + p = nodeinfo->cachep; + if (!p) + continue; + + for (idx = 0; idx < MAX_NR_ZONES; idx++) { + struct mem_cgroup_zone_cache *zc; + struct zone *z = &pgdat->node_zones[idx]; + + if (!populated_zone(z)) + continue; + + zc = &p->zone_cachep[idx]; + seq_printf(m, "%4d %16s %14dKB %14dKB\n", nid, z->name, + (atomic_read(&zc->nr_pages) + << (PAGE_SHIFT - 10)), + (atomic_read(&zc->nr_alloced) + << (PAGE_SHIFT - 10))); + } + } + + return 0; +} + +enum { + OPT_CTRL_ENABLE, + OPT_CTRL_ERR, + OPR_CTRL_NR = OPT_CTRL_ERR, +}; + +static const match_table_t ctrl_tokens = { + { OPT_CTRL_ENABLE, "enable=%s" }, + { OPT_CTRL_ERR, NULL } }; + +/** + * This function can control target memcg's cache. include enable\keys set. + * To enable\disable this cache, by `echo enable=[y|n] > memory.cace` + * in target memcg. + */ +static ssize_t mem_cgroup_cache_control(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + bool enable; + bool opt_enable_set = false; + int err = 0; + char *sub; + struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + + buf = strstrip(buf); + if (!strlen(buf)) + return -EINVAL; + + while ((sub = strsep(&buf, " ")) != NULL) { + int token; + substring_t args[MAX_OPT_ARGS]; + char tbuf[256]; + + sub = strstrip(sub); + + token = match_token(sub, ctrl_tokens, args); + switch (token) { + case OPT_CTRL_ENABLE: + if (match_strlcpy(tbuf, &args[0], sizeof(tbuf)) >= + sizeof(tbuf)) + return -EINVAL; + + err = kstrtobool(tbuf, &enable); + if (err) + return -EINVAL; + opt_enable_set = true; + break; + case OPT_CTRL_ERR: + default: + return -EINVAL; + } + } + + if (opt_enable_set) { + if (enable) { + __enable_mem_cgroup_cache(memcg); + } else { + __disable_mem_cgroup_cache(memcg); + return nbytes; + } + } + + return err ? err : nbytes; +} + static struct cftype memory_files[] = { { .name = "current", @@ -7156,6 +7386,13 @@ static struct cftype memory_files[] = { .flags = CFTYPE_NS_DELEGATABLE, .write = memory_reclaim, }, + /* free cache field */ + { + .name = "cache", + .flags = CFTYPE_NOT_ON_ROOT, + .write = mem_cgroup_cache_control, + .seq_show = mem_cgroup_cache_show, + }, { } /* terminate */ }; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1beb56f75319..54c4d00c2506 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -530,6 +530,14 @@ static inline int pindex_to_order(unsigned int pindex) return order; } +/** + * Per memcg cache currently only allow order 0. + */ +static inline bool pmc_allow_order(unsigned int order) +{ + return !order; +} + static inline bool pcp_allowed_order(unsigned int order) { if (order <= PAGE_ALLOC_COSTLY_ORDER) @@ -1271,6 +1279,43 @@ void __free_pages_core(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_TO_TAIL); } +int mem_cgroup_release_cache(struct mem_cgroup_per_node_cache *nodep) +{ + LIST_HEAD(temp_list); + int zid, num = 0; + + for (zid = 0; zid < MAX_NR_ZONES; ++zid) { + struct mem_cgroup_zone_cache *zc = &nodep->zone_cachep[zid]; + int i = 0; + + if (!atomic_read(&zc->nr_pages)) + continue; + + spin_lock(&zc->pages_lock); + list_splice_init(&zc->pages, &temp_list); + spin_unlock(&zc->pages_lock); + + while (!list_empty(&temp_list)) { + struct page *page = + list_first_entry(&temp_list, struct page, lru); + struct zone *zone = page_zone(page); + unsigned long pfn = page_to_pfn(page); + + list_del(&page->lru); + + + // is good to put into pcp? + free_one_page(zone, page, pfn, 0, FPI_NONE); + ++i; + } + + num += i; + atomic_sub(i, &zc->nr_pages); + } + + return num; +} + /* * Check that the whole (or subset of) a pageblock given by the interval of * [start_pfn, end_pfn) is valid and within the same zone, before scanning it @@ -2603,6 +2648,41 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, } } +static bool free_unref_page_to_pmc(struct page *page, struct zone *zone, + int order) +{ + struct mem_cgroup *memcg; + struct mem_cgroup_per_node_cache *cachp; + struct mem_cgroup_zone_cache *zc; + unsigned long flags; + bool ret = false; + + if (pmc_disabled()) + return false; + + memcg = get_mem_cgroup_from_current(); + if (!memcg || mem_cgroup_is_root(memcg) || + mem_cgroup_cache_disabled(memcg)) + goto out; + + cachp = mem_cgroup_get_node_cachep(memcg, page_to_nid(page)); + zc = &cachp->zone_cachep[page_zonenum(page)]; + + if (high_wmark_pages(zone) + READ_ONCE(cachp->allow_watermark) >= + zone_page_state(zone, NR_FREE_PAGES)) + goto out; + + spin_lock_irqsave(&zc->pages_lock, flags); + list_add(&page->lru, &zc->pages); + spin_unlock_irqrestore(&zc->pages_lock, flags); + atomic_inc(&zc->nr_pages); + + ret = true; +out: + mem_cgroup_put(memcg); + return ret; +} + /* * Free a pcp page */ @@ -2634,6 +2714,17 @@ void free_unref_page(struct page *page, unsigned int order) } zone = page_zone(page); + + /** + * This function can cache release page before free into pcp if current + * memcg enabled cache feature. + * Compared to PCP, PMC is unique, only processes in PMC can access it. + * So, if the conditions are met, it should be prioritized to be + * released to PMC before being released to the public CPU cache. + */ + if (pmc_allow_order(order) && free_unref_page_to_pmc(page, zone, order)) + return; + pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { @@ -3012,6 +3103,49 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, return page; } +static struct page *rmqueue_mem_cgroup_cache(struct zone *preferred_zone, + struct zone *zone, + unsigned int order, + int migratetype) +{ + struct mem_cgroup *memcg; + struct mem_cgroup_per_node_cache *cachp; + struct mem_cgroup_zone_cache *zc; + unsigned long flags; + int nid = zone->zone_pgdat->node_id; + struct page *page = NULL; + + if (pmc_disabled()) + return NULL; + + memcg = get_mem_cgroup_from_current(); + if (!memcg || mem_cgroup_is_root(memcg) || + mem_cgroup_cache_disabled(memcg)) + goto out; + + cachp = mem_cgroup_get_node_cachep(memcg, nid); + + zc = &cachp->zone_cachep[zone_idx(zone)]; + if (!atomic_read(&zc->nr_pages)) + goto out; + + spin_lock_irqsave(&zc->pages_lock, flags); + if (list_empty(&zc->pages)) { + spin_unlock_irqrestore(&zc->pages_lock, flags); + goto out; + } + page = list_first_entry(&zc->pages, struct page, lru); + list_del(&page->lru); + spin_unlock_irqrestore(&zc->pages_lock, flags); + + atomic_dec(&zc->nr_pages); + atomic_inc(&zc->nr_alloced); + +out: + mem_cgroup_put(memcg); + return page; +} + /* * Allocate a page from the given zone. * Use pcplists for THP or "cheap" high-order allocations. @@ -3038,6 +3172,18 @@ struct page *rmqueue(struct zone *preferred_zone, */ WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); + /* + * Before disturb public pcp or buddy, current may in a memcg + * which already enabled cache feature. + * If that's true, first get page from private pool can boost alloc. + */ + if (pmc_allow_order(order)) { + page = rmqueue_mem_cgroup_cache(preferred_zone, zone, order, + migratetype); + if (page) + goto out; + } + if (likely(pcp_allowed_order(order))) { page = rmqueue_pcplist(preferred_zone, zone, order, migratetype, alloc_flags); From patchwork Tue Jul 2 08:44:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huan Yang X-Patchwork-Id: 13719108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73ADEC30658 for ; Tue, 2 Jul 2024 08:46:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09FA36B0089; Tue, 2 Jul 2024 04:46:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 027DF6B008A; Tue, 2 Jul 2024 04:46:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6EE06B008C; Tue, 2 Jul 2024 04:46:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B55B06B0089 for ; Tue, 2 Jul 2024 04:46:25 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 247DD1A1E37 for ; Tue, 2 Jul 2024 08:46:25 +0000 (UTC) X-FDA: 82294181130.30.A2B369F Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2061.outbound.protection.outlook.com [40.107.117.61]) by imf30.hostedemail.com (Postfix) with ESMTP id 4016E80004 for ; Tue, 2 Jul 2024 08:46:21 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=qTZi65VU; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of link@vivo.com designates 40.107.117.61 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719909960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2MV7Q2+/9roovDyuynNKu6Tlm1HAL+DluQZMl+IyKQU=; b=n4/ztMddBMNqdnlt8o/XwaHNkGSxt0X9lAlnAqaan+EQCV4QStYgVE1eDuWpCUefitvbfp 1+3ccqCnH0wFMCJQWGrWgPccZAhUr9eRrVpNiflo6Q4aWMZOdCTFJe7vUvS7SGZXan6AHU qF54PJdvz2Gtxm0lTzSaBpMuLHpwj1k= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1719909960; a=rsa-sha256; cv=pass; b=H3IrD5ZRsbAH7YEOavQ/Y60EYFqCuPBaswleD/KKepKpEG9DEwyy4Ph/U12bRFE8RfqSrq oPoe0jr/6dUGCMLNP/4syJRFQTJ2zKcgNQ9FqpSy74oEQkQWEtMyEh3OTwGMw0ZSJ9s9x4 2RUExY2hja6S/UO5B7v0Z+HuAy6vi+8= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=qTZi65VU; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of link@vivo.com designates 40.107.117.61 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iRFhxV3/yKnBSTgIU5u9jy0ZCM28mdMIAlHoeieG/eFg6NLSUQXT3pZzpQAlHY7U2ccj0Z2X7boPS3Jf42v9+vA6e2Uw/ybaLAdI04QwJFRJBNmru4aKZij6wQ2u9EqjI0Jj49DokNl/G0jd6np/g299T/QrMBwDpPSXBHMtYAaw4WRmrK4ATSkvyanvjhFhwEFb14bEaabKdh+mJW1cU8IGmdl3YhV/MDYgC8ML9ZIb0bULSoFmW32RAhvPrWs1l1xBmVRUIKIDgG4iyI1SuHECnal+L1AoIQWn7G7H45acgSF8qOa6TnJHuK+WQGuTwrMtHCXUQuh+vtuV3TgM+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2MV7Q2+/9roovDyuynNKu6Tlm1HAL+DluQZMl+IyKQU=; b=i0h9VaPXUyBTcOb9X4EXP6HaKFAtkPo2UXDaVSZEZBdd8nuTwAsntkM8zXVXL2t+4KJ227PUHFKfBnBaGPcihgJZZ28LzmHS2Qoyrw+mN07MOUL+NTn/mfQ7oQhfyfP2sZdqEKc7NUd6traLZmvaF3fvC2ngZlWoiiCOnk4vFg26fL1yx3vXKkyxeMQ+liECpID0iXARg77u/vZesCkJKjNc/h9NH25zp+2Bzajdy3faK/J2PpqrmxRRZ2OX7BCrEny8ejl6JNbgbzKIdZzNXbHtoF2OWjz5AeOSzEzZ8NyMvq84xfJxffm+K2JGQCXd5kwJKQ7wtIiYELYJV/7TOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2MV7Q2+/9roovDyuynNKu6Tlm1HAL+DluQZMl+IyKQU=; b=qTZi65VUKb02QN2AQ5YlH4npS6tSgw//8x+xsr3mqdz6FHeEqqgSyqKMhDT6BeAvSnvQFAqM6lgOy7ZY9NX8G7LwgOnoyxl2IhDIKFbonz2pkX0dFC3s56cSnlRsBxpTcNpkNDAttqyGrnkNt0lzJkdt/aSUQVeGWBepvH4SZ//Dla+jqJ9TeNxLJb70Yfxt4sGTqKTt0ED1mat+e3ylzwO1XDiakBCpRPXEd8AonMBV3u7ETUqkqzyY3n0lG1cFRfssAngV82QF/3cRbHwjJfxaG6cDpn2UXw2O+wsc4CDFy1RhG6xkp2+gVklN+OSk1CoQI3/ZBfqmehZGBoYV/Q== Received: from PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) by SEYPR06MB5790.apcprd06.prod.outlook.com (2603:1096:101:b9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.23; Tue, 2 Jul 2024 08:46:10 +0000 Received: from PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f]) by PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f%6]) with mapi id 15.20.7719.028; Tue, 2 Jul 2024 08:46:10 +0000 From: Huan Yang To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , Ryan Roberts , Chris Li , Dan Schatzberg , Huan Yang , Kairui Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christian Brauner Cc: opensource.kernel@vivo.com Subject: [RFC PATCH 2/4] mm: memcg: pmc support change attribute Date: Tue, 2 Jul 2024 16:44:05 +0800 Message-ID: <20240702084423.1717904-3-link@vivo.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240702084423.1717904-1-link@vivo.com> References: <20240702084423.1717904-1-link@vivo.com> X-ClientProxiedBy: SG2PR01CA0148.apcprd01.prod.exchangelabs.com (2603:1096:4:8f::28) To PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PUZPR06MB5676:EE_|SEYPR06MB5790:EE_ X-MS-Office365-Filtering-Correlation-Id: 2a2b0f9d-f637-4db7-7d21-08dc9a736c00 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|7416014|376014|366016|921020|38350700014; X-Microsoft-Antispam-Message-Info: 49AJizRIli6T8+XohSyCV4OOS/8am/tL8wB9DywapHJf/AkidjaKey7NaYYcczjOUUBTzfcH1oTB7PlJkowYz/9Ff4UAjmdztuz9BUMcfOCrrjF+CaV0YFuMmMjLp0qIsEb3/sCBqpETEfNpJOVqgR0/Grry21JRoEqaFrPG0EDZwnL4UGQMVpaVwXF3np9oBBP8DwXeqLzeGLC+87+CRxXiuZbq1RmKXhZDM0fyz7mcdZHCzGheHC1jsu8+0F0su5Lx+SP4SocdaqyVLm/ePC5B7YS15CAli1ND07eALBeayVWdhsKzvZWbyzdY5hs+fDh1OyXrszlQYbpyRfJFkAYLheQ9SA2SNOVQ1tZs+ISq/4882UBqJdZL0mWRTxdrVE6AnYbrkEasDU4qCCJa2vNdE8Ky9eOAzQhm0KZ5sufcMS/d8Pwc/E3MWdG0nyQNNVoWcbb87B6C1qfSZ4YrKsXq0e8CVlLCSHGin/3oQ1AirPNzc7z4Jq+M/qTh6EzzMo0/+EsBUWjJXHJF7uvKLHqGAbXMwBFlC7VS4BwWHjp2Ya3FHZTFzQzQxizuIiuftj/2T1mZGfEhSzdJpfFQRAlEaGDXk5kcq+IC7btwXdPqP51yILqkGmz/L+yXui35Flp0EEllvg+veXMrQRBgYf5lBYLdQyUSjXV96pgLq5JMCm1ZWsMZ0JApCByrjEuBOxYgDreUjNzeSdJPnMo4/zUDvnGNzBfp2MRk2dbEG1tFKoXlNXhjD9w73ddtY0QYFiyewp2MnvWO82QeJr3nHK5ww6uOgMYBMdPm64vK8WfTcl8DkQNePT0k5mikiHBwje3SCfB+xvajcDFSC3z4vhGRwPLt3doUoXVvebUG5H2A60Xtvfj2pX9T0CCdG+dH4j/NAkoltub8BAH9R/vpdjhMTqLjpCFTfHR2LeoTHhfsAAYvFfP7+kOozNcTJwQGOzRUKaTUMXz8hNzpUCeuCvG/iT/dErlo+ML+xkKdAPLgtKWqhbdHB6UwA30aQq73yT7edJKu0SB3SedH2qkKJFoyQ9+1lijS/5d7tym6JKRUiKuyf4yLBWL2SKOVOIOVgYa8jnVM5tmdOi9Lq9eQ6wPMjEjCXHJLtZfooOfzubvedj8Ld6mv8XIet1zDuW5+fNjQ31um0cMbQXNRTl+S0kKmxPfsgwtq+n0fB2vEh5ZEZx682XBlBoLb/pxpBr3BYb1QZtWaVe0GoqdFQQLh/gRfqXGQrzqZWLM1oZ00RrNBg1GuXKwPIX+4weoCD2qY4pOBo2ZcjH6veJFOKggC1xwH0RtGXFf1IXoSZMGyIZQE/jeUeOUySMQDmHm87k6x6hynoeOycc0qs0i62fcz7uKEbjVCYNXQivj5/Ouxcg0vQCcaGAMvuMdpHZpRveXPxEhy811/C9RTDrApHcqSb11IkzJR95RftcJ7iPQeSzU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PUZPR06MB5676.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(7416014)(376014)(366016)(921020)(38350700014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: x6kusNf0ShESwMV1tB9QubYUsfGsHTzqJOKZFnzJmUmaqMNTMklNeVr0WJ8BOB4oyJGmfCgs0c59lPtrYHQqPmjnEgLd/a/EhJiK+8e0NQC9mOEXlmN9lAhihpYw+NyTMlNjhvbSrqfTEwfSC3mNka4v83ZGumy4qL6kVnXJYhBnWmuSNiik4RKuXODYMQUxZB8W4oMu9rOb4J7mi4Rp8Pqj3krfDBT/tepTCdHEKNV58UM3CNz4dxbI6BqhUjhGimm3ZCRfq/WmzxNQFcMfEutXNo7U6OIkJVfH4mfTFRZbCbzdu4+/yvu2tYHLHUy8bDsTo8bJ/CqidfwKtGWPN0g4IDHop1sfiaGJJc0NCkkwS/M6+gWJU47D5R+PWdMzAyJgTIqFmqXUkyqaSHGiPOrWKhHFtABDFu6rFuGd7iv/NRDVCBnMcJSAr+396Mle//z0Bgu8hIxj7d5eZhDoXVhmbm9oDN0kA0S+hOQ+J5JGtqoxfd3VEgXT+6kdgkPDEKZLawF9mPw5wbhOd8FlCjHYeYW4wuDcEzRDamGANGW/4stFjaoZb2dk8oi5g7HblcfZNiaFqfRRlh3iKkebuE8ZptRJY3Uk7pAWdxLRPCITN7owK9kuCAWQLE0sNCTMy4j+kdp7aZ8rXSNmaDSQWpNZbJ9UACrukz1o7HhHmvwGZDr9xrcRukhjNGaQtkbDHP1NkGGJt379ymGznesXx8nRKTLR0Th1rjLGww9jpS6sUHvu4NS4pP+TSSIC1TVJcGhsb4HmNdcYpW+DjyoAYXw9oexlbAi4X3zJBrMmZWzAOLqgnE8ncXNK9xxy2w5sxZ7EEqSFXLEDvdcUUlo73fBnCzmhRgF3DZC/17J+EUafdFg1bfk+QJcjHtzJo+n1QSFnYhnL3FbJqxr7YzXpxoteiLRdtlVfV5MmkYkjUFppOleWzB/W8TR16a0lgy00NcNp0dYvM6ME0twpLtd9qcSS+EJxGyKJJSicsnC7q1ByD2Nv5ZMd/iYCjXxJrkm/Pk4q9Mgk7O9GtJYRTa/U0n5ehADKAjHLxDYgED8mJzd76xq67SC51c6cqXY6fb2T8+ylAIj/khtYPr23Ei2vQeAeuQYC79lv9sVoBHt89RbfXqsicEBZ+Lk9I15N/Z8iqzjrcUXdql/Ai/lBZ79GVC6zbPNry1o8/J7D765vlK3ZbvYGbw1nQIsbqxGGOiTfQ/mi2JW7Y5oJjcyFgS0XFu9vBztjrMHiOsEcg5iiYvVVJHmNALDg2IIl0aDd5/LhXHnI4HbgLSSr50iJRHK6TmVcX0uADEihtdxqYPpmH46sAf9ZmkJakYUQT28Uv7zXB8Uyoy1/nX+qlbkfOTsHgy4tsWIZiGbh1fi9pq1Z4WjkSYjk7/5U8JfFxt6g5Erc6Rz6M1WTomY+v5Xch1v6/kvmyDDIukptMO6CVwDWghjrYdvHG+I2GI3M5utcQWbPy9kajfO7FmJfWXa6UQgFBLGHyERYdekKgIHWsy2xbs9cmCRAomShpfiwZrGHHexPcumIPi4sZHKIC5CgkiO4RFWtzfQWfJ/1mDrQ59VPjfG3TU25Q0jyWxHE2uBLyEep X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2a2b0f9d-f637-4db7-7d21-08dc9a736c00 X-MS-Exchange-CrossTenant-AuthSource: PUZPR06MB5676.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2024 08:46:10.5289 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6KY6lu++Iu/ILojwIxMOEIrWhiVYpDb2UC6OKonul2hy/y4jSDyVr4WZSMr3g10NKUvkiaYmE33qbMpwZAkQJw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYPR06MB5790 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4016E80004 X-Stat-Signature: bkywyk33kbmqd5agd7ipp8py18i3mqgn X-HE-Tag: 1719909981-197138 X-HE-Meta: U2FsdGVkX18gSzcwjHhu5NbOQNsJJlo25YiDZQjA852PfmI9eIaaZJaRKectFZVt9T/HhaP1v7Lh1uDLd4oyp4zdBMi8OByICCZSJgGWOBEHQgSjHbTxEZazbKq9HO8S7dBDmO+ADGnGhjXClHSJmogYJsWpNPkWXMJ+fZHuTkg9kO3ocjJ+3srQHQf378Du4RWDLpcfcTeFjrCkl5I/fliQRCrqO6M7DipzncQSDVhUn0OygPb87JK1yabh5m3zZI3ujwHWW+5VjflPgHz9c27Z2yj1CQuG8wnBlsQm0VCaoukGOnnte9Uma3vvBuxjMvKozx4O/JQrn48YyXgfKyMBsVqHaazkhX39b+mFfVc2NAgwdoAdemDzB36pV6EnyNlK1mCaSMXQWy3M13EyLesJdNAnTIKAZnvkdi1q6t8CNJ6cvOgxDMpYxCGNMz6oloVEo+smKPoygoX6B92zNkm77tjuPIt8jb5Oz5+GXnifOID0RBdwseU4QlYQghML58IuDZc22BwDQezgGSfk2GRqO6Yc4zoej8UPpT4s2E+/sDOxtrI5ZdOLpT4SmUt/8gfbOTFlosnTW2EtivfyqMvP3TrixdJ6DFuO+HKqKjcUMvLOCBg5fHT2N+SCSXrPsVQTU28MqcqHP/Vsarpe/Nl8VfTJWEuWOEoRDvYArltLwN9k9kbXgXNFbB5xscjRR2LXgPjIDIEyYARtbk0kceYcJ6ejY0r3MxtyQcK3srVjDpSwaV0o3R0yhJ3VQZhvdPAcKb+fnqtei6A2rqxTM26ym7W0KLcLFF9HQRF7Qj6wwW0FtIOAEOxzaxx2Yxeo0TR/4TDsikWfxnUSQKfZsc4crF9tayWNvshLQA5/QQ/q6ukTXAceWzKe7VkzBr3G9wsAII9ZWdGaoiN0m3Q9DbpEvAZBh6MczY3LPvqNTfpeclilQiZVQ0w5YzFWCqaLugHhqSdiVgbEKIMtVdP 0hg4G1u/ nnsOzCnXEkhLsuotPBTylRBxdvig7O9d9nisReCcc62ZqOzEfmNEUWHfIyXVSraOP8TaenTsxGgURWY0XTaH83R/PO8RsLQNDiGgWtehWLOf3tX74ZnYVUoNWENLFXXmiKDh0svM+47me94UyxEOobuUoowirm5QvUA65YWW5kZs045i6sy0K2GWW2Oc8G05zotT3v6fqrcEFvlr+noRZqnJGpVQPYNdJ7HsY6HfTZJCR7jWg2QRs/i/49PCzZYKEur6/fcicWA1oo3I+PWFBdOl53Q7/WU7goQDNY87LQQEagS5TNAFZDFchEHB8226LR7VE3A3h+q2Yr4Qr0CxffItt6T2Navx5aPEhNfetnORODc4pn3exD4BF9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: pmc have below attribute: watermark: only when zone free pages above high+watermark can cache pages limit: max memory it can cached. This patch let user can change each attribute by `memory.cache`. To change attribute, can type `keys=attribute=vaule` into memcg's `memory.cache` if it enabled cache. For example: echo keys=watermark=157286400,limit=209715200 > memory.cache This changed memcg's only when free pages above high+150MB can cache pages, and can cache up to a maximum of 200MB . Signed-off-by: Huan Yang --- mm/memcontrol.c | 152 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 151 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 404fcb96bf68..9db5bbe63b34 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7253,29 +7253,168 @@ static int mem_cgroup_cache_show(struct seq_file *m, void *v) return 0; } +#define STATUS_UNSET_DEFAULT_VALUE -1 +enum { + OPT_KEY_NID, + OPT_KEY_WATERMARK, + OPT_KEY_HOLD_LIMIT, + OPT_KEY_ERR, + NR_PMC_KEY_OPTS = OPT_KEY_ERR +}; + +static const match_table_t fc_tokens = { + { OPT_KEY_NID, "nid=%d" }, + { OPT_KEY_WATERMARK, "watermark=%u" }, + { OPT_KEY_HOLD_LIMIT, "limit=%u" }, + { OPT_KEY_ERR, NULL} +}; + +static void +__apply_status_for_mem_cgroup_cache(struct mem_cgroup_per_node_cache *p, + unsigned int opts[]) +{ + int i; + + for (i = OPT_KEY_WATERMARK; i < NR_PMC_KEY_OPTS; ++i) { + switch (i) { + case OPT_KEY_WATERMARK: + if (opts[OPT_KEY_WATERMARK] != + STATUS_UNSET_DEFAULT_VALUE) + p->allow_watermark = opts[OPT_KEY_WATERMARK]; + break; + case OPT_KEY_HOLD_LIMIT: + if (opts[OPT_KEY_HOLD_LIMIT] != + STATUS_UNSET_DEFAULT_VALUE) + p->hold_limit = opts[OPT_KEY_HOLD_LIMIT]; + break; + default: + break; + } + } +} + +static __always_inline int +mem_cgroup_apply_cache_status(struct mem_cgroup *memcg, + unsigned int opts[]) +{ + struct mem_cgroup_per_node_cache *p; + unsigned int nid = opts[OPT_KEY_NID]; + + if (nid != STATUS_UNSET_DEFAULT_VALUE) { + p = memcg->nodeinfo[nid]->cachep; + if (unlikely(!p)) + return -EINVAL; + __apply_status_for_mem_cgroup_cache(p, opts); + return 0; + } + + for_each_node(nid) { + p = memcg->nodeinfo[nid]->cachep; + if (!p) + continue; + __apply_status_for_mem_cgroup_cache(p, opts); + } + + return 0; +} + +/** + * Support nid=x,watermark=bytes,limit=bytes args + */ +static int __mem_cgroup_cache_control_key(char *buf, + struct mem_cgroup *memcg) +{ + char *p; + unsigned int opts[NR_PMC_KEY_OPTS]; + + memset(opts, STATUS_UNSET_DEFAULT_VALUE, sizeof(opts)); + + if (!READ_ONCE(memcg->cache_enabled)) + return -EINVAL; + + if (!buf) + return -EINVAL; + + while ((p = strsep(&buf, ",")) != NULL) { + int token; + u32 v; + substring_t args[MAX_OPT_ARGS]; + + p = strstrip(p); + + if (!*p) + continue; + + token = match_token(p, fc_tokens, args); + switch (token) { + case OPT_KEY_NID: + if (match_uint(&args[0], &v) || v >= MAX_NUMNODES) + return -EINVAL; + opts[OPT_KEY_NID] = v; + break; + case OPT_KEY_WATERMARK: +#define MIN_WATERMARK_LIMIT ((10 << 20) >> PAGE_SHIFT) + if (match_uint(&args[0], &v)) + return -EINVAL; + v >>= PAGE_SHIFT; + if (v < MIN_WATERMARK_LIMIT) + return -EINVAL; + opts[OPT_KEY_WATERMARK] = v; + break; + case OPT_KEY_HOLD_LIMIT: + if (match_uint(&args[0], &v)) + return -EINVAL; + v >>= PAGE_SHIFT; +#define MAX_CACHE_LIMIT_NR ((500 << 20) >> PAGE_SHIFT) + if (v > MAX_CACHE_LIMIT_NR) + return -EINVAL; + opts[OPT_KEY_HOLD_LIMIT] = v; + break; + case OPT_KEY_ERR: + default: + break; + } + } + + if (mem_cgroup_apply_cache_status(memcg, opts)) + return -EINVAL; + + return 0; +} + enum { OPT_CTRL_ENABLE, + OPT_CTRL_KEYS, OPT_CTRL_ERR, OPR_CTRL_NR = OPT_CTRL_ERR, }; static const match_table_t ctrl_tokens = { { OPT_CTRL_ENABLE, "enable=%s" }, + { OPT_CTRL_KEYS, "keys=%s" }, { OPT_CTRL_ERR, NULL } }; /** * This function can control target memcg's cache. include enable\keys set. * To enable\disable this cache, by `echo enable=[y|n] > memory.cace` * in target memcg. + * To set keys, by `echo keys=[key=args;..] > memory.cache`, current support keys: + * 1. nid=x, if input, will only change target NODE's cache status. Else, all. + * 2. watermark=bytes, change cache hold behavior, only zone free pages above + * high watermark+watermark, can hold. + * 3. limit=bytes, change max pages can cache. Max can change to 500MB + * Enable and keys can both input, split by space, so can set args after enable, + * if cache not enable, can't set keys. */ static ssize_t mem_cgroup_cache_control(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) { bool enable; - bool opt_enable_set = false; + bool opt_enable_set = false, opt_key_set = false; int err = 0; char *sub; struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); + char keybuf[256]; buf = strstrip(buf); if (!strlen(buf)) @@ -7300,6 +7439,14 @@ static ssize_t mem_cgroup_cache_control(struct kernfs_open_file *of, char *buf, return -EINVAL; opt_enable_set = true; break; + case OPT_CTRL_KEYS: + if (match_strlcpy(tbuf, &args[0], sizeof(tbuf)) >= + sizeof(tbuf)) + return -EINVAL; + + memcpy(keybuf, tbuf, sizeof(keybuf)); + opt_key_set = true; + break; case OPT_CTRL_ERR: default: return -EINVAL; @@ -7315,6 +7462,9 @@ static ssize_t mem_cgroup_cache_control(struct kernfs_open_file *of, char *buf, } } + if (opt_key_set) + err = __mem_cgroup_cache_control_key(keybuf, memcg); + return err ? err : nbytes; } From patchwork Tue Jul 2 08:44:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huan Yang X-Patchwork-Id: 13719117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4766FC3064D for ; Tue, 2 Jul 2024 08:49:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D73776B0085; Tue, 2 Jul 2024 04:49:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D22DC6B008C; Tue, 2 Jul 2024 04:49:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B28986B0093; Tue, 2 Jul 2024 04:49:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8B4FC6B008C for ; Tue, 2 Jul 2024 04:49:14 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 1BF251A202D for ; Tue, 2 Jul 2024 08:49:14 +0000 (UTC) X-FDA: 82294188228.16.1D3BBCA Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2082.outbound.protection.outlook.com [40.107.117.82]) by imf21.hostedemail.com (Postfix) with ESMTP id 106381C001A for ; Tue, 2 Jul 2024 08:49:09 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=EXvsFRi1; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf21.hostedemail.com: domain of link@vivo.com designates 40.107.117.82 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719910128; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JHncJhtTb2CVCgR6KyprnXFeC/NMYgOPEuy1IeHa92A=; b=rhGGNjNKOS9iR5QrlhsdDEFRdrjnkvbCEVBsn4my4/BrCmO4T7/nVivd8yqjr/UxutHl8w li4/zkN6sY+ywyQMDKE37sw0HL7O0vaW0QlrBxO9a3q76Qic9m6PtmzVKWwTcMDcuNE0qW w4lTCOTrxsEXLB8mKBfVREfslZV2cZ0= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1719910128; a=rsa-sha256; cv=pass; b=VFgrzKP2P+w64N2WlARr4HpZHP4Qr3TD9T4dductZVTHXy1yy8mYxlGou4VmdBmz0l38u7 qwPz3e0eiCgb8OQKkFNAQUMBFJvBKdaNt5fA5ikPx3/o4SKylDsP2Vf4d/NBIpvi6Bfk2l PEluud8obH0qAvZCoYJXwUopVqNHIv0= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=EXvsFRi1; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf21.hostedemail.com: domain of link@vivo.com designates 40.107.117.82 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NG0QT6WaQyDMRNhuKpwKm4IjyN6ujj4mv9OjwCybe07stJvxBXnODj5XQo4R6+aGR1LFplJoRWF1eSM9nuazxdi0oO8QlzchB14CR7t2xPm8Mp3d5V0rZKwLRQRQ+bBvt/LgrMcNN+9nBYscp92Z14c43nA2ch78kyODkZaqEt97YZ8ELfc3yPESpAglE55NNWGg4L0XfFPH2fJJKJSeZ8UzX956EDiqglbXuS09osPj7Av5qsrAPPJhhHTJz5KQseffF2SRUvbCu1IUxwp6yKb7ejrZoex+9guTWCpCKr0Xg/SflpwsWufTnNppbV5w+YUYQ7jZ75vxIeNOglCbiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JHncJhtTb2CVCgR6KyprnXFeC/NMYgOPEuy1IeHa92A=; b=gWM7HRpHHNti1miCfZWpwf4Er+XP5/Z0vnPPDJqfzCyDiBeGajzkFZS+FanDXCUTBSmZR5WacOMC8LVwcX+2AYBVlTbTC2kiIvEK9l3hT1xIGQo4nb78M92FLRVBcJRsvFtMCRaZajwQYFUVI49Dcb6IjMxTQA0ItjGyqqh1C8nCxkRL/7eou/icHfrOYugVadL3udXI6kEacox4OFTDcRM7orZt6dXhvEf4nUBuMQqRBBlCpQxJEW5maMVDGvJxv9+X0gcmfd86dPSsRiSP+MNeoBKNxX3ZTPLEaj4LbtFJDi0EPKmbRrGP18f3IUGQ9/zV6+XcSbFp/Il6K6P8Ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JHncJhtTb2CVCgR6KyprnXFeC/NMYgOPEuy1IeHa92A=; b=EXvsFRi1MNmtpSI2shGOY8GWVHvzvVZNozT6or9XUJHiM5GIYo/fKOl8hRUt7nW9s/hJWbln3yw+OikN5mC7fp9Q0Ttbm//1MIIQJ8qzRB9V0omtK5CEVi/TumwHze603NHGVAqB34zT41OlcZ4qog+xAJkN3tI8AFhNDedKbXo/8WELMroCD+hP2emNMVfGbT4SNBCsIDJ8E7jE+eyR6Fqpuwx7kwgTRe23qn4iGoT8vs6Z/6oBcDm/PXWJJESM9Vrm24EBY3WwmG71+qafy7n/WzZVVZ0kln54An/ha8rR6aJMybtp7u8MxkgldIRTkeTsVfUvp68wuuaMtbHSMA== Received: from PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) by SEYPR06MB5790.apcprd06.prod.outlook.com (2603:1096:101:b9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.23; Tue, 2 Jul 2024 08:46:15 +0000 Received: from PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f]) by PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f%6]) with mapi id 15.20.7719.028; Tue, 2 Jul 2024 08:46:15 +0000 From: Huan Yang To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , Ryan Roberts , Chris Li , Dan Schatzberg , Huan Yang , Kairui Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christian Brauner Cc: opensource.kernel@vivo.com Subject: [RFC PATCH 3/4] mm: memcg: pmc: support reaper Date: Tue, 2 Jul 2024 16:44:06 +0800 Message-ID: <20240702084423.1717904-4-link@vivo.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240702084423.1717904-1-link@vivo.com> References: <20240702084423.1717904-1-link@vivo.com> X-ClientProxiedBy: SG2PR01CA0148.apcprd01.prod.exchangelabs.com (2603:1096:4:8f::28) To PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PUZPR06MB5676:EE_|SEYPR06MB5790:EE_ X-MS-Office365-Filtering-Correlation-Id: e9c9a2ec-00b6-4829-04f8-08dc9a736ef7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|7416014|376014|366016|921020|38350700014; X-Microsoft-Antispam-Message-Info: R+ALswNfFrf7uwYXx6CNHD6PVJXXWXMzyIzOMBrmf6WIpIEbzYtLvmBLFD9v+0eCUNthFpaiSC9Gigy4S+GSUyxKpyAiXRw28OV9k2w9LTGmwnk1CFz5Ld0+rUn3AZpMsR1/j79+pp9Ok87mEW206l2K0A0RuGyuoG3lqkKix9T/Fk7lEQXmHwJzMOF9kakAEpI3BAGFqCiaIf80+hlTIBdekNoQvxQNACAyyv8V18ypMD0hY93bTS/0+xhnS8kAm9vRSQT8x5VX+DZNGVLcwZBBfiqqOdjOyG4lOgIGSxXWV4ig7ivvBfbxU0dylMuRZLE7yE9kt/ejp08mcsu98zZBJTCWYfyl3akI2vfYGHtaYEYWl1SjiIoBeDwI9ZK3jt4vkMlXwxqQ9Vy/WFAoiuUqnVV0WosJwHgttx5o3UDJmXZW5QJCzsXhWcDc2MDvk4oYkaGK7CnaWWnfWuxU5foTowNGMiB4FBpLyKa2q6wYFVJMpPUcKuGSYByIb2WkIS2RKEBDM15V2IJdhn/KwS2a8AhUygXngxxlwdrFTUnXjRrkaY9DUTETw3c6OKw3SBsXPBqMytLefR/t/skwpNy1gWa2RX++wL5Xk+32XbRdgZupSE73qJJbcwGfLSFvRwMXMuBD20veYtypoQobT3sn/5SXcjdB/z6oOd45TNIaa5o2xXg9TkqlYrZ0v3dARE5WKAlWvX5dPaKRU3EMhtzhOkv1NBX0KXW6+q2xp0MvdWhanKrP9alMN/MmqkFqyXZ5Gw33V55Cz0dbZgYRrewqEjfyn8eKWqTxnM/NkmfngMcwnNvV51bmbt+SxF8Tb7aq+PWSpQSPXVTFFIyXuKm7LQUAvDe+Ve6aXmm88W4cdWSToovYtP7NtMq23yOvOiGpu7I+0unOMaoju6LfO869Z0FImcy61AYt/vnQpI/sQQW+lxGqu4QEsVLJIV0lil/cUwIj3FrkNq2bLaPQqB5HGTd+uhtIV2Weja5B96WX3DCck5J0+Ci2MIoCEtqh/OOh393KNHbqyOHxKRr3nXoz00P6lBu6cP+t+y4Etn1Ujd6F8Vty64kxqJlmSebZwQT87VXm205iwdpfqJZ4d/chUJP6sOlE15zsVbBGfEiYhEUkzXNkpLThqXPdfxLBqPoOrsxxDGh807r4LU9cmdPu02Xk8mJize1FWKHZ3ch3lYczMHyIR9Dsn4CoiMybHIpkxas2U8PICwR0+q+CGAJRpRAdyPLrlWVXU39nmPFa2CcJ7HiGrWn/ExEZUz9+v/qpLWJj2mEAZwbRytA8s8lA4vRn7oxBXKwtL1xBsmGAljwHbzQQogMHb3+vBBFQeeoGbiuezULJMeRQkeVlsTtteihdbpG7jq/OtSaL3J6RTaDd6fsntly+K54Igcz+ia5eJy9C4iBEdtxE7uOEUr1hkmAkyH9Uq2Ag3N1cgqo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PUZPR06MB5676.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(7416014)(376014)(366016)(921020)(38350700014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: r45KEwGVvi1egnDNkWTzmCZ0Dq3RfYHlV9fz3IMCmCE+tjCO1/SRF+yW+Yf+sU9C3Fd1wfn/vBpIDkVCzhkmIGkO49J+N6mgsjrye7e06+izDoLlaFFHVXGSWJ7eMG9Q+RPa1U7Pq9FAhf+mzu02y5L5VMMqPxu1llfbLijAPzQVnVRv1lys2shyonsFRvQ1NmX2OgWU6QYGoymV96VWZGjNX27xTyGUYCOTXnTYFBXVb3CR/FzV3QNn5DIhouVPs/q1sB9nWLUlK6CrFd3itd3lwnW95HJOAt5ANZOPIEb6qgnzCRVLzsvWhVj9n3OgK86HO7p/tOycorKw+0svgBOhJiyPqDpik/q48ZQ/H1cjguNRb/J0RtNYI9i1LGerUTPXjxXkEEGzXIkF8B2LYecxB4Yo0bcwfMrA7SKCceqVtOBWm6wjOA0YYs8lIowhPs78sVJcss8pM9TwbEEs4foE9HjKfFs0giJuVxezPIzlD9ZLJvmy2pgR7rxNwD4vp0BWLhAkJygx0bqxA7edaY+l0MFES3H1K9uRr5fT4fjHVKa7a4SVkDVUXxfTwvk0EWQgPJlG1DL9JtgBDjHkE+vmCAnJUkAstvYUpKD/xbMo7Zm1luGWdfgrOJt+kjP6s1Z21i5IAVpXnd6JyJMKsuRxYcwZwUyqjAqoPl8kFphfotD2NNa+3DU42kOU7a0muEjvUbTYN/jTittf5gQyZ5SN3P8KhyVnYvbnX5pjcQB5e/mZiXyRUhfylKFhprJvSH7PawEFbU88kbIv8XXpcPvEMiVEYM2vuxyn3y2LYnOwNY7m08gjqAUjR8ftDpp3Qg1hWQZOAfwbZ6ToLJArYQEKPr4VjIQ/8KM6LwiPhnYk4iEitCNDKE6qOB4yM3ZAVmszc9cZnsDM632zftROOtJRgYFv9Q5bvREee6MIZ01e7JUlSpMf4WjT3TIllWYMSxFCIncKT8fJAUbQhjwBlyEXrLW4bbMQlCSCXTrIr1S9bO09sqBCtHZUd4fTGsjKMYG65gqyt86BUjLqB9eVeSUlw44FK2hZcUxfdDoh0tR3Tpn9Gqd2EWXLcPR+zMf0KQ7n5AWynaWyZeKrb5ylqSzkG8tnvZMvKl6OjCl65y73BQ/YlOwpXYJFvzJXYS5Gi1LT8VIC94zybgxXB5NLx4TUiZ0K8VmZI6nmQtpwX7qnt/0YlXCCkXbgT+w9PZZqTiBjVG8QMUFn4HGhmWAfJTKYgGRab86PtrcohDzMshXkDAfhQlh2m21ebf9OBhSj9NnEqaAAk4UwvmWzXrJonR17O3zzb1ZUynJFiVJ0XfA2Qhx7hFLIwhoOVym5cgIpUFRx0RsGF9Q8DzIDrNUnNVrwXxM8fbO6ybL0PVW7gANQTIAO/M6eL0g/ZQWFnKRL8TQMT7fNYuG6Tq6T4fgeDFmDdvZSnSSTiQ+gvV5kO/kmZR9xiZNTWgGyj8hqGLEhFcxVZZZgu5vNVDOpKveYmpos60LFaMhtFMtj9zHCfaRJRLg+ZskpO1uIxBmkpWUB9jEqwFGlAiUggtRtThI0vlqBj3CQz9HWTaLUk2akv3MQkIuKEY7Vy5tdr4IOoCUS X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: e9c9a2ec-00b6-4829-04f8-08dc9a736ef7 X-MS-Exchange-CrossTenant-AuthSource: PUZPR06MB5676.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2024 08:46:15.4736 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IQlY2azfKDb1kg7oQIMdIERcJVFFW+6CsslydbbvgCpB/EyythP7qZTREJ9qZsJ2abYPA7x8j4htXKNs8EJ+2w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYPR06MB5790 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 106381C001A X-Stat-Signature: u73okzh5fmeaphugsjjcbejf3m88xbjd X-HE-Tag: 1719910149-130406 X-HE-Meta: U2FsdGVkX18zcySt1Nqs87jYF/nPmuPqN2VvD6rXi9jxTPFE7+f6a31ic6ULgdhtmV2p6VGu93wIH6sAc+2RFSouws1lofb17ud97BRK/PvcDdT2tz9o4HhqXdy3qhWkToTJAOsbtMIlWShOg/GSkQ20dlMCGZmYubCCvz3FKL8vGBL8OAyRZSlhgD2zjU1BelYi/e/BB2TtBqYjfnK2mY8kmUvMUrNJ3qjwFWB3PRHm1WGGCv9hpW1xAX9Nv5MmtFYqz84xpB4rDO6sTjJ60+BL0QxYR7slpPxlTTDOVyk4HXH7G2bNLSc022vZRPWIXj4FhL3az+U2mbR28BYqsKfm1NGYCgrS7EbEWdm/yVnCaawfR4+czSoDFhe3ldEbBuL5NsghAhIZ0QMPslGlCQ5OZ37aervb/5f6MPGBa665SMnvConolvnJtswS7yhZ1LntWS8r0nKps5HV8OPalYyVslLs3L9uroYZXjsvtBk8VIJ7vbAVY5gwsFMCw4n5TmRbToMs6vh0XuGToAxdh2Qzwmscsvvfp/p625i+3HQLwiMTu0oDxuIOe4G94ObkU8bJ4X0QraXtnXhN0nfKiQL80D3WLYfgJWqUnMXsHlp/6pV2+XGg1pQU4vs+pc50fJCTanqL7RiFJ+oEaJTZyJASIgZkYhDJNrVQ7adICq/7Wu62DWnF4FyX7JFcPoGpZ9rDjIvweUIQgZR91jDjNyAfQnP6TjQPO8yhh0A+uxl37lPskDAQCR3dNnwkARb1cYZUA74JMa6la1BoyFWgh3OHBhAqvKD8X1AVUbovf5HyXSVmDWluLLYNtt0qE6JADZjoZWR1KqkS+N73j+GQe39Qxy7+xNdWamE1gSqgglzYRSNGi7TP45mA47mTU0TEAJ988j/pZgeSZDQSJ/uYRtMSpFVwTAEcWa3A/ANGKjACbf6omXFVHMXYrkm2uIpCvNpV1mKODxxEtarHNhs 7cLMJ9yS SVUKZz/XjMMB0hVjuoxkeDYbHejjn9fQSQxfD/RSYOwfoOezeeL6j0fah5E0+VqiJXi0wE2bc2ZCkoVcXlOVivZS+CjzN5w7+8ewD41UZYozaviHWjf701iQIMCvTd5mVGPsEvdBBUnSpaIlW22vUR6zJ9uwGH99HRh9fXCnwAEkkOeAX3IJ1REuZnk0wWaIDvIzl4PSC6ruXZbu0MEg34KJb2EuQFh/aJGgq9dg9cmM6iasOW9DDj28De0FOZ6J/K3IFNa6Gm2qZH9maQAnsE8fpZAM6W7jlrF2m/gVAojsh4+1Pe9jFgxr6PyZ0+nEsc/BAvqa82whTkT8F7FvAXZ3YeAYge3IFuQzLUomd6G+PUHPRxvh+Z7sINw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If memcg enables pmc, it will cache some pages. However, if all processes in memcg exit and there are some remaining pages in the cache, those pages will not be used unless memcg is deleted. To avoid this situation, a periodic reaping job has been added to each memcg when pmc enabled, which will reclaim all the cached memory in the memcg at regular intervals(default 5s). User also can change reaper interval time like below: echo keys=reaper_time=8000000 > memory.cache This memcg will reaper cache each 8s(type unit is us) Signed-off-by: Huan Yang --- include/linux/mmzone.h | 6 ++++ mm/memcontrol.c | 77 ++++++++++++++++++++++++++++++++++++++---- mm/page_alloc.c | 1 + 3 files changed, 77 insertions(+), 7 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 773b89e214c9..b56dd462232b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -608,6 +608,7 @@ struct mem_cgroup_zone_cache { struct list_head pages; spinlock_t pages_lock; atomic_t nr_pages; + atomic_t nr_reapered; atomic_t nr_alloced; }; @@ -616,6 +617,11 @@ struct mem_cgroup_per_node_cache { struct mem_cgroup_zone_cache zone_cachep[MAX_NR_ZONES]; struct mem_cgroup *memcg; + /* cycle cache reclaim time unit, us, default 5s, 0 means disable reaper */ +#define DEFAULT_PMC_REAPER_TIME ((5 * 1000 * 1000)) + unsigned int reaper_wait; + struct delayed_work reaper_work; + /* max number to hold page, unit page, default 100MB */ #define DEFAULT_PMC_HOLD_LIMIX ((100 << 20) >> PAGE_SHIFT) unsigned int hold_limit; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9db5bbe63b34..ae6917de91cc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7101,6 +7101,39 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, return nbytes; } +/** + * This function use to reaper all cache pages by cycling scan. + * The scan interval time depends on @reaper_wait which can set by `keys` nest + * key. + * Default, each memcg which enabled cache will be reapered every 5s. + */ +static void pmc_reaper(struct work_struct *worker) +{ + struct mem_cgroup_per_node_cache *node_cachep = container_of( + to_delayed_work(worker), struct mem_cgroup_per_node_cache, + reaper_work); + struct mem_cgroup *memcg; + int num; + + if (!READ_ONCE(node_cachep->reaper_wait)) + return; + + memcg = node_cachep->memcg; + rcu_read_lock(); + if (!css_tryget(&memcg->css)) { + rcu_read_unlock(); + return; + } + rcu_read_unlock(); + + num = mem_cgroup_release_cache(node_cachep); + + css_put(&memcg->css); + + schedule_delayed_work(&node_cachep->reaper_work, + usecs_to_jiffies(node_cachep->reaper_wait)); +} + static int __enable_mem_cgroup_cache(struct mem_cgroup *memcg) { int nid, idx; @@ -7141,8 +7174,13 @@ static int __enable_mem_cgroup_cache(struct mem_cgroup *memcg) p->memcg = memcg; p->hold_limit = DEFAULT_PMC_HOLD_LIMIX; p->allow_watermark = DEFAULT_PMC_GAP_WATERMARK; + p->reaper_wait = DEFAULT_PMC_REAPER_TIME; atomic_inc(&pmc_nr_enabled); + + INIT_DELAYED_WORK(&p->reaper_work, pmc_reaper); + schedule_delayed_work(&p->reaper_work, + usecs_to_jiffies(p->reaper_wait)); } if (static_branch_likely(&pmc_key)) @@ -7184,6 +7222,7 @@ static int __disable_mem_cgroup_cache(struct mem_cgroup *memcg) p = nodeinfo->cachep; + cancel_delayed_work_sync(&p->reaper_work); mem_cgroup_release_cache(p); kfree(p); @@ -7207,7 +7246,8 @@ static int mem_cgroup_cache_show(struct seq_file *m, void *v) if (!READ_ONCE(memcg->cache_enabled)) return -EINVAL; - seq_printf(m, "%4s %16s %16s\n", "NODE", "WATERMARK", "HOLD_LIMIT"); + seq_printf(m, "%4s %16s %16s %16s\n", "NODE", "WATERMARK", + "HOLD_LIMIT", "REAPER_TIME"); for_each_online_node(nid) { struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; struct mem_cgroup_per_node_cache *p; @@ -7216,13 +7256,15 @@ static int mem_cgroup_cache_show(struct seq_file *m, void *v) if (!p) continue; - seq_printf(m, "%4d %14uKB %14uKB\n", nid, + seq_printf(m, "%4d %14uKB %14uKB %16u\n", nid, (READ_ONCE(p->allow_watermark) << (PAGE_SHIFT - 10)), - (READ_ONCE(p->hold_limit) << (PAGE_SHIFT - 10))); + (READ_ONCE(p->hold_limit) << (PAGE_SHIFT - 10)), + READ_ONCE(p->reaper_wait)); } seq_puts(m, "===========\n"); - seq_printf(m, "%4s %16s %16s %16s\n", "NODE", "ZONE", "CACHE", "HIT"); + seq_printf(m, "%4s %16s %16s %16s %16s\n", "NODE", "ZONE", "CACHE", + "REAPER", "HIT"); for_each_online_node(nid) { struct mem_cgroup_per_node *nodeinfo = memcg->nodeinfo[nid]; @@ -7242,9 +7284,12 @@ static int mem_cgroup_cache_show(struct seq_file *m, void *v) continue; zc = &p->zone_cachep[idx]; - seq_printf(m, "%4d %16s %14dKB %14dKB\n", nid, z->name, + seq_printf(m, "%4d %16s %14dKB %14dKB %14dKB\n", nid, + z->name, (atomic_read(&zc->nr_pages) << (PAGE_SHIFT - 10)), + (atomic_read(&zc->nr_reapered) + << (PAGE_SHIFT - 10)), (atomic_read(&zc->nr_alloced) << (PAGE_SHIFT - 10))); } @@ -7257,6 +7302,7 @@ static int mem_cgroup_cache_show(struct seq_file *m, void *v) enum { OPT_KEY_NID, OPT_KEY_WATERMARK, + OPT_KEY_REAPER_TIME, OPT_KEY_HOLD_LIMIT, OPT_KEY_ERR, NR_PMC_KEY_OPTS = OPT_KEY_ERR @@ -7265,6 +7311,7 @@ enum { static const match_table_t fc_tokens = { { OPT_KEY_NID, "nid=%d" }, { OPT_KEY_WATERMARK, "watermark=%u" }, + { OPT_KEY_REAPER_TIME, "reaper_time=%u" }, { OPT_KEY_HOLD_LIMIT, "limit=%u" }, { OPT_KEY_ERR, NULL} }; @@ -7282,6 +7329,12 @@ __apply_status_for_mem_cgroup_cache(struct mem_cgroup_per_node_cache *p, STATUS_UNSET_DEFAULT_VALUE) p->allow_watermark = opts[OPT_KEY_WATERMARK]; break; + case OPT_KEY_REAPER_TIME: + if (opts[OPT_KEY_REAPER_TIME] != + STATUS_UNSET_DEFAULT_VALUE) + WRITE_ONCE(p->reaper_wait, + opts[OPT_KEY_REAPER_TIME]); + break; case OPT_KEY_HOLD_LIMIT: if (opts[OPT_KEY_HOLD_LIMIT] != STATUS_UNSET_DEFAULT_VALUE) @@ -7319,7 +7372,7 @@ mem_cgroup_apply_cache_status(struct mem_cgroup *memcg, } /** - * Support nid=x,watermark=bytes,limit=bytes args + * Support nid=x,watermark=bytes,limit=bytes,reaper=us args */ static int __mem_cgroup_cache_control_key(char *buf, struct mem_cgroup *memcg) @@ -7361,6 +7414,14 @@ static int __mem_cgroup_cache_control_key(char *buf, return -EINVAL; opts[OPT_KEY_WATERMARK] = v; break; + case OPT_KEY_REAPER_TIME: + if (match_uint(&args[0], &v)) + return -EINVAL; +#define MAX_REAPER_TIME ((10 * 1000 * 1000)) + if (v > MAX_REAPER_TIME) + return -EINVAL; + opts[OPT_KEY_REAPER_TIME] = v; + break; case OPT_KEY_HOLD_LIMIT: if (match_uint(&args[0], &v)) return -EINVAL; @@ -7402,7 +7463,9 @@ static const match_table_t ctrl_tokens = { * 1. nid=x, if input, will only change target NODE's cache status. Else, all. * 2. watermark=bytes, change cache hold behavior, only zone free pages above * high watermark+watermark, can hold. - * 3. limit=bytes, change max pages can cache. Max can change to 500MB + * 3. reaper_time=us, change reaper time, default is 10s. Set 0 can disable, + * max can change to 10s + * 4. limit=bytes, change max pages can cache. Max can change to 500MB * Enable and keys can both input, split by space, so can set args after enable, * if cache not enable, can't set keys. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 54c4d00c2506..1fe02f4f3b33 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1310,6 +1310,7 @@ int mem_cgroup_release_cache(struct mem_cgroup_per_node_cache *nodep) } num += i; + atomic_add(i, &zc->nr_reapered); atomic_sub(i, &zc->nr_pages); } From patchwork Tue Jul 2 08:44:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huan Yang X-Patchwork-Id: 13719115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 788BCC3064D for ; Tue, 2 Jul 2024 08:47:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1F4F6B0088; Tue, 2 Jul 2024 04:47:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EA8496B0092; Tue, 2 Jul 2024 04:47:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFB586B0093; Tue, 2 Jul 2024 04:47:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B37F76B0088 for ; Tue, 2 Jul 2024 04:47:42 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 725A9A1BAD for ; Tue, 2 Jul 2024 08:47:42 +0000 (UTC) X-FDA: 82294184364.24.FABF7D9 Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2044.outbound.protection.outlook.com [40.107.117.44]) by imf18.hostedemail.com (Postfix) with ESMTP id 7DE081C000F for ; Tue, 2 Jul 2024 08:47:39 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=jbDyxQC+; spf=pass (imf18.hostedemail.com: domain of link@vivo.com designates 40.107.117.44 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719910042; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wayXX6if5q49kSH+e7BH2GqjBW2PQyGEKDlKzSgTM/M=; b=pKiBEfLO9iMMm/02pg/cvWG3uXOgZS/qAo72UgfFI1AS9pTuIUk2RCsDHYSXcpFD0v2NCQ KgYmUsM8kCuYBDkgDnNNFVuPeRMzCJ/Een6CuAm7janAsMUJ+/l4RUSKkUltaGLOFzl4Qh 2VpfVerp/qsZbNfhi1x3F//DGE3k27g= ARC-Authentication-Results: i=2; imf18.hostedemail.com; dkim=pass header.d=vivo.com header.s=selector2 header.b=jbDyxQC+; spf=pass (imf18.hostedemail.com: domain of link@vivo.com designates 40.107.117.44 as permitted sender) smtp.mailfrom=link@vivo.com; dmarc=pass (policy=quarantine) header.from=vivo.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1719910042; a=rsa-sha256; cv=pass; b=P/Teqrfw0coG5gxoynXj9tr/Cj+TbHo4xOKwH7fOTyDJeQqg6svwGUEnQ4SFzaCCMlAAZ8 aFf9qT8Lx7ca+/S5A+BiwSnuBGV8QgknAl6T0qDmFwn/lpUytblvKDKgwwLudLhitwcXyf iIJt0V/4lm7a8dJAJ9JPaFxkqHj2wWo= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J89eKsVjJVafsKba0ue95qSrgWkHY/NfjpKu5FkaidCshw4qSUuviuh7NGddXzA6s3pqsFFohWrmDaz97YYjOPF7hai/RVyTD7r+VRsKximo/GksF5p/ZZaFaTtp7Fkj1tzJfRTckefZOcELfRGVSg5upPtbr/YQBPgUJlDpgoSt/VqqJj4FWqQFJAJpYpJlQQXSQtgaY0a9JZ2oOZbVH2LIgwL1DWeqNtkaBA5dun2j88+SIqcgV3MVI1rn7bhWGo+/h3/q4mxmIzwZDd6FRLOo7vhxLAs5jCEOG+iw+R3oIrxuNTQjQCBbt3dAd7pPk5g1b+H1/Z3IoPZtftuHbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wayXX6if5q49kSH+e7BH2GqjBW2PQyGEKDlKzSgTM/M=; b=GzE6U8GDjkpeKwdzvJMgMAelLrT+IiKm9e5HzN4LAzseTAM9Z1BJ5oZ+Y7h7dyzk3+zz9569X++cOtMj1OD7G/M/1YwqeeivUj9NL2oYnyHMBgtBGrLH6loniuFZiSMMz/q9A33QRo3GJ8Md6MAxS3cUlXdsEaA8o29WRtS50g321bzgjrSDC3fbehKJ5HqVJubl2fx151BqcRcyfsxJF3Nmk4Q3zKzxgho4fAEVyM8oz0zJrDUcLJVAfeFQakV2HykxiFgFNOqQvNQ65y1pguPOD9/NbgYMF5mpK64dDBwi+dM7Eon2uFMiy3gl/4ZMSDN5H5ZEwWv/nn6lUg2PTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wayXX6if5q49kSH+e7BH2GqjBW2PQyGEKDlKzSgTM/M=; b=jbDyxQC+prZBsQ+Bb1fY+q59ALKp3VoiOI12VUaQGpE75r52B/+htLdIVeCnQNUgKqaacaF/6HhWB1I2TuHd2l5XoxiozliXXp/A643ztyUPlbhEYVs5wJRIpx9Kplx1/PM7dwQjdD/Gvpqe3DCVbY9RFCk49t1URN6/QJl0qXys7jG2CGOLXwyX9zzykd5bHbYCgNcANGWVTg/MIFONn9UHKcYASqBpfgOn+D8iYiYb5hKfNS73H42QaQIycvEF7iuGXlV/Ce4s9bkCLbXLKkZAmaP2fzOL2895x1tBmUMhC7uVG46COfWqf887BvTvv4beX6mcl5+x3WRsuoYbwQ== Received: from PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) by SEYPR06MB5790.apcprd06.prod.outlook.com (2603:1096:101:b9::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.23; Tue, 2 Jul 2024 08:46:20 +0000 Received: from PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f]) by PUZPR06MB5676.apcprd06.prod.outlook.com ([fe80::a00b:f422:ac44:636f%6]) with mapi id 15.20.7719.028; Tue, 2 Jul 2024 08:46:20 +0000 From: Huan Yang To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , "Matthew Wilcox (Oracle)" , David Hildenbrand , Ryan Roberts , Chris Li , Dan Schatzberg , Huan Yang , Kairui Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christian Brauner , Jan Kara Cc: opensource.kernel@vivo.com Subject: [RFC PATCH 4/4] mm: memcg: pmc: support oom release Date: Tue, 2 Jul 2024 16:44:07 +0800 Message-ID: <20240702084423.1717904-5-link@vivo.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240702084423.1717904-1-link@vivo.com> References: <20240702084423.1717904-1-link@vivo.com> X-ClientProxiedBy: SG2PR01CA0148.apcprd01.prod.exchangelabs.com (2603:1096:4:8f::28) To PUZPR06MB5676.apcprd06.prod.outlook.com (2603:1096:301:f8::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PUZPR06MB5676:EE_|SEYPR06MB5790:EE_ X-MS-Office365-Filtering-Correlation-Id: 8d1a0e68-ab3e-4099-7ed0-08dc9a7371fb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|7416014|376014|366016|921020|38350700014; X-Microsoft-Antispam-Message-Info: vsqpTpPI6N2WGgg0iSkF+In/Pxw7vcXP7rull1Detp3BnAezK6peCymBMaYE4BTFC6VUgTVrFrfSdpv3j3OnmhBlaItao8IymvyMiZta04sqX5geHlLQrPeiCvpfB9BZxggphQxFj0B13CyQ1IcW/6n8Rr3dmscfIRVkKfxwc1Qm/uCaQ8JAuPP1fmtxqkUW7kKqOM1rPGhanwlCUIVSKXEwu42ZIywJdRAh0CqACi7cnsMw0J72BSgDFph+0otu3EnaBqT8IAJp1bgcGzu6XgewjGax77qA7XONpmp7r1GLSQ+sc4gHSexyEMdxYmPj3fjUF6Ftjx9hBPY+YmhRnQpLIicquovcuTcMVFSwM+6i6UYQj5uxuI2FqThOeuvRXOmFRcSgel6nWqrtHyRJ+k93lm4bJj+JiLfUvQgJAuyaRETxJVoAyq987wSmodwxYIkUuL9VqRilkh7sPn8hfTIkvDa41pgjzS7TslaCVrVgzSVBfwI+t+CskecOyWG42lcABFw4l16ai+RxNS/CI/xyePVe/dByfkzw4uc0TM2CoRiGFFPMs/QYEhbxSKbR1ja+t4kbUYqWE4CXbPoAG4o+n4GEmyiNfQyVjqjZV5smuauptrRJfJzW7xIi4rYrZLLKi7ddAPKbxhA1crli10kkLViiKmQ5WRA68a7ANoGaJNl+Bdz487InmmjR7fWscyO3zsLdICv9+Wz3rTYATn3cFobDUgXlHQ93KRVUgKcD9EhRvIKmTpoU5q1QXS8C51vlN0L8kW9LO8iwVNR+b0H36stJdheAOBLr6A4g34DllIgz2LxrGLa+gCuFXXRRng3FpPIMcrkSsh/Z9NNgUP3B7d4zDgzBKrHNFOPDrVho8uAQR5cT7VBpmIiN+2otuTj3xye7GJcty7PbxYb3eOk7GeaOsJoDc9XroVRhMixenYQt+2RPiUspsRJS0DGwcc9lqCryrG/E2SfRN/DktqjhbjyPEU16mrc3HCCcZvYBnUKgYa/Rx4Z7s4Dhw4kB3fBblsQvAs6p57QxcsXQYwnjKsrhQP76LH8E4OhfuCI73EidLSOTdac5D37a7InU1rU+MHaHDGx1iencOn7C9QuDSlCCstDJKl5pSZJde33CrlyVi+BZswztEX2ymA2J+oNWBj7YaYpmtQQPd/AIxJZkW/ZL1mWq4LRwzv0lyYL0iKXLNSk1ZrW9G1TXl9Kr/VbZzmN/UbDrJeqyxi7oLTf80kuAlKGr757JnHMvNoOgdARgEh3+UJzWeTIh9IZtvYzce2SndrehSFj/QqC9rsg8ph/CnGYJ+82MRxiHJqn0czHyDVIq2Qewtukhq9+lFDhjrk+vXCo/Nhw9WmnfvrB3huyslvturCbfdYC83RR+zRrio16D5qzzcY3h0rR5+zQrWDRgDEq37NgJxjmchkF/UEvDNUOQDrvMsxCbcho= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PUZPR06MB5676.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(7416014)(376014)(366016)(921020)(38350700014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: G00yIZH0KEXv5OXfIs8l6NEO6NXL/A61iR75yv9JrQ4l9AbLuHyffUj7a5J3d2gY4FFKiG50arGFekXXHwmcwiP1TVEfJ5+X8k8JaqIJxEpkrn+2MQnmtIKbEiG26ITnRSNlhBjhggQHSl+hdj8WUR7tKjC2qAx/m7RoM1rsrs8KFH86Vb0t6tdSk7W+AAowq/LfC/ktSMvcNnKQa1+L+qUsM1iqyXCII1obprL072SPiPYSARlvA6iLrLKNYhEMSbOUzT7wA5DcqCLga3rSXvefwjQeowPJQbbmdt8SnEk9S6EdbcKT0AzGya17bOURMeoo7WaeYAWOA6UCBCKpDXZhY0fToSzqbwqO1hSbSxF4paYKYxZ8Ys3QguAz+K2dBqqC7OrC8DujIUwT4Y0m485VLXTvVtqH3YQ8f7gCa2QAN0Nk0jtHCnZw+pqUdnISEoAp2Kocpz32oDfyFPBNJvPfCZfvh5iIzSO2U15w5fgcPMIndPcVppPhD1a5625W2n5hn/uaKN02jA+97hvrCttA5QoUft2OyxyAwND0cOPUwSyrf754LWRHAvGz6vswU0/eqmeFpbRpmEWcbqX/vLB4xAHjOQCy7dz9OMMDD+hh4RVeMk++XHoB7pk4U+bo0ud7e0ktqiGtK0FCTYKbZYg/a3XhHMypbn8xtHyJZaE366zSpIc9t9o8o9n0nL+kjzfLxx1ILNwQ6oLoBgV8WwmcppeJZmriG45r3+UL8iPsJxqcmXDDKYCVDh6MrgpnNMpvNRs9clGlZWRTuJrHlPlARJWIOYaoQAuhMZGkCAHQaaKGhinKXekj2++hFYrm3lq1ayAu2iEbNgjprj9jYb1PwPEUDjWU4YOex29tH1k5T+/fyLvLBaPvyuCsbmUy3zwJZ7cy9x2f+pODrkiCD3PIntCpSMwihFOytiOuBnW2aZARey2Onmy7AfZnoRPliiFz6cuyjlg1dePCg4pvROMjZulm9WvB69IaVNruS6JhxEe179k0jrg2KJwp/Jv4m+Ucu+OYTD4v4PeRJCUMb8yeqU5ZZVCxDGexGNRxry9ris3Cd2GlkRPXzJKL43BgbsXOHsHj8IFLskUiCy/Xu5GkA3WX4kHKSDb4sdCnMvUEFSh6DGqI74aqL8ZM4q5khl66VaGGiKE5yNzpD+kznrzOnrIOFOkGeJVpAMRFVfj6DXfD/AkVvDq2oRY8DXP5jkks9u8pdmy16PSnd9KIpuNAzG7TpJrdhBt1BiVzXRrqde6vaDcRgo9/M8QkXYsDXU2x9cdvzCUDUr1nJyEDj2tq7h54ffeXv7asqoP7gSJz+YLRsGUFyKVSVqpw6ydEAttdptNFTxm3j86kN63QsiqTD9KKSP/X8DlDH39+/lAdnGHAv7HtB8HkfFSOsiOX4Z/AU3lTUzLMKjQl18ZSuWiRGhPRgoQHjkHcdVXzhyy0rTAjCzuthXBXTgkGEtKfm4gH/A+AyZpT2qAqUnCDuaNCB6ZB6w8PSH2DZuhMK75cmRRK3CtpfXHVNBG3tmeP3Ns9pYY3m6vE6hKCBO8ZqBDEqdwA+zWHWkPl98DkrvkqkTvMPWBgApWyp+O0YKGP X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d1a0e68-ab3e-4099-7ed0-08dc9a7371fb X-MS-Exchange-CrossTenant-AuthSource: PUZPR06MB5676.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jul 2024 08:46:20.5457 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: NzEupW0V4BOAb11XvdtEl5741crWW/za4brGf8wfDzJOF1VhuEWE95CWnRRZ0d9S8GmZWXDn1bEoyugO4NVe2g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEYPR06MB5790 X-Stat-Signature: yzdgc6s8zz3dykczoukx8hrayk6xss99 X-Rspam-User: X-Rspamd-Queue-Id: 7DE081C000F X-Rspamd-Server: rspam02 X-HE-Tag: 1719910059-145365 X-HE-Meta: U2FsdGVkX1/cW7bWZkUd+7OMQsVQgw8KV2pmJxv3tKHQf93yLZ3egVr/spQPDgmiCvGghhdT0qCb0yjR4JBtqfnvdJ1RKg9+kIZGxfTgXCSLl8wERsiiGDZzIdJYpetLypuJIo4gCdoHgnKFAJlVwxPQxrkMnLe/aJScI94+xz5mgAp//5tndHB1q7CyLG9BMpNKezISxOl/v54ClxMQhc51r6oqeGgsxOuHbB0QwfH5i7m75phacyzxr7c+24N8WFJDv2rmCgJ7YtxFfuWxnW7HJCzVOTu02ATYudBE224jh+CESHeMdeQMbxljWk/ftFncl/iCzaZQ1YMmyGA4y9ovq59IbFoaWmIhG7h7ZjmLOapPWY3hbmNrsgvzvJm77uGSsEKOYFB1nrr0cD8wBs4A0snoulo5z7tVfAgVX1Q62Feq+Npxc07vjaUuv5DPoRry5YaF/OdSmKR+fmN2OaWnzRgI45qouJViNmhrcD/7sIpkBLcQrW0kLaj8GLJZrhn+voKYx16XIUzbx7FsWnMnEEpLPxL3yrCNEE9gBFRpuG/GL1NR22rIE6OS7OZboI+efHR6WfuFM9MP+IpualpRXcdE+ZZ0dfO/xBGdITAkGDov500zuZH+hI/iPHjRWjaCb5d4IhO/0WzUXO/4YOLaptHdFi3oGVAC5G3X1zQ4uKhZhxOHYntaDbB3a3vUP8h8IUsVtd1PgYAshAo7OwNqLIFF2ESOl7Gl8Ocobd5GBZ3/c1rlZTY32QUFDR06WQlILMXtCNGchwO8oPoSOq51/A/vrgBbJzZHb5j+Gv4vU8SJ46JcNUwN9Wy/NdW9AF3Mxyb44bmIE4LEwjLWYfouIyAGSZU/Vxem/BKTv3mCICdvq6vHTVn/eeLhpUOiXXMfyYrm+decwa5A3F40YxRCrSEk6Dyly3YtKTwrZGqjDbW3EGZrFFiQV75bBbh/zMmvoIB8pxFEBi/8hyc dIS2ElgV D/+MN4q/A6S9VyUesnVspnU9xA3/fad5qv1kXsppdUxMQNbHcPHmYaLBpOT6MTwwZjj/fX4OkDmb+CcOi4TzkUdKBvvSFyk/mT40PZselrqsj0nmIixyLJL1jcusr9uXjWc0bvckJF3XI2FOtEDvNhDpt7RJtS/qHf2qDNKvRrLBKgl/UKPtUzkVVqt/Kdm2bYbcNWR502aZqImEdE4TgY+MJ7Mg2osSTIMCOuFejwzlgDL0N4OEiE5QFMytTeawdmqb7upFkjzX95AZ8icRncwL1UVa7pREKdS8F5Vm/yLZhRDkAJK5aGlos9EoEvgtMfRFDMXuaU/oNTBhE3pYfgl8vZJxR62sE7HasGBRyUeopULH7SUYoR2hm9g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch let each enabled pmc's memcg register a oom listener, so if oom will trigger, release all hold pages. Signed-off-by: Huan Yang --- include/linux/mmzone.h | 3 +++ mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b56dd462232b..640a9cf51791 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -622,6 +622,9 @@ struct mem_cgroup_per_node_cache { unsigned int reaper_wait; struct delayed_work reaper_work; + /* listen oom event, release hold cache */ + struct notifier_block oom_nb; + /* max number to hold page, unit page, default 100MB */ #define DEFAULT_PMC_HOLD_LIMIX ((100 << 20) >> PAGE_SHIFT) unsigned int hold_limit; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ae6917de91cc..3dfb2a17c1fd 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7101,6 +7101,33 @@ static ssize_t memory_reclaim(struct kernfs_open_file *of, char *buf, return nbytes; } +/** + * This function listen to oom event, if oom will trigger, check and release + * all holded pages. + */ +static int pmc_oom_notify(struct notifier_block *self, + unsigned long notused, void *nfreed) +{ + struct mem_cgroup_per_node_cache *node_cachep = + container_of(self, struct mem_cgroup_per_node_cache, oom_nb); + struct mem_cgroup *memcg = node_cachep->memcg; + + unsigned long *nf = (unsigned long *)nfreed; + + rcu_read_lock(); + if (!css_tryget(&memcg->css)) { + rcu_read_unlock(); + return NOTIFY_STOP; + } + rcu_read_unlock(); + + nf += mem_cgroup_release_cache(node_cachep); + + css_put(&memcg->css); + + return NOTIFY_OK; +} + /** * This function use to reaper all cache pages by cycling scan. * The scan interval time depends on @reaper_wait which can set by `keys` nest @@ -7176,6 +7203,9 @@ static int __enable_mem_cgroup_cache(struct mem_cgroup *memcg) p->allow_watermark = DEFAULT_PMC_GAP_WATERMARK; p->reaper_wait = DEFAULT_PMC_REAPER_TIME; + p->oom_nb.notifier_call = pmc_oom_notify; + register_oom_notifier(&p->oom_nb); + atomic_inc(&pmc_nr_enabled); INIT_DELAYED_WORK(&p->reaper_work, pmc_reaper); @@ -7222,6 +7252,7 @@ static int __disable_mem_cgroup_cache(struct mem_cgroup *memcg) p = nodeinfo->cachep; + unregister_oom_notifier(&p->oom_nb); cancel_delayed_work_sync(&p->reaper_work); mem_cgroup_release_cache(p);