From patchwork Tue Jan 14 02:19:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FA74C02183 for ; Tue, 14 Jan 2025 02:19:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C277280005; Mon, 13 Jan 2025 21:19:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27292280002; Mon, 13 Jan 2025 21:19:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13C46280005; Mon, 13 Jan 2025 21:19:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EB0CF280002 for ; Mon, 13 Jan 2025 21:19:53 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 72E28AE531 for ; Tue, 14 Jan 2025 02:19:53 +0000 (UTC) X-FDA: 83004451866.14.580CFCF Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf29.hostedemail.com (Postfix) with ESMTP id 8FAD5120008 for ; Tue, 14 Jan 2025 02:19:51 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JfeZkClC; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rVaRyHY3N181Ob6HytxKBrw588chz5TJlaChctBoK1M=; b=YGT3e6nWAHM+fzat7HMan8Gjw23+wDddV+TzLDPAVXb+dVorY6Ev3bo7nMiC8UC5Z6VzEo EOeEJzT1dYrRYcgks8y3JuBqDVi1iZW+atzXyU+cIzZ+trckXY1FPyTOMlAd4nKmyOEInN PQpyu/3eWQE1D49an0HDuKgpXzaYX/c= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JfeZkClC; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821191; a=rsa-sha256; cv=none; b=Rk4Kj4S+DbHAZrKLA1QjcijpFLv55Tc4B6gdvNpwSyKG+d9yZGfbwyWa8W90ELQynRG1zG xhidIla9rzqA0BcWVi1ai/Vv0LPm8YMUU/I8CEzlmGLXO2oc/MoYPidYG5ZUv/36UD6L23 6GN50/Attsls6UTO7qXJrSsr/aoqzjg= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21634338cfdso73813725ad.2 for ; Mon, 13 Jan 2025 18:19:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821190; x=1737425990; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rVaRyHY3N181Ob6HytxKBrw588chz5TJlaChctBoK1M=; b=JfeZkClCCnG5e8XPez4mMPhtFaLi+3YfEGeInD5RRgmPf8OTM+hJpq9UDvmSZDH+TR AM7WcXNHgcnGXNfwYpm4tNiQlpERaQ1qyEMcl3PttWXvQHrnPXAX5OpL1FcnLMBxhynv 0R6E8zY67lVImlHQQwzGmpbPsFL1ZnvTKbsBHPFDQUIZLNF4T33vqbdHPm2GyuUtHDB1 knyrWCvL9txwX+fiZ5d7epFqpqdA8/yPMTFd8yQPIiSrJvY02CCxpx14VCpqiyZpnZWe uzIxSBQanISWETl4tD7eVMmw3Brfh74fyUzaeQJsY9G7+b3oy6TPFIRkg/uSt8GdwFBe nCgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821190; x=1737425990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rVaRyHY3N181Ob6HytxKBrw588chz5TJlaChctBoK1M=; b=KeYV2QXnq+8rQly1JD1jYddOR+HE8PhLZHcaH99vkGw7wGqpxryp7uiO7bfTqCLOtg /7XRkvqjAH1OGRqJ7YH7oOZ24nLNBycHrFED1A1pBztKVj3CqGpST5WgVdasRPYhugE9 jt8cX3QdfysDczd09SJvV1HwsbBBEs0bXqz3OIhV+bcPx8X/QaYVP4voR8JwFeIGzmvZ O52KiQ+FLLdFcrru8XdaxwuJ/oPoFv3y/uEqqpm/FhPIzJJmAJXuk80Is3nPxfIKpAZN eahnGzMslKlgeq+UHRlcVmqnnIx0n5vOZiX+DBBiNr0GdKTKJGoym+SRN4r1uAj49RP1 ckaQ== X-Forwarded-Encrypted: i=1; AJvYcCX6667JLm4ZEDYN4bKJ6wSYOeBk5dQjLLGD0SH2/uULPo9K7tOr+OnqDJTBT7nnldoVeBEsy3IeBA==@kvack.org X-Gm-Message-State: AOJu0Yxz+g1SMl7EflzyicP0VS6CWdDNY7EZsqiMAnXYP10PGDxkxts8 ujDWsPkqyAacM9QD01RMNIEk2HEKYJnLJ+rWXFK56itPhTa0d0oD X-Gm-Gg: ASbGnctkNMluHR4yf+fbUcsQVLo3oQ9+bdG9WeC12Hc2rTtoba/R3wQh+qd5kvFX1y4 0hMTzbeq9t6CMK8CLNswgEPfg2lWcnPaJ9IgLel1vjT7VK/+08GnnJRd7DRFzsLlYk+CVTKsFaS ybBjzyp6CzPUEYyVYJ1Qu03t/7Q51vOmuAI7V0kFPoOQFsAkmz/X9wHeHa/Zo8pqkzhpvF964rh fNDfN2fUT6HhjFKjsaun28W8ClxHsJ0/ArPYa5pbs2WfGXeF+/GMyYxLkRLUELY5Ik+TBN3CqUd zinI07g8 X-Google-Smtp-Source: AGHT+IFgd3F1vhcoVYBusGT4YkcS3cyvI9lPQCSRNl7H1SCdyLOMHx5+LSBkcObgthXTbJSLr0qoDQ== X-Received: by 2002:a05:6a00:3cc2:b0:726:f7c9:7b36 with SMTP id d2e1a72fcca58-72d21f325b8mr38139980b3a.8.1736821190203; Mon, 13 Jan 2025 18:19:50 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72d406a4dfesm6523242b3a.156.2025.01.13.18.19.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:49 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 4/6] memcg: Use trylock to access memcg stock_lock. Date: Mon, 13 Jan 2025 18:19:20 -0800 Message-Id: <20250114021922.92609-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250114021922.92609-1-alexei.starovoitov@gmail.com> References: <20250114021922.92609-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FAD5120008 X-Stat-Signature: b79tiwnugtckozrjayyfm95r853odte3 X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1736821191-478634 X-HE-Meta: U2FsdGVkX19buNe9rsi8hgy657FRAYbcr+TWkT3tCq8OxUMIDKo3zVTH+bGczFTfXLpNLm3sWzoPcAjCWMqRPg37DTUrMjP5v6DwpNbEOJlvFBjJhV06vAP1NkcQir/glJSSG6KZbTDqluf06nq3E9fVpbKdSr27iz5SqgzFTN7YAknFvI9cWKLVeYN+cAiqs/JzBtNVaQhDuYLXKwRAyiRKNkr1dv8E5huxQt+VVlE4F6qz6YCdV1hqrr4qlPJawovzG3zl8RcNPkMHp2yODvHayBlJ7MNaZmYvZTrKLvWz4EduQ+3nb9GU/d5ow68i+JQySBjFJR7yOyg4qmKEQHmV4lH3B2/A5TBlTLOrFAWmNTVPtDYjx5Mtl7LB4HrlBlPdu3NT8l89N88J7OgLYauCp72ioEKXJdlL8XLG/aUTG5SjnchBpY+Sk3qspS52ApDuJKwiQ3K/PE5SWIUtynJkLncrKNspkQfZCMhW8byRHM9FjEBBnCLShrc8I+INFjToxVZt3OffiZ12Uw+C52CDXsyYW4e2V31/XIfcGcgd0dMuy54ODr18QHKp1dOysYqTZsCu3I3MAMv79n3UFAxKs4pOJBQGROKEwY+PGd4Py5gWL7JluXN1fHpwVPlhU8WbTlgyHLe3zcqTc9gU8RYlQdH7yl7s3eatrgyb24W4ten5vEgULxWObagISo6ZHEKz/AgRmPLkPvq0akioEitt/BkoHK0y274F/JIrKiLljEkmmPzXiGc9/dNSg2gnNIG8oxTeTE1NHmi1XTOFpJq9e2WTqwxYomnDFnydPngHnrnkOnkNQPKN8A1tm3jnyt/qogde8Si0veOS/EJ30Qt3iLnepG99vmtFkzqzKsieyiDwaRBmwlSOI3VvhMGaoG8XEuJmn81DvNqc4Qa5C+aqww9pFHB5dNeDs70Y3K+s6NkMqFa7tUM4V6hivgJOsdTXPSPhsKwIiqDg1mj /zXB6rbE Ut7/Hj79I1I8NStf9hC96NIevklonFtkHyp44au/WcK1PDXfMfHfOjwKfIKO9NC9RmdEzBR8VL88CO9AWPPJKIwruy2TUtRlwQ7BpZEiqibrCpu3Vp+PamLL8yzsZHJ5JqvqSrJGyHkJgnmfzuUCvcfD3xVnLz6QAYiOHQo4yAlG7yTgqWBDEj8Tipzr0BLyGWsds5W+5IS2Y5y4+yy8VRZZ/cq272T+ejadkTU4EHfsMRT1zBsRCQdaU2BinKlc+03D0ia9GFza5R2HK2NGr8XgvOmkHpQHatPHTuJkCPpjzBK2hM13lxSkRSKcZjlUhYRKQ3Ea5NnWRm1vu6OAbaIlxjFK9nJN/dGZElBe9t25C3y8YdeYC5q+85sowKLPaaXQQqGtALfuJwNDVcKyNoKT79x0AmdYgdaE3+ApCY+v0OOsJ2sTQJE/c7m6BB+GqPa7RGXvWJLQySTsqbkExabrZlbpxZf791ywLhyiVcjugbBJFuCvikx0wwA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Teach memcg to operate under trylock conditions when spinning locks cannot be used. The end result is __memcg_kmem_charge_page() and __memcg_kmem_uncharge_page() are safe to use from any context in RT and !RT. In !RT the NMI handler may fail to trylock stock_lock. In RT hard IRQ and NMI handlers will not attempt to trylock. Signed-off-by: Alexei Starovoitov Acked-by: Michal Hocko --- mm/memcontrol.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7b3503d12aaf..e4c7049465e0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1756,7 +1756,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, * * returns true if successful, false otherwise. */ -static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) +static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, + gfp_t gfp_mask) { struct memcg_stock_pcp *stock; unsigned int stock_pages; @@ -1766,7 +1767,11 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + if (!gfpflags_allow_spinning(gfp_mask)) + return ret; + local_lock_irqsave(&memcg_stock.stock_lock, flags); + } stock = this_cpu_ptr(&memcg_stock); stock_pages = READ_ONCE(stock->nr_pages); @@ -1851,7 +1856,14 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + /* + * In case of unlikely failure to lock percpu stock_lock + * uncharge memcg directly. + */ + mem_cgroup_cancel_charge(memcg, nr_pages); + return; + } __refill_stock(memcg, nr_pages); local_unlock_irqrestore(&memcg_stock.stock_lock, flags); } @@ -2196,9 +2208,13 @@ int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask, unsigned long pflags; retry: - if (consume_stock(memcg, nr_pages)) + if (consume_stock(memcg, nr_pages, gfp_mask)) return 0; + if (!gfpflags_allow_spinning(gfp_mask)) + /* Avoid the refill and flush of the older stock */ + batch = nr_pages; + if (!do_memsw_account() || page_counter_try_charge(&memcg->memsw, batch, &counter)) { if (page_counter_try_charge(&memcg->memory, batch, &counter))