From patchwork Tue Jan 14 02:19:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13938318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 634ACC02180 for ; Tue, 14 Jan 2025 02:19:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D217C6B008C; Mon, 13 Jan 2025 21:19:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CD1F06B0092; Mon, 13 Jan 2025 21:19:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B723E6B0093; Mon, 13 Jan 2025 21:19:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 978E96B008C for ; Mon, 13 Jan 2025 21:19:35 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 29C271C6EDB for ; Tue, 14 Jan 2025 02:19:35 +0000 (UTC) X-FDA: 83004451110.17.15A7E46 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf17.hostedemail.com (Postfix) with ESMTP id 4987F40017 for ; Tue, 14 Jan 2025 02:19:33 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Hrh0CNro; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736821173; a=rsa-sha256; cv=none; b=H4moS/YYobBrWrIUfPFzhXgG3NcH2OW7ZYP5W2KXyqHGagAL27xiowU//grLvAhHYhhW5Z hc7UA2riYQKAamfg6C10RpQ5xcqnDSn4mXCr07RfzmuhUpLncXjBBShPhfQjMonpdM6Fst iFU4E+WbtRk18LXqsYyaqDHe6VU9kVI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Hrh0CNro; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736821173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=JnxIlzgPMp8m5mNwVmolPyMCi6FeHvtuBonEllyjCg4=; b=LpIlrJtMEuUlG1E/PHgoMeYK+5+SSOTf96HOQ2HSrtZg25BKtK1gHNeZ0H3Fq+uZPbW87U aX+5IWfO+x6nK7ayEl1bU2LH2TEF8+hWcIPTdh4/iU79Tmx5/Iwc/OyqSEpY+1TJCq3B6/ hZSQV4r+WcDCiF7+YlqBxHL2oPKk7WI= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2166f1e589cso104363295ad.3 for ; Mon, 13 Jan 2025 18:19:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736821172; x=1737425972; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=JnxIlzgPMp8m5mNwVmolPyMCi6FeHvtuBonEllyjCg4=; b=Hrh0CNro2OtT6aHCKIO0W7wsJ9/rpSVnWUSqfupHn/duXz+V0wNln4qeFD2CjWN3OQ k7CKrh6UGuARx4kzhbeK4j3cgUu6A9IfWOgZWYyD/xAGabaCwwsrNEwJRyi+eEo3/EyK /mUOCkBLG2PeWRVCOq1Rombd7cYsCvrrCH73GQuqYZfM9nB9j6UNn7SpFA4O+V+M9sIZ M2XIdQgc7gQRfsyTIMSOFx4vsfFf1rWor2yRLNeb9MRLTF2V3F5TNAj/GtoLKbw1EkFr REj6Ow2oekWqJ4EGh1OYPorITfg2WoFKBudD1UQ2POl57LFzHxcKP5O3RPbvTr20i1hA 9bQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736821172; x=1737425972; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JnxIlzgPMp8m5mNwVmolPyMCi6FeHvtuBonEllyjCg4=; b=iTXd4HOWrh3lh/TM27XeDko7s9MTK5u0iJSk19g2yjAK6b7pGJUyPS3qQT7JWeIk1t OkOBcoJJXLUZ6Hd1YQq9d+eDSfIm5DGiJkXljkjAHC+EXMG50JIgTrCHzUst3GBcud2F IaA8Qe8QZhtwh0IgRPEd5YY2v6dSKa1EOA8LKXY7gchrCrYpVp8YVQM4za6pS7USjbjT QGSE/uB5xDrQ1awCIh0UTlZzKbSgkflgxsUnyMjX3lxYz/JNO/2hWlCsaFn1IdSLMjLq lcVhmdiu2OUpna5j30yZCHWGTNGKW/MZKdX3AQV7gpHxN5CSSfMsLgsfVRDJNkhYVePS yjgQ== X-Forwarded-Encrypted: i=1; AJvYcCUI0Kyn4knQx7kI+phuZwyON+nqpcGUhD0OTGm4hjFg1PCwo8Az7hwFmupjF/jB5lGEQAy54yRhNw==@kvack.org X-Gm-Message-State: AOJu0YxFPitgSW4aDyxtzQwJ9q8mvDIiGWvUgWkKsBo3Lqtr2lB4S4WT 1zn4nGCYJfBo5glMx9lEeCiCXQhTSTv4B9OuplmYD6cXyBmJ8gks X-Gm-Gg: ASbGncu7V7mMt4D+ccrT4fH7+2Adrpt1GU9K8nRoaPuxcroMgR2TMdDQGhpbwIU5OaY +ml+vMxV/68DTHZpk3YAS2+GOALkNB+hstrkhd15yhJmzrt+4wrxtH9BfmkegvPFQh0YZRkMihB YlsFuB6FlAPCR+s+2fWPPs3qDKZ/tqlIsH7V20snH9GrXmDQc7MrtVmsm5BZRwBti3MXvqc6kuU GicbvYgJuiLLjYHoymAsu7fi5wi28ofRDbgQXRjMlMfabkJ/5rYNnW3BgbGHiKIwPAv/3fpWmmG nEkeGQqV X-Google-Smtp-Source: AGHT+IGs+L+RJhb/gY+29/U3wyZWsPjK2+A1FHlKlCHVaSYdjRzTWST3E11vOglMDbQRC1a7b6KlbA== X-Received: by 2002:a17:902:ec8f:b0:215:7447:ebf0 with SMTP id d9443c01a7336-21a83f665c8mr376178995ad.29.1736821171500; Mon, 13 Jan 2025 18:19:31 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:4043]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f153486sm59669375ad.105.2025.01.13.18.19.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 18:19:30 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v4 0/6] bpf, mm: Introduce try_alloc_pages() Date: Mon, 13 Jan 2025 18:19:16 -0800 Message-Id: <20250114021922.92609-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4987F40017 X-Stat-Signature: cnu94wxx4rtt9wk96g59qrsp44ifbjg8 X-Rspam-User: X-HE-Tag: 1736821173-321986 X-HE-Meta: U2FsdGVkX1+NQMrhV6rDoBujUbM0QBlQq320tv7LjWTaaItlytpIaSETpknyN23LcZSxHrr7k79kKT/GrTp41cDMypQwqhaNI37ktbX5WoJTB6L1Y7iz4J9VbZ5bD3KV3zHi/TJbn6xDrEltRW5tGuHsAVHx12vNI7amXonnpuUiZQOnfXtQwnvLZ+Ry/U4ojMdkOWod+w32rtHfS5dPstUd6rCOTdM+BOck3JhiantfbIR9Ko1mY4iHqwKPpdOON/qmV3JkfBflKofSMIpjE1Nxd1hURE0Ex8lmbjfrr2hbB+35JCx1bLQP9ZEj3EF1A+Ig/o4W2ZDRsKOGIc3vicrtmIOAbuQrIsmos5mFzh1D+nfBaiaIWxo4H4QOYMxaHhAmXl/7YmIbnBMOR5ZnDjlu7T9uXUOjw9KWA4ptpRdBWUHzfd06jxmlb1H1C9lyrr6q/SQur64eWSeSrQFpNpLq1yxCU50PlF+NlMNEqMePXRW88Ixx0KOZOd9ehgB6rZoxIKZNMn60b5BfzOENeHXsEehvHjYxSGmXxGyb8jxO3yWRlcUqko7aij3WbYlP6P58U4zVvewrmV9mJNZcO/FsbxXliqOh4/zjpzr9DYljL/amXfZtxx7wtYZwlzH4dXSFKwWg33ei/DFTfpADLh6y7woUNL3W/qfW5YYV8yiwAS0EtxY6+F4eC144e4qa6Yes5uQg6C4zIwhI0jWOdVopWnOTVzlrBb6g65oGpdsscpcJNHGmQ63sJlhAkOAFuUyzBjDE+VjBMskPbnzc2DUo4yd1PC/teSTf7e7j3sarVIzimKc35ZhsXiFEzKjbPQ6yc1gmTcJ6rQuoo3NEack2duOcON7GCycoPW1MoPttJnQ1taRjHPMM1p5Bn5bJL7E2sT6QXMq8NkDL1UzymWD+XtsjTH/A5I0S/1ZVluu5SBN/JJiCxzZdIaiXUt3P27ZNJWx8ckqTsNUCQSd GromFWnV 9TMn8lnE6xVM7X7k51/yAsgNNlbc7gi9e0Ht9ZucStAvPAkMZp95CmAtrQ7ORlOXESnyD8cytFY0aR8AvBI7a2FBKLu6SRW58tyxl8sLcs4Joue99ZVf6JnSCGnRzypbi5cf42qB/1cy2U/XOkU6uwWfioWSQCJCmOo+yYv2yrg7G2ek6QoRrzpdoiNllc4YsEE3FFxlbFg86FLkWj2qM9GtzuoWq2xODSAZHjnYbF5Slq5p0DuK69mI+QK2saWIebHqpcrIbkt4Al/3r1Kk9PLdzTeEro5JOPbzN7snzQ8g3ZgTxKE0ZAtPeWNCbcWB7AjT4R5yAUciGJiR0ZDt8AA7JGFFRVI3GX8UfUSuuVlVlw44ECzVRfs6dFiJlD4ygMII5z9B7n1bm/k9qQqchZYqq5IuFzfRhQCyY0kzHhW6rsPOVem8lavEZyRcTK7PEpkoMGSeYDJeUT38= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Hi All, The main motivation is to make alloc page and slab reentrant and remove bpf_mem_alloc. v3->v4: Addressed feedback from Michal and Shakeel: - GFP_TRYLOCK flag is gone. gfpflags_allow_spinning() is used instead. - Improved comments and commit logs. v3: https://lore.kernel.org/bpf/20241218030720.1602449-1-alexei.starovoitov@gmail.com/ v2->v3: To address the issues spotted by Sebastian, Vlastimil, Steven: - Made GFP_TRYLOCK internal to mm/internal.h try_alloc_pages() and free_pages_nolock() are the only interfaces. - Since spin_trylock() is not safe in RT from hard IRQ and NMI disable such usage in lock_trylock and in try_alloc_pages(). In such case free_pages_nolock() falls back to llist right away. - Process trylock_free_pages llist when preemptible. - Check for things like unaccepted memory and order <= 3 early. - Don't call into __alloc_pages_slowpath() at all. - Inspired by Vlastimil's struct local_tryirq_lock adopted it in local_lock_t. Extra 4 bytes in !RT in local_lock_t shouldn't affect any of the current local_lock_t users. This is patch 3. - Tested with bpf selftests in RT and !RT and realized how much more work is necessary on bpf side to play nice with RT. The urgency of this work got higher. The alternative is to convert bpf bits left and right to bpf_mem_alloc. v2: https://lore.kernel.org/bpf/20241210023936.46871-1-alexei.starovoitov@gmail.com/ v1->v2: - fixed buggy try_alloc_pages_noprof() in PREEMPT_RT. Thanks Peter. - optimize all paths by doing spin_trylock_irqsave() first and only then check for gfp_flags & __GFP_TRYLOCK. Then spin_lock_irqsave() if it's a regular mode. So new gfp flag will not add performance overhead. - patches 2-5 are new. They introduce lockless and/or trylock free_pages_nolock() and memcg support. So it's in usable shape for bpf in patch 6. v1: https://lore.kernel.org/bpf/20241116014854.55141-1-alexei.starovoitov@gmail.com/ Alexei Starovoitov (6): mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation mm, bpf: Introduce free_pages_nolock() locking/local_lock: Introduce local_trylock_irqsave() memcg: Use trylock to access memcg stock_lock. mm, bpf: Use memcg in try_alloc_pages(). bpf: Use try_alloc_pages() to allocate pages for bpf needs. include/linux/gfp.h | 23 ++++ include/linux/local_lock.h | 9 ++ include/linux/local_lock_internal.h | 76 +++++++++++-- include/linux/mm_types.h | 4 + include/linux/mmzone.h | 3 + kernel/bpf/syscall.c | 4 +- mm/internal.h | 1 + mm/memcontrol.c | 24 +++- mm/page_alloc.c | 170 ++++++++++++++++++++++++++-- 9 files changed, 290 insertions(+), 24 deletions(-)