From patchwork Tue Dec 10 02:39:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13900693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42CDAE7717D for ; Tue, 10 Dec 2024 02:39:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2BF66B00D9; Mon, 9 Dec 2024 21:39:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DB0D6B00DA; Mon, 9 Dec 2024 21:39:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A3B46B00DB; Mon, 9 Dec 2024 21:39:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6C7A86B00D9 for ; Mon, 9 Dec 2024 21:39:44 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EA9481601C0 for ; Tue, 10 Dec 2024 02:39:43 +0000 (UTC) X-FDA: 82877493174.23.679895D Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf22.hostedemail.com (Postfix) with ESMTP id 6CB81C0009 for ; Tue, 10 Dec 2024 02:39:19 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jpfTOBtF; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733798359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=3LLi8Dl+VMtnVMEMMd/PCpnnCqhn8N4DHFfLuFQFr6U=; b=D734+G1jQUFDV9KkJNJtyNrlHOrHnlnMDKdgAo0/G4s5hhCN2NGTmhVKpDnP1V7rrjC9W9 m4zQ7or5dNelr3gzHF5J6ZcC+jAA19pxSp49q7J7NI0IuCtDjaIAQvlXnDOizmWQfGWD43 29blWEZB/axvodLQXnVgknGB0MsWDOk= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jpfTOBtF; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733798359; a=rsa-sha256; cv=none; b=si/zS/8WhRKbvvNsR4ksMGUUS4xWqBIw0Xapg46Yh6OOTsQgr9AcdRBGxNaYyZBXfpdosG Hdzc4lLwTUUkNmGSC3W+uftYr4TxsHwfs+v+3ioYljJFVzTcjoQz5sgRSCpuT5stObLPdn acLJVwwXWhGvjlye6zIu90s8KMdgdwI= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21631789fcdso14456305ad.1 for ; Mon, 09 Dec 2024 18:39:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733798381; x=1734403181; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=3LLi8Dl+VMtnVMEMMd/PCpnnCqhn8N4DHFfLuFQFr6U=; b=jpfTOBtFV1xkvZsx4FL8ZcjCm698shBteM8XsKS9qrChSsrOZ/pYN1gtI3FJJKyuR1 YGgue8jhdMZ1rdgzTsiFhueo5EbJz73oRlOyAYO8utqQ25lQY7xJOqgPnd3t3pl+Qhpf WFlvetSp3/AubSCu925tbFNI0e9q01LpwEHAZeqq5f+1BsKgLZSAvyye5oPPPjw9VJGH J3cJUg7XENYvxvqy96M36OnisxMB8Rtdh4SqUNIhwPuCAcoW+fYdWHoyDXdkpMuwL7tN +ocU/fohUaayClyUhC/RjtOsij9j9nYfSDmNWXhMIWmqHVW26ssC20LjTTwK+pw7EmyK MDWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733798381; x=1734403181; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3LLi8Dl+VMtnVMEMMd/PCpnnCqhn8N4DHFfLuFQFr6U=; b=LmqvIDhbAqRaMJp9KkynSR/VycEuE/iJLPykNsw3TfqhKuJF7OZ95pfKibFLrpPM/9 iV71QsUVldayX9X45d88v5bo+nQ39rb+eAsxyBUGSkRP+s27RT5NC5uJry/82ZoMRacH UqrMB0vMev4/KE19pxc4HvI+ZTbfhe7KqC5pXrWt10VqONZqOkTxyJ/n7PzNTmgCR2xr 4MT6/NHAD1RPX5juMjFElljgMl02sV9ldlqNWx/3CHDDG2q816zWwfOmRtvY35IXyoIy JnNj02Oh/cmR7lcXUuhQJd/SMaAACuRUDJ6sXfPwy4fgfrRBwrfNj9NXf7AP+965ZD9O WOgw== X-Forwarded-Encrypted: i=1; AJvYcCXuJvadPF2j+DVr1dT7zshpGchTZWofilXmZyCwamcXfkl1LenToocrZy1Pl5F2EvNQ5WJltWz1iA==@kvack.org X-Gm-Message-State: AOJu0YyIp0XhObCc7HetnBA9WO+ag45z3NRmYp7LnJhv4OfpTEaReTWz 7geEsCNpWnv79w1OxAJjxMaVaF6LV5Xt9n2Hnwi/JqMLyDHX5ujq X-Gm-Gg: ASbGncvGMKYOHCpJL1DDB+DjCt2GAnEYcJUngU/wWTiDJu3JrlA6xfGST/QtXLh2ec/ GaDqq9G0s3IAZ8YGLtc+/VilAmstudov57yN6eKFRNYS0/DVZkU7v/byqAoOvMpDl92dBmcCJo8 6Sp7IeDIdBOIg1kt8tyoHwon6lL+SgqCq5jOg0ZE6W+flcr1rxeI1nWbCqvaqXtmYyt5tdU98xK TGHEtpLMZ/xdParOBX8TwbPxuT6XpmtHixFKu3azeweiuorLnGWEmRT10qTJtQ1331pfH6CJHnf dgtW1A== X-Google-Smtp-Source: AGHT+IEI7Az27IsEgQkhAQk9YZAyRRzn6cHjP5COEJuaiwQiuLYgIM3ARR/xQ4q/YDKLrDtzp5BQaQ== X-Received: by 2002:a17:902:f610:b0:215:4f99:4ef5 with SMTP id d9443c01a7336-21670a76c4amr23554425ad.28.1733798380660; Mon, 09 Dec 2024 18:39:40 -0800 (PST) Received: from localhost.localdomain ([2620:10d:c090:400::5:83b0]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2161b77ea74sm60455935ad.229.2024.12.09.18.39.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 09 Dec 2024 18:39:40 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v2 0/6] bpf, mm: Introduce __GFP_TRYLOCK Date: Mon, 9 Dec 2024 18:39:30 -0800 Message-Id: <20241210023936.46871-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6CB81C0009 X-Rspam-User: X-Stat-Signature: tmhwmiqwehjrhk4pge4yjjaz5g1ch355 X-HE-Tag: 1733798359-754559 X-HE-Meta: U2FsdGVkX19fEsinadHQB4zdtZBd4iAjOCL3MOMt9/+ovDpbMC6F8331djnla3c+fAD5hH2sRFnk0cACw1YCZ9dq8MN3n2JUBrR2ZT7hEo626XiUWuule+XMWUDSUm5wSnSHz9CHfXT5YyCtLveFPSpB09gI6+2I7RIU7DZjkSZ0w/gNAl536pp8w+8i6XpaoDm2g6CQ8VfJ1LpyQuuJ9BcpcIk/1qMYvYMwH+hqo8Y7syLZmE6T4lDQDH2N1+aqk/vmZYib2/cB5D5JJVCzsptKUSaoqn30JcuhRkj8/FapeSvzLaEt5rGgm4iyWeI1jyUaN1NH7Si4k2sUlRgDrGki5UZ30ga37ozuiXpDoZ4mPyOvgNWuDHTOxC1qJ9G/jYhwi6W77kHEJ5JaN4fOBhC0HPAizysm+ZzXqx9g9zwLOaFwdazQ2t/j8SjFe9NnR/9//egt2i/ZeaDJ01+ru4w7Zhbl80vjNJ0x8EGW8alExZxluCGoVWtEfULGh/6CzEqzn/YQ9Vk6wHoOvVFauZ68U7I9q6hQS4FigvzKHAEfrwvziyHH1eQqDY581OCAHXE9Y47De/VkHO8/r9QPce+DH62KgizRwTbKb7aS2yekW5gMVVge8Mx9eMvYBv6AbaoDd8zch4THADui7ykhCPM5RhS+xam5kjpglozzwcVL/eParti11xX94KfmBNpmcuF5HCuxNBtO0y+153rHF7Twju8fbdJQUVVipV3M/CLiNpqxQUV8E/RCD0jYHP8HPVe45CbftNinDz90ztJmeUQrBBKFKmKeSM/r40Qbb4j0VI2do161vuFdrAadEtk5jZNpGDL6l2LAzfqvNoTqvBA8T0Beh6mIXwKi+w/qUjqwWj5qxslb4FQNqy21r5gMqIXEzEel/Xuvv2cKN5zVOnzW+qBn2hEkAXosvnfVuh1jTL79BK9rbvYRAodbCIqMBMezYocuuc75+fuPoYA q3wwPMij bQVCghcyZeYe9WY2gAFAzs8xH7gbBo74VvqMEgDHErW1PpsYvYnchX/eBWxB7PEkSxh6+Xk4m5g2XGjV7djLYElQ3dZVLfCPlRJg1FJdilUfFiAgD21V5ku6Vrann7BL0HhIOaATXp0d8NS0YmmdHSFu/yYLNW7yQcDbOdRM9fJLx2edSku0jG6LjdSTu+bz49gucLZFIGqiubkczM5eb9euRAEu2qB7ZLOZWUzkKFoWsYlP4dM7fneZGbpafivf2oecYx1asb6BjPypEje66sbIbNbEgJKUOyzRVRijSfb7p/xcO+x9hTG46Y7KxVWKTpuPfr5yU4VysNAcICXA6/+8OyoqA9C+mkKvj1zr+dZuzC9B1saS9hx+N7g8RYVv7mh3oFSJGwF1rEfGp040A21gxuTBjwz5PHacY/czrriqhck1XEKTsGrSaKm0nt4s74yx1IHUvluLmYZ4W96f0VPy2boUeSuDXxIYyeODpess9m+I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.023806, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Hi All, This is a more complete patch set that introduces __GFP_TRYLOCK for opportunistic page allocation and lockless page freeing. It's usable for bpf as-is. The main motivation is to remove bpf_mem_alloc and make alloc page and slab reentrant. These patch set is a first step. Once try_alloc_pages() is available new_slab() can be converted to it and the rest of kmalloc/slab_alloc. I started hacking kmalloc() to replace bpf_mem_alloc() completely, but ___slab_alloc() is quite complex to convert to trylock. Mainly deactivate_slab part. It cannot fail, but when only trylock is available I'm running out of ideas. So far I'm thinking to limit it to: - USE_LOCKLESS_FAST_PATH Which would mean that we would need to keep bpf_mem_alloc only for RT :( - slab->flags & __CMPXCHG_DOUBLE, because various debugs cannot work in trylock mode. bit slab_lock() cannot be made to work with trylock either. - simple kasan poison/unposion, since kasan_kmalloc and kasan_slab_free are too fancy with their own locks. v1->v2: - fixed buggy try_alloc_pages_noprof() in PREEMPT_RT. Thanks Peter. - optimize all paths by doing spin_trylock_irqsave() first and only then check for gfp_flags & __GFP_TRYLOCK. Then spin_lock_irqsave() if it's a regular mode. So new gfp flag will not add performance overhead. - patches 2-5 are new. They introduce lockless and/or trylock free_pages_nolock() and memcg support. So it's in usable shape for bpf in patch 6. v1: https://lore.kernel.org/bpf/20241116014854.55141-1-alexei.starovoitov@gmail.com/ Alexei Starovoitov (6): mm, bpf: Introduce __GFP_TRYLOCK for opportunistic page allocation mm, bpf: Introduce free_pages_nolock() locking/local_lock: Introduce local_trylock_irqsave() memcg: Add __GFP_TRYLOCK support. mm, bpf: Use __GFP_ACCOUNT in try_alloc_pages(). bpf: Use try_alloc_pages() to allocate pages for bpf needs. include/linux/gfp.h | 25 ++++++++ include/linux/gfp_types.h | 3 + include/linux/local_lock.h | 9 +++ include/linux/local_lock_internal.h | 23 +++++++ include/linux/mm_types.h | 4 ++ include/linux/mmzone.h | 3 + include/trace/events/mmflags.h | 1 + kernel/bpf/syscall.c | 4 +- mm/fail_page_alloc.c | 6 ++ mm/internal.h | 1 + mm/memcontrol.c | 21 +++++-- mm/page_alloc.c | 94 +++++++++++++++++++++++++---- tools/perf/builtin-kmem.c | 1 + 13 files changed, 177 insertions(+), 18 deletions(-)