From patchwork Fri Feb 9 04:05:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70AA2C48297 for ; Fri, 9 Feb 2024 04:06:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08C6D6B0078; Thu, 8 Feb 2024 23:06:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 015096B007E; Thu, 8 Feb 2024 23:06:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFA306B0080; Thu, 8 Feb 2024 23:06:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C73BE6B0078 for ; Thu, 8 Feb 2024 23:06:20 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 70B4C16016D for ; Fri, 9 Feb 2024 04:06:20 +0000 (UTC) X-FDA: 81770928120.15.9466682 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf29.hostedemail.com (Postfix) with ESMTP id 9ED1512000E for ; Fri, 9 Feb 2024 04:06:18 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ePLHTjce; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451578; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HiZXvAtRKK8jkFx0ziwixfKGMxiAC5XovXkV04D0O98=; b=BggN3NeaphuQzqYCPUcpBCNwH+kZXpXG6WS4Gs/CgTGFyFpyFVTk/NZ5lLlQX5Iinw4Q/3 rf9vgIWTqoadrICBKRMbNrWMY/BcTunvoGuEDYjGVSWv6GBuWAxmkJCM7nYIt1er5yiS2l jfw6yoxEMwbsCcDYZlNxGFu5+KWU+YY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ePLHTjce; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451578; a=rsa-sha256; cv=none; b=RwQC6P4j1JBy2DuNcjy88mHogwIEOyZg9H8q3NgAoelZMONlyfV00SF+cCeNX3Obcux7VK 205n6olBta4eZnVPQ9aNi0QCIzBEKgoOuWJnH9xeWyG2gO9H9HjaGyKoyqkrSGSNENI/FL +VUZUR2k0KNmhSrkVpUMwHjPcuwOOQc= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1d72f71f222so4490505ad.1 for ; Thu, 08 Feb 2024 20:06:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451577; x=1708056377; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HiZXvAtRKK8jkFx0ziwixfKGMxiAC5XovXkV04D0O98=; b=ePLHTjceC5eeNhevJc8+sDXcBgwJ0bJTH2aZUqpGMcdSOFbgjbDLZtK87FCjyYCvQy +FeY8ttxdwSigFXaQugfmW76VJf3557sGYA9Quh5c6aQdOq5IXlqk3vaoJNRtt5Y4jq8 FalyBQkYs2dCmlEtlx6D5stqa7GX6h/zlUGBzBXssiRMx++zAzgkoRjN93EBmlLtEH6v BN6JAiXI/LpEDhndAuxy+rkwpJCZhrIvu4nX9JTfA9nj6VKqFPSJFBKcm+HIzfwy6ED7 S2NTzc+33hzqPfUi8oaDK19rwMjN7GyOrcu1gD1H0nV7PBWsEwjt0PDPcYS38wrahI5L JbqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451577; x=1708056377; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HiZXvAtRKK8jkFx0ziwixfKGMxiAC5XovXkV04D0O98=; b=Qmj/whvmKaZMG1nsOOgDlSJkJv3RvOeh1Zdc3FHonDP263Yyp0cU/HVhsVMUS34SGg MINSh2JfP7eC7u6eb/pGp7JAzvcAmfIzqqMA3eSQDgNKj/M1GsIs7FuJMOzMYG4uXIkS U8EwMDgUysP8rmG7N4saxYrkjArpcGZgcHb1McrJRz/Ufq0zW8mFiIyXtksFj/NLwNt1 oG1c9XRx9xZO1OmuF9bbRml1SYoKHPssv+Q8ZGNJ2ZShEnYXie0K1O0Xeta571qtRq0O 1VyAhDvSHvbewAu0Oc2/rwcHRJYBd6ttah1+n9CGucOVB+mG5JmwvI3QMmwghOYsgcUp iYRQ== X-Gm-Message-State: AOJu0YyXl1btnRk234dRCDy/48HS2Y5FLoE9awb2jq25xyxd8wLvmZLn HPK/mkntxdBDTDx148FcGOlHZ1RPzJHH2kjXfNQxgCAYw96jAs5b X-Google-Smtp-Source: AGHT+IH0qd7DUNWL2d1Urm49Tarwt8Y7vlvkifg1t+rPqEzlKKds4sNBaEDojklBlzOW65EyuN84oQ== X-Received: by 2002:a17:902:d512:b0:1d9:cc68:19c6 with SMTP id b18-20020a170902d51200b001d9cc6819c6mr505956plg.43.1707451577482; Thu, 08 Feb 2024 20:06:17 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCW0PSDSVSrkv+MnkyI+/hIYocwwt2ruGu8w0QAeQbBnRSNdSiWSEkbUIE0BAXyb5Zg9Ii4TSozKKj1jxxKcrHkZkTO0DaUSIlKkZtOh6JNQ36oXTkCUzMAX4yMunEMiwp3ti9w3FRrGlWp5IUZloJgZyY9JqybOu8LCa4fyoIMPi3oxP7rXgATE/KrV1IYoYtHNxPVCOE9ctb5w8jhL8M+66f6h0UCln/TA1zrEC55iEka/PM0Pl2LQsdp714N5FuoTkqyOpR7eEsHj8Kl1nmymCmhFm2o7SJ3zpjLN6vSGYhbm6ybyG83M8i1dSN/5Sz7UClBMTTVQu6Yl7l4iU8Sx1HbZa4/un5ZV0lD5c226Y0vpra+y0w== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id mf3-20020a170902fc8300b001d9edac54b2sm544084plb.205.2024.02.08.20.06.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:17 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 01/20] bpf: Allow kfuncs return 'void *' Date: Thu, 8 Feb 2024 20:05:49 -0800 Message-Id: <20240209040608.98927-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9ED1512000E X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: j1utkywicpd1m4itxt7hezejjgq5cdha X-HE-Tag: 1707451578-272578 X-HE-Meta: U2FsdGVkX1+7ksV2bsu1f7hxq5ycx1cCDmZljdM5ADPLfAxeLQ8dYdGFMUPgq3vAU7+3wKKPE+SpM6i8pMxngGIhL18UADmlnZMxr09aYYMtPWkxPUfFerr0VEwlOTclFSr7OIDFZKAl7J9HMrrz/BI+0LANTiiHuViXrbKZVLob6UUSvD8EpGQR9DUwcdzCk0ln+k7pBDGC/eQPNejog7V43XiOemxfHPbGU5n8koxRGyIAzax1DvlPAwdjH1smzBv4fLX848EkWsXpaI0HsCyOH9/SdTiunJjBNUNPPo3+12VP1BQsDMUN5m1wv52KeIKXK3yxzFHsLfn9u6itD1r6gPiM2qsAIBhdqLnd5ov4gR25lnxlCTAU2sk9uB4Y1sEKBqVgF7LrfIZTaeuEcuLkxzXKxo3Fmny2RFn99zJ7R3ePcgBJ7Su4IPoan1NPqX+vEYn+N+rY0sMbv/lSDNlWHa2DfmHcSXO5xKhyTGMY1D+5cEQP2CwLIy5AHJklV7/aR41ubp5ImltOjxXh+pG9XwqVt4SUNP5WO0iixXooHILDk1vgHzo5tkuC569yruE9W2d/o7kO1R+sS6KKzCqUWMETo0gEwjDlAfb3hA9A7tbpmsyllZXPDiOiY7CNtOlF2AGt6f0QBEhqfGggcGAJ10tBoMTS8VQuLM6kvltXUXM8DI4zpFC4v2+nCxcN3wzLMhGfY1CjB6Jw55kFIVMoMk/ldVRsZmPq1pIwY7Mv6WHPKvnBwwmFLZVpSTvqeifNU8+BDhxSVRR+nd84DiM1bh53TzQvZ+dMhONvn4dFnxnjYGOMZC6y9So6AAiATRUo+qedZyKH4Ddwht728ntG720VTvb+ctjRs0FPjMdC7wbThn4/t3J6tPLhnDsASAc+H3Xj49UaKKfnrW939vA2VzuX02jjkl7eK16LtVXVDcHu96ZqCMkntsISFHmn8nRUPKEwDcIvxnEac2Y C7Bg/SVX 8turES+L9l+lxFN/afbH9zVi0IzuDIQT9Rg/+aISV/nziVuSWGUU3Oe3fT36gkhIEnNDptxDlnqKuPt7Ie1DkrC3CbgUMmY0xAc+KOApJIh8pqijC84j0CVNps0u4RFP53Ab+wiww10KkZ+DFN5BAwitdA7c/JA+ex0f4Mo4xCp3WfdxE8fhcj8UEV5+J6pDp4eJwhnPfqWKiQGykj9PDOTTZUbMgvLb/jsdp0RDjzuOyI3Au5ra2cUrqZTOGbGWn232AA1Wp1jEvIsQCeKnZavfZWvfln1N5sCoffbHVW7IMd9+Cza+DPriZYDbPOWA3d/Xs+mjNiGYHmcFRhWHsu+ZY87P4QfUtySX/xjpKHS1yftXk5foPdSwdEgAG3jVuyeAhe8TOSbb1WDzC3k97IjsAWI5GKKesN65SZBoyN0IT6Jae8it+V2JduJxWG2FpshFC0RK44teSTK77wO6UOOQgXjtdj6bWdPMCpFx1YmmXmiAZ1DMadlZeOxLA0jCimhPoyNzC5+VpEZA8F3ahIHkSvK5AF1KiZO++w9gd54g9l/w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Recognize return of 'void *' from kfunc as returning unknown scalar. Acked-by: Andrii Nakryiko Signed-off-by: Alexei Starovoitov Acked-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ddaf09db1175..d9c2dbb3939f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12353,6 +12353,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, meta.func_name); return -EFAULT; } + } else if (btf_type_is_void(ptr_type)) { + /* kfunc returning 'void *' is equivalent to returning scalar */ + mark_reg_unknown(env, regs, BPF_REG_0); } else if (!__btf_type_is_struct(ptr_type)) { if (!meta.r0_size) { __u32 sz; From patchwork Fri Feb 9 04:05:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F738C4828F for ; Fri, 9 Feb 2024 04:06:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ED596B0080; Thu, 8 Feb 2024 23:06:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 176706B0081; Thu, 8 Feb 2024 23:06:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F31156B0082; Thu, 8 Feb 2024 23:06:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DB14A6B0080 for ; Thu, 8 Feb 2024 23:06:24 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 853791C19C5 for ; Fri, 9 Feb 2024 04:06:24 +0000 (UTC) X-FDA: 81770928288.25.F5D34CC Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf25.hostedemail.com (Postfix) with ESMTP id C44AFA0012 for ; Fri, 9 Feb 2024 04:06:22 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HwuxSRfr; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451582; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RYMIDAZLgBUgMpP99zrxeTSMtNJZxQwPNyQYFWZYzso=; b=XNYaj9T+wc0VmXETsrJP4MHQnm1F/A4sNqr0AAzcHakTCuj4hnH30fwzeoi461Z2rtMUPk eB/h/Uyf+NudBAVX3WbWHtixYsOx1VcNVnw9LSgwUv1K0P3fQIG6z8lUFxl1G9NL0JKLOe bxdmw+dhle75MI2v533ZPF8lUk7pEWk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451582; a=rsa-sha256; cv=none; b=crmhwvIgpYJi0A9Nhz6QK8KXaj/n/FEvPuaeD+25Snn7cOyt7iw2D6G2QUry1fXgXlost7 exQo60LLU2AcpKuxBPga81sXsSwuxp4kUoG+Npm5CDbL1QPuH32adOeldQonUYAV19++wH O6upJmjsAiF4IQTia+2DO4PG0y4g6Sw= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HwuxSRfr; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d9bd8fa49eso4309415ad.1 for ; Thu, 08 Feb 2024 20:06:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451582; x=1708056382; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RYMIDAZLgBUgMpP99zrxeTSMtNJZxQwPNyQYFWZYzso=; b=HwuxSRfrEXp+kQNpXeoVKMN3yYJuenoQ8eYcnCq0ciw5I4VzrI4ujzeDG+bLaJLyb3 77GCN1hkJdE1J1H1ToXJHuXHPDil+lYK3zICyT1kwXTCE33fp7KLbMyQ0fs19vwKtcE7 7G1v2I6sGpnP1e9b/Wt8P+nBDxSKbe2FIh/vKAvAQM8WIs0210YJuA3/iASnW1aBYEIj tILl1blZzW2AzjQIxQcvpwiVD50O3mawZnzE8hyqR5z8EevIrsnRHhwGQnUlUq/aPDJF f0MCaXY2TyXqnMhs4CGkw90C5JNsXlqOIs5TlXh55zYckU/6yRZ/u1hEWMRxOHNqvRAc lcpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451582; x=1708056382; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RYMIDAZLgBUgMpP99zrxeTSMtNJZxQwPNyQYFWZYzso=; b=Lbmm8N/kHMoQk8p2aics+v03K4Xgi/4cp7iwmMSOEARLndQr5gJo16Kcdsrz/WbmHk CjOgessNEwVkt5TvDymt3hM5KOKej6k7UrJmcXTFuUXMuyHSrAaArn6310wzgrjs6WH/ r9RpjzPMQHVLUT8FgiJZTW01vggvCppmHyGtJfWQiuI730W11i2rp9HlzA+PoO0nnvez hGVbFCy4XF4KObWbByYchgXsDqVX0KfOvDnfhFkBX1cdupcf1D8V0sL1NEE4MuSKS7G2 fDdqPMPZRNvFUm5DUXHmK+CEalMWeoYhGOScy7lBhWienA3ex0rynkUKFud6LbO0GIOa GG2A== X-Forwarded-Encrypted: i=1; AJvYcCWZ8MRb3PCEKrsqyS6c8OQgaJW1mBkbKwq8SDB3thv/AiimBgw5DIdHhn2+M01XycmqPhcg5NsWjUMY/ay7pScIlGs= X-Gm-Message-State: AOJu0YyjWJuKoZ1cpWsMWXEGmNGe/qyk5/xd+6BtobnELtya4dOcvhq+ kHhj31grhgADGngAz0E04lqSojEDgn/MA0AAW9NtsYnpzd65t6Vyxrljpqab X-Google-Smtp-Source: AGHT+IG7W4w6iclNa9lFghGZkrapWzAeDlADw31sucf55LWFs70wKngA8+n/+OS3WjIjVQMSK9dgJA== X-Received: by 2002:a17:902:e549:b0:1d9:6dc8:b44f with SMTP id n9-20020a170902e54900b001d96dc8b44fmr349803plf.1.1707451581624; Thu, 08 Feb 2024 20:06:21 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCU9TUB8D0L6Glwm5xlI3JvnBt7zEVn8IOywhbt58OfQBsWE4k4qMGs4CWoKHxyxuTzW1lD9onprEAHbY/BQVbHwXNg9VmhIZInBkRh8ifJLD08aWjb3T+K0tDAVFrboxdJ+d8n2YA52cdz0aPx0veAq9E4y39FA4QYAbouAb2omG9KISCOMfSfKL6a8oDHvwuifT9JSGK4q6WX0S4EqCjbHGubv+u7IqGmSdLK7SIiYC28SWmhCj/ydGdJrKud4QkLYUh1W+4dZWd/d9cSw1xwgy0LsiufRVZBXcgKL3wjIvYg5eJlRcFjB7OjTKhqUSIarEy3JRhCDPQNO/jtaRfHoaboeoMjjcVd7WbWx0EEabK2p/ysZuw== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id ku6-20020a170903288600b001d8f82f90ccsm538134plb.199.2024.02.08.20.06.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:21 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 02/20] bpf: Recognize '__map' suffix in kfunc arguments Date: Thu, 8 Feb 2024 20:05:50 -0800 Message-Id: <20240209040608.98927-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: w13jtyir5b5gojmxz5ytz51q8k16jruj X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C44AFA0012 X-Rspam-User: X-HE-Tag: 1707451582-700297 X-HE-Meta: U2FsdGVkX19CColYlHLLqOQwUNqQamQkJTxpvqfLCOZZ6InHOzMUhkpHIl68wKcaz1S1oJFv4Yj6WwHdodTEqJxRTNq7rIZxygsi8WfKJEAmHjlzKfNJJbt/3+lCsRUBBWAF8Fw3M6eQ6FXy80KcHDk1azdKwIBMfWaJsXPNCRASfLzTNauN+3hL+/SecHQ9vAZ0htxa0PDPfzPSfZZ03ohWqR8krjRsGSTl+/FSEAFaiF/zpMLGAWB+gs3XAh7S/WexIIJ1mhcWxIp+IRBnDbE1D5IRX6GNrqkAcaUEeNeBUcpeCzcuJGNFEcw/a1pFTEKxiCxbj/anRq6o1MS/o4wWqAZOgmhEvqJup74Ppb1rzauyQ9oPBU2GfO7+9hLAkHuqiUBHw89o0XhSSm84jigWCAqSjJGxF9dnUTDDxt2Q8b9fmM+i3vHenT+nISoC1ARfXcQvKorE1v+1c2v/IQm4SVorLJneQT59lMHighdlNJpSKmNXBHNJIVHqWHZnVL+c9GpXv1+HnGzu4kDA7Dpk8XlNy6QiHD9ZHUTosdc3M6dp4EtBZ/i+1JxpG8Aeo+UNqsSsBBnhGG8Uy+WpHUOetKemMamVkGV8fvfU86syppJwKOxof+9rIbVTedOClQAfC4t/+pbCmD7eRkbve2kqV8dHO8VXwubrLZCinBz1Ukux9VfHIyPcYcj9UARxLk/rUQ/O+0IvBsdNv1doTDek/pk9Lz1lnbBnBpUVAtqtHK6w7SkdIXb5sBCrzoJynmKyd0dtQfHDXn6h4yALEmiFQd7LkoBc4DQXRCRwqJaguhqdiMuG59lNBNOG3jTMbXRwmrbwL+cf6QOmUq/KyEuRd1zdJ00wAOv5H/98PsDoHOj12y8IVdz2mnDDFCkKZNHsFAcei1GY7IIbhJ/mozOdvneg4qGaqadGJcCHHfsemYT/iOE+Waw4wA9OJHTewUh+lQJp3z1vmvEwu43 WiLuHsmq BB6FUJ3WmCMdbNTNoej5xCxKztAdnVOi6v+/xCP8uHPeGEuvMh/FZrEjT2mkDWqPIUscshDU5TUcgXHh64ruBd4iPFMhiPRU5j1N3tnKZUGL4DyuhnYMPjY06s88RkqWRJnjnMNW3ZxEiUjuSxBRx/b51JJmuU16D0PL91/Lx2zdaDSwvgwbrfvvCps6Q8H959Sjw X-Bogosity: Ham, tests=bogofilter, spamicity=0.010018, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Recognize 'void *p__map' kfunc argument as 'struct bpf_map *p__map'. It allows kfunc to have 'void *' argument for maps, since bpf progs will call them as: struct { __uint(type, BPF_MAP_TYPE_ARENA); ... } arena SEC(".maps"); bpf_kfunc_with_map(... &arena ...); Underneath libbpf will load CONST_PTR_TO_MAP into the register via ld_imm64 insn. If kfunc was defined with 'struct bpf_map *' it would pass the verifier, but bpf prog would need to use '(void *)&arena'. Which is not clean. Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d9c2dbb3939f..db569ce89fb1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10741,6 +10741,11 @@ static bool is_kfunc_arg_ignore(const struct btf *btf, const struct btf_param *a return __kfunc_param_match_suffix(btf, arg, "__ign"); } +static bool is_kfunc_arg_map(const struct btf *btf, const struct btf_param *arg) +{ + return __kfunc_param_match_suffix(btf, arg, "__map"); +} + static bool is_kfunc_arg_alloc_obj(const struct btf *btf, const struct btf_param *arg) { return __kfunc_param_match_suffix(btf, arg, "__alloc"); @@ -11064,7 +11069,7 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, return KF_ARG_PTR_TO_CONST_STR; if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { - if (!btf_type_is_struct(ref_t)) { + if (!btf_type_is_struct(ref_t) && !btf_type_is_void(ref_t)) { verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", meta->func_name, argno, btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -11660,6 +11665,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (kf_arg_type < 0) return kf_arg_type; + if (is_kfunc_arg_map(btf, &args[i])) { + /* If argument has '__map' suffix expect 'struct bpf_map *' */ + ref_id = *reg2btf_ids[CONST_PTR_TO_MAP]; + ref_t = btf_type_by_id(btf_vmlinux, ref_id); + ref_tname = btf_name_by_offset(btf, ref_t->name_off); + } + switch (kf_arg_type) { case KF_ARG_PTR_TO_NULL: continue; From patchwork Fri Feb 9 04:05:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89924C48297 for ; Fri, 9 Feb 2024 04:06:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 152346B0082; Thu, 8 Feb 2024 23:06:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E20C6B0083; Thu, 8 Feb 2024 23:06:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E94F76B0085; Thu, 8 Feb 2024 23:06:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D2B8C6B0082 for ; Thu, 8 Feb 2024 23:06:28 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8DB1AC0291 for ; Fri, 9 Feb 2024 04:06:28 +0000 (UTC) X-FDA: 81770928456.03.2CB779F Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf02.hostedemail.com (Postfix) with ESMTP id BBBCA80013 for ; Fri, 9 Feb 2024 04:06:26 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LxrBm5w1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6BMTbGEmBryFuIGcZHloqpvwHOH0k2D5DfDjYGSWlIg=; b=OHi/Tr/HRrJB7qCNiOhjj7fWS3cru98w6O3gFfsr7s4e7cQD9HGTqmkm+nSkEJM2+pZbqD UPxlH82zv6K0SC0lzSgjHXG6eLRVw9M+T3iUmv3YS/VlkoJWIeqGNxy1wImGdj1I/k/Yln 7CERASEq/t09UvYB7POC5xtIOlVm8QQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LxrBm5w1; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451586; a=rsa-sha256; cv=none; b=79YeVPvGnk88eL2C2WoZlYEO9hT1ktBFRRR0v9kDSM+Q2ajl1MHXM/dgmkun/iNW/+I3I6 M/88YC2zdEHtQI+kHCWQXaMBb3+903/jnju0ihVnPoSHTfnVwBatRRXAYg+UaiLUT/BJbU lZTGBJGpmtDMN4YMCkZCAHwvwyvqlCA= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1d8ef977f1eso4579015ad.0 for ; Thu, 08 Feb 2024 20:06:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451585; x=1708056385; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6BMTbGEmBryFuIGcZHloqpvwHOH0k2D5DfDjYGSWlIg=; b=LxrBm5w1BL65KktdysnXPhif0cH7DvqEWvLipSmVVvECNcQnZMAMOUE0PwT/oJUf/N xlamza0a+dSwrpyU+ZUztyk5WsZCcNTIl29BS5e8fzrVmKHnIRQ+zNA5zg8NuuiSOjPJ 4JFAuGX1yEs4VOxBpFCpekhNZLmP6/7gLlRvr/NngSWzDDApRUni68t+vqzjVHkIWVgx JKIh0Nl9/DYDnOwMeF5fj2xwazQt69Q/QfIIQeyKG7MuVm2XbGVBBbpNEy2TMrjJt9BJ l1BrCpgi9W1uyBNW2LgLSqhWFqCGzBPCX6uFfJ0m0Rj2QHanT27kHKskJTUOf/Lh+3LU Jxzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451585; x=1708056385; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6BMTbGEmBryFuIGcZHloqpvwHOH0k2D5DfDjYGSWlIg=; b=ry4F/NSjzQfk+y92v4Aq3ilHrMYqO9JUumNWz6htlUGgU+5DQ2p8whmBUhwPaf2Ln2 RYR0awiwPoP3apKTOb+GN0lCZZfMcPphSpzrMPS+AHCrohfSKPR0Wf3YWqUC4duEK1pN nNyQtkP9JDPM8CYIzbEhpLiUmbPcIBUn5DRIb7y1ZAHO5J3WIX8E/jLzl8nu1iQ2ghPR 6R/pq7+NFpUHtgfv9Z5iWBAKmglqYzvWe6nQCu9CQFetU8Kqeuh0z6OzAsHm1rP4SKBv XwJIsIMPm3iHfjQYC9KR6Tpe16+p78W21ECUIrQl4WxVah8eI3Wc8Z6cciGcBJ5YO48u WKRg== X-Gm-Message-State: AOJu0YyDjWnzpSNqMipnOgzyhf4XGrZo7uK/bKcjHGqs4+7yRVVumCSD 8GXDjG/EazmYTq2fcbEkS6A44J7FarbFD3zSFslAMaJ8RfRHQ2wK X-Google-Smtp-Source: AGHT+IF5Mfq/bPHfBQYoGGtl+p8BmK45T13+kVLjl6HOkUeWFmPC1hzclwCAAHXTr58YrlBe4W+2VA== X-Received: by 2002:a17:903:40ca:b0:1d9:5f11:d018 with SMTP id t10-20020a17090340ca00b001d95f11d018mr520921pld.1.1707451585566; Thu, 08 Feb 2024 20:06:25 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCU+VQojeGmZnI/6ZKbMA4QjC6Z7cs8p2E3pelD8tmw/+rxHsyJp+sHxsISw306bbIwagnzXrqksuerDCdyJyzHnkQ8d3DIm2mi1k1zoOdJMP/wTNoNNR5ir3f4CT7VEYkubNEGmEnkgZRPf9eGmcjXs9cZMp9inOKRZV5yXU3IAzhgmqi2+d2s8sfmHKBYfLJ0cTlhfgPzGfXc6HritoDUz+EJpKOETMg/w/f5tKSmrToubP2DeHNnK+Nb6neWYKAOjYI7Nbtm4Q637JP9enCDVLsNvdrPpk40p+0X64NQlSkbO6KUf1lMsM+50n70zWLAH4TkyqwOFm/a4C0sXyAEf8d77rzbEMPOXScNRvVSW7DsnXrjq3Q== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id ki11-20020a170903068b00b001d937bc5602sm544378plb.227.2024.02.08.20.06.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:25 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 03/20] bpf: Plumb get_unmapped_area() callback into bpf_map_ops Date: Thu, 8 Feb 2024 20:05:51 -0800 Message-Id: <20240209040608.98927-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BBBCA80013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gcfoh1j7j336shmmttte4spuy7r4teap X-HE-Tag: 1707451586-616659 X-HE-Meta: U2FsdGVkX182MFdoYyRCiiM5Oygi7D9qdxy7Vl6RMyimfBLcKkN4gk6Rq/RqT9M0FJ2ebKrjm8vIB063JUA03zSaFvY9iDU4ih5X7Z/vzlq4Wvw6PpZjf4mSHf4UqvFltQRCQ8ZjtBiaod7vYonr4CCI8VQH3vmCUDFdviuR5WtpylNDOWdLeaZFGcrXh3FLIGps4fmbth5+XSBun7RzK6ze9BKa5P776Nv9CUmQllmaPIAPXlWTnKR35Z+/HNaRXIxLTey8fUKieTK4BMOid+oViGQXFN3ruuFBqTDmDzneptVYz6TjejkE3vfXKWNO8nMDQy5KyrFYKgi/vBHiqC3RFC9pclHWNJUKsDQh6I50J4qc9V0cprAEBzfhFKu2iJdpJcsYmQvD/FmEbGwiGWknMw1tGv27GWX+VlrYuAash3lvuyCgzi9I/j+2AGgTF1Ii1XH3ElzXeGeAcjR2vWKom5y0237+vTSD35ZVF9+56KMaPnxJG2HfqH1g8rG3s7fozQexL6GJEHMClj2l7sopxc6m+2Szjcxp4F0hQGc5PDgxtxiDBQV+MURSNM+fINKqEzzaSoetMQW7DO6K2Jm1eUaC1i0t66jfS00TVDkA726+edzBt1hbCisdNE9YtpUYU0vk9IKjk7ePvopNuzYf1nSIXw43Xq6NuRUIdtCLyGzHdAoJzGnNGdIBlVRoEMqxrr3HBq8hUZbVuRl4mx9oF6h9J7mcBskOIMmBv41gVPkPdWEG937EOG/Gv2AJGRjRLKap6fK5PLSUFudoZWqXCsj0bWA7fT4lb20L9Dj1PjzWZfVTRfZsWuCqWMlkpDQAUVZyLNvpPovpx2rUJXdjEjrl1/Mb718eH4Ok6lTAMzqq1h44WOywHTnrmEaxxSOSH0zfXYuTtaEx4TaKyNDn3PT0JAQp31oR5a5CZN3crs9OePzLIERWupYbirHZrZrWvKL9J5s626UPQK7 t0/IArdH KGWdAHUlr6N0rzzoVQLt+jGrH7pHrLx5B5CiGAtsW4SQ4ZBbMLLRRlG4fvx0aen+cCqxuzYT5Ydl2HbHMdS+UpAXxIUSBz/dCy1GviYUrTJv1huDp0xk+dlzMNRF0AxzQ9BDW1mnBfun8dQmd2S1JFBBWLrX1QyS9Qc/Bz0GSwsyObdAHQw5T4sNvq+CaHgm5W0der+K4UrqU+eSM6AUW+y62+2Kc3Cg+LtRtSRG80Adatz18dg+5cdKPmdH1aYHwvG8WNmnsuZWXJa4rVu1/a+l/p1CU1LhYMibs5rcyGEneP/LfLvBlH/rQdg0mReE5gaqayxQYGcE1/mYULL1xdMzz0Kmo1KEi5HrrssdO9JVySgabWMTi1dzGeqwn+7YYc4RKLquEqN4xYZlPl9/vNolzXkU5vxn1sqx1D73ZYqJyI2eYlqcwyuGRZ4NqDTOjEsxBa2K7vhRdvwiJYmdbiKFF8A3oqsPFCVw12fgMl2xViwSo2pHDnr/cQK+sjkP6Z8tCEPVn2bu346bd28zAYynmhJUZgJIBc40BSy5B6YKHtJZ7X1uu1dEhCgzp00UK0nptlH4Z32K+v0yLmK2ev6O2FBnvZvvYoBLW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Subsequent patches introduce bpf_arena that imposes special alignment requirements on address selection. Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 3 +++ kernel/bpf/syscall.c | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1ebbee1d648e..8b0dcb66eb33 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -139,6 +139,9 @@ struct bpf_map_ops { int (*map_mmap)(struct bpf_map *map, struct vm_area_struct *vma); __poll_t (*map_poll)(struct bpf_map *map, struct file *filp, struct poll_table_struct *pts); + unsigned long (*map_get_unmapped_area)(struct file *filep, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags); /* Functions called by bpf_local_storage maps */ int (*map_local_storage_charge)(struct bpf_local_storage_map *smap, diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index b2750b79ac80..8dd9814a0e14 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -937,6 +937,17 @@ static __poll_t bpf_map_poll(struct file *filp, struct poll_table_struct *pts) return EPOLLERR; } +static unsigned long bpf_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags) +{ + struct bpf_map *map = filp->private_data; + + if (map->ops->map_get_unmapped_area) + return map->ops->map_get_unmapped_area(filp, addr, len, pgoff, flags); + return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); +} + const struct file_operations bpf_map_fops = { #ifdef CONFIG_PROC_FS .show_fdinfo = bpf_map_show_fdinfo, @@ -946,6 +957,7 @@ const struct file_operations bpf_map_fops = { .write = bpf_dummy_write, .mmap = bpf_map_mmap, .poll = bpf_map_poll, + .get_unmapped_area = bpf_get_unmapped_area, }; int bpf_map_new_fd(struct bpf_map *map, int flags) From patchwork Fri Feb 9 04:05:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCF9DC4828F for ; Fri, 9 Feb 2024 04:06:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71BD46B0085; Thu, 8 Feb 2024 23:06:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CB8E6B0087; Thu, 8 Feb 2024 23:06:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 545EC6B0088; Thu, 8 Feb 2024 23:06:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 415A86B0085 for ; Thu, 8 Feb 2024 23:06:34 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DE561802DE for ; Fri, 9 Feb 2024 04:06:33 +0000 (UTC) X-FDA: 81770928666.10.CEE06D9 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf23.hostedemail.com (Postfix) with ESMTP id EE5AC14000A for ; Fri, 9 Feb 2024 04:06:30 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AZ1NeioH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8KvQuM8OQ/k4hdbGFhyp69M4b0glecGcjvBA4TzLzwQ=; b=4XAq38QhCIo/VXi8v/Iel8nuYpoBf0wo8KD+XGJheAaT2iRS7JoGvh0ePcG20V7f1nSr9u UfRSMuk9fzFhHHkeXnjhTKpLMA7E5wC70SXww5h//CNdCz9tdxhNHj4hHAfCi9e9abqT9u fug3uaJJrGh/xAbP8g566iDzjm78pLU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AZ1NeioH; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451591; a=rsa-sha256; cv=none; b=deUUK6dKhHphXSoJQ5xJNkYrEeIW6CMpWS0W8ttghpSaZW05SopWEmZbvMnoZkjFJ5nhHM ObkTAzjhYgZCJYf08iBK06NjaancTvUj0OtkxsEw0J/FwFRCWMCD124eP8SRMTAtqQdSLt wcZPh/nGOE/ZTtTPm+FcIb+AapIRQ3U= Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6e065e5f721so372771b3a.3 for ; Thu, 08 Feb 2024 20:06:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451590; x=1708056390; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8KvQuM8OQ/k4hdbGFhyp69M4b0glecGcjvBA4TzLzwQ=; b=AZ1NeioHPp1J2nGZG5aagPUgUMJLm91rL3pdl4wkDO8z2fYxzpaEVjxqhA3ckwyejv Bbqx1iqbjyQU/AUmVF39OtX2wQCbWqdSrhAdeuePk8oONrcunlS3O3iuVopMQ9iqG7FJ vMCtQudUIKtVLLjoNlkKhAqv5YnPlSALRlm3hFV76JwOiX4+IHWgNDxX8G7mD4dL486y STy3vpJjlrxJLpiQidXXGIdpXlnmI9tHjRgB3xvGUhy4twpYJVkYe0OA82OVeLpYbVWq t/0AIHU7FWlfnJBaEZ9X9nZS9KDVnyrlKUaZWJd5PvElUWav9YYb9gXEkQwVCqzE/2QM sMtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451590; x=1708056390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8KvQuM8OQ/k4hdbGFhyp69M4b0glecGcjvBA4TzLzwQ=; b=vxTnbDI2a4ZQcgaekeZ43LUf+Q4XU2fZVpOGmBbuiFn5vUU8I9a2/rWgj3uCSLmwrk VERGN0TK92uC+8MgYIwmQ0+JWDYoWXeSuAGINECq7UrJpiZiYx40INt6ROS0b6cHytGn XIyt42BXrHiGeMMiX1l82sA2Hi0Eo9gLB4wBvjzuJjAbXhYK2UDftRTvvXnANYYNianO mdOh3m1mJBDrN16/r5vOHFCCkYnoG6FPAwW8nUKdWPFgfFhVD3OIx/dn9gz6HdDP3KSU 3nUW23+zbXOTfmCJYlT9yYNVCzzpEjZvktCBGXA8wG4erdJP77ZjtOd92bvvXpQrE1Ig iawQ== X-Forwarded-Encrypted: i=1; AJvYcCWEXP30p2nzVhSgf0Cf4EdkZPwry0r79IXFwR3NBSGZkjAkfc0UJMOFlS4hKgnkn95lgSeOFkmADrNufVCY9fs9Lx8= X-Gm-Message-State: AOJu0YwfzIFu4DejgfokUb6/x7+vOTjn6X5IIMW9xIf0i3f028GIBCau Vkg8zISHPmb9ItH4cToyHVv1MfCO2N9j+kSd7UR4zukzxp0lMs/Y X-Google-Smtp-Source: AGHT+IGQBdQ96AWfRShK3THRmZ/s+UP/PKdWL9/0jO8LeSeDC7xm1ls7NjuGAIDwxkvXo++YHCAN4Q== X-Received: by 2002:a05:6a00:bdc:b0:6df:e85e:ecc6 with SMTP id x28-20020a056a000bdc00b006dfe85eecc6mr521111pfu.3.1707451589662; Thu, 08 Feb 2024 20:06:29 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWdg8hv5XdH6EKCu1LTwjNtjdDavpVVV78qHLOLm6RZihoGfapXop+ABS9UTa9kAd3MiFUVzdXtkUDGGX96nIPHXik+Vtby1jDoxmr93Sadbon7ZAQZgHUtdc4jR8uEEianIkut2yaoRtShsOlvTSBLyu6qrKADVSDnSWmXWKw7Z+GYkTkVIxTMwWhVUkuQ3tyuAMr64xwMkrKnAnn2HdWivAzaIpaikmZVk5QXlulka1ipuvoCbLwhJ/vJd7n5LB6etkbgJ2LbWrom4JWoyjucVlWdpM7pX6nH6nTuIfGVEzy5qQr/g6YbNCh7o033IAymJ/6+pJsAzwh3lWNBTSOlquwUKBcGvK083zCJn2ES30w86coAEg== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id x29-20020a056a000bdd00b006e0418993cesm591583pfu.8.2024.02.08.20.06.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:29 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 04/20] mm: Expose vmap_pages_range() to the rest of the kernel. Date: Thu, 8 Feb 2024 20:05:52 -0800 Message-Id: <20240209040608.98927-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EE5AC14000A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 9m8xzpzosw4w9ak39ayxkbztx4y1cntg X-HE-Tag: 1707451590-353151 X-HE-Meta: U2FsdGVkX19JpV6UDbxqlnT+t+ZQJje4noN0GITxFPLrNZvtB98BgEp4dKFpJaq3U45iBtZmubMWLAAeDuKQiKo3fexWX2pXRdf5reBF/rltTDIMD6Nsop6Ynb9HB8Sz+GnUMdYfXWkeI2qk5B2H1RkHx9fglPQR/WiS/1os/6R89n6Zq60PhG/1hI0eQbt1Ll+NY1xRmdLJvD0RSs0ONyqaU6cDtD8Zz3I7HL0v170Od6Es/2EhnKF7IFpZlzrWX8IlkQKKO6UK2ovvPEh2o/r0DP0j/AHBngixtWPg2+FBEVHq+VKnrUHeVfOkk3yHjc8ZtTz+i1NtkXLuBzXMFB8iYQ3qFBqyP0Ebv4ax9L0rsj3W4+L0BHxOpSNOHvVPuMLlvC5L8WhfocYNxEFL2RWfTPwPxV8SvjH1rxiB6NPtr0OC1rue8VAfw5CteJm3U1X2Un8Suf2wbMFesBQP2hILgjcNxMY/IwJT3zD33/AIIL3wpZLUQJ6B1RcxiNMByBsL4Fmf0R5qBCuVSqg6hQIWU8aUlDb4EGFl45TNMVBMReFw6yJCiOFj0sLLBsBQ4ghfBzF6jDb4+j9NmG8NCk+Ab5AbTWtvin3fzchY9FgazJ7lPqEsQiWEcK69UPunLS3fqZP6HzZ9ggVv4RlztEOOwJfuFHkeZE48yeHBRjFHu5DZlj+VFCQTaqBjxDX7701ec463IkCw9ql4QZquCdUTHpXWzz2ZjPkMFltuTgpPJ+bap6ONlDQ2EU2pwCUjwoV0YK1/NDFKFNuj5w04yriEUmj8GtxCGzPPdpdeOPXuqYyDxGTT7oTdJsyxMCFyHObg4pjDS1qthRRpeT8wP08xJFPc6PCiBSNSVNZnz0iWNjh5AZn4Y/9ToH/q1Tdx7UWJXsn4fhNorp4EOnGbvo4tz06eDro1jBJpkbSosLinwi5pfZmhML5+eYq+p+xpjAkEI+LnT8iYKNjX12B XmJMn0xZ ijc6aDqfCtKlnz1dtCNQgedOAhhc/xy4JQ7I4US4RQsPm85Aaej4EsA7kMq7xI40KR8qcoYyxh3+E6OxGZWowmIuNSIzVYsNMEpb41fxt1mcpiSOIvsE7B4unL1Hfri5ms3NovffihQRNSl9sbBbLwgHyRsFHjckKIiA5YSW4fiHUmynxzCKQJuCsolhqxOO/mj2IE4mP2kmlk0lCcPcrdfDceeYOBY/teqpZ2TvNS0peDbgYZdQ0UmBJ1TW/s6qc32KhRNoFOOvQ3AhdIqztd9gcq2Tw0w2BzHBRvAx1EARP2JZeSquwBd9rui8lEZ3AtlO3+f1WoKmZxhFgcxzIgIf/Cx1J8wImPo/IS7NN88ZlHp57heGJTiWlLi5cDwx2ySgE2hQSl2/PVd6zmrDcPCORAzuoGSJKQIo6UnIuyQWS5XFOiHOuGvU9zvi7kGe0qvw1JGtLpQfm8rUqkv/nzs7DVL+T/GYHMpnD7RxyI2UBLDbKbX/JvDs7Hp6g/+jQuYIZeCAZToFnePWhK9oWa2/7pze8CHVdCFiSoZy1iCkqfKiSV7niI6tIe8qXOPqjmK/wh/zzdfIid8GX6cBQp4e5kF0JJ0AZylhq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov BPF would like to use the vmap API to implement a lazily-populated memory space which can be shared by multiple userspace threads. The vmap API is generally public and has functions to request and release areas of kernel address space, as well as functions to map various types of backing memory into that space. For example, there is the public ioremap_page_range(), which is used to map device memory into addressable kernel space. The new BPF code needs the functionality of vmap_pages_range() in order to incrementally map privately managed arrays of pages into its vmap area. Indeed this function used to be public, but became private when usecases other than vmalloc happened to disappear. Make it public again for the new external user. The next commits will introduce bpf_arena which is a sparsely populated shared memory region between bpf program and user space process. It will map privately-managed pages into an existing vm area. It's the same pattern and layer of abstraction as ioremap_pages_range(). Signed-off-by: Alexei Starovoitov --- include/linux/vmalloc.h | 2 ++ mm/vmalloc.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..bafb87c69e3d 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -233,6 +233,8 @@ static inline bool is_vm_area_hugepages(const void *addr) #ifdef CONFIG_MMU void vunmap_range(unsigned long addr, unsigned long end); +int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift); static inline void set_vm_flush_reset_perms(void *addr) { struct vm_struct *vm = find_vm_area(addr); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..eae93d575d1b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -625,8 +625,8 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, - pgprot_t prot, struct page **pages, unsigned int page_shift) +int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; From patchwork Fri Feb 9 04:05:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B77F6C4828F for ; Fri, 9 Feb 2024 04:06:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DD2F6B0088; Thu, 8 Feb 2024 23:06:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 065056B0089; Thu, 8 Feb 2024 23:06:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DACC76B008A; Thu, 8 Feb 2024 23:06:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C070C6B0088 for ; Thu, 8 Feb 2024 23:06:37 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 67FC1C01E7 for ; Fri, 9 Feb 2024 04:06:37 +0000 (UTC) X-FDA: 81770928834.16.D0CA82A Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf11.hostedemail.com (Postfix) with ESMTP id 52FAC4000B for ; Fri, 9 Feb 2024 04:06:35 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eOb2vdzW; spf=pass (imf11.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451595; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CbW1Ux0GyRomZa2QV1OutRN2C00HubCW5vkIGuoKpjE=; b=yd5hjyfRJr4+CC1GjOOA7FubRtb4X7QtmNi3i8Z90f39TnPEC2WceCqv1Xu1bvOTANqQJu TXh5iW4efgXUSXqPhAB2Yq2XphrS/wAjla8Nhk9CatUlQ3lNV5GRgNXdYA9MFqeIhV1J0F cKlyy2+m9CU4l+Xc0BqdezuweLzOYCo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451595; a=rsa-sha256; cv=none; b=QpvIWy2ktB7VUcH/wLEEDdn8SybXHrBo5CvJ+z4xtmtB7FrIlkneDoGvvcC08qUF6bnlXH YqQW8B1FfPXok5BjjbE2UGyGxcFHPl2kk2lZNmcanq4BhOhmNGHQasjlnBiUEigwGA5ght nRtDZF9zgFD5BptNZw9xS7NqH6L9Tz0= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=eOb2vdzW; spf=pass (imf11.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-296a5863237so1239476a91.0 for ; Thu, 08 Feb 2024 20:06:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451594; x=1708056394; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CbW1Ux0GyRomZa2QV1OutRN2C00HubCW5vkIGuoKpjE=; b=eOb2vdzWrslUOBvl0upT5jUxAaR91uf81faekii/pB/YibyG2DVbEnTMqrLwyBDIp0 8+mf6EyYOaKqgoepSM2aYtW9tJ2VbZcKX2Hjvze5Q3Tlp8TDxRD6gEpaD21EjW6g2it5 MaZcVf4EIAf+4vjtNPkHt9mVBCITpeCb6dZe0t9qkrqOljxvYmF0D85fQuj/gI/mb4w5 DD+ljDqiwb/zvKQ6B46ORofF08HB+z7Se69Ra47sAiuMpK/TOygs4X+mvebGwAhnbNfO Ye9E6HAZw3UhZVxk+Kp/ZoWk/G1C0bzEGH3uJ/eqy/YoctqIzsUmcn1MegIpaZjoc4Ow b9LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451594; x=1708056394; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbW1Ux0GyRomZa2QV1OutRN2C00HubCW5vkIGuoKpjE=; b=v/WbSTbyGeF6HWxfjo7RBLqfFFSNQb4iOYqHXRfE8IOckgwnb9h0BKg63UOssMiZ3w 9wd7220RkaOiFScD2QDgzSIV2Q0v4tSljwTMkWHAyI+TP9kEg4NoBkPoPNW/EFBKeX7+ 6KipJclLlAvhZpReVFXIUiRX3C5rgpTJYa8p3/eLNvSQTOYzvUbSmdLT4+g2T/82DLPn 2T7nqYbAvZdNTxaeetNei4OChTZ8UQE1JXoVUMhkmK3DJx6PzZoEQh34Aa0xZfaJDqUh AJKJInfHrRvAVtt4zbQCTXyEsGtv/JWCo6TA1CXR5Nu2D5WReYvVwVYIcVm9K38/LEad olQQ== X-Forwarded-Encrypted: i=1; AJvYcCUQVtvgvgi11rWb346aSIz5fzfQDDmZ0a5kpPRmLy9ckwjO4a8dpAq2sYo8kvEC20V0e7zVpbAmG+xUtA1bnjp2MDM= X-Gm-Message-State: AOJu0YysE6GIEv6UfBp52nrXq1jstTtlTm/zHvDkmYrIVZagQXhcfcBw PdrfdWfj8+vMZlGlKxecyOOKKZkDgVsQGsBFuDI+PfW1U4l7aU9l X-Google-Smtp-Source: AGHT+IHXQWcnr8EgdwX+jDOt+brv7WOQ4JqLcKYBLJJYhVT5dTCNwjVtOKt42l0jWIJeFtNrIqw36w== X-Received: by 2002:a17:90a:f68d:b0:296:530:996e with SMTP id cl13-20020a17090af68d00b002960530996emr692978pjb.20.1707451593996; Thu, 08 Feb 2024 20:06:33 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXGmdtHWr7aWPEnLy21kkm0M5ajC/CAPNM2vLsQ/qmOrob/yEZ64zsIHUAM7Tv4Y2yn/DeeUFhBB+RnfdGM612UT3kb7vzpjw3wSPSsBoLe8NU2IBLKqCxKV7NP7ajHq9UzjqQNrpyikIwZliwB20cEoMuqVqcjgaUuPF4xMiQP86IAFBSMcGx0elohbNToZnLmkMqC8i6l+NPkG7mh1WMd4tWfhhPM0doGowtdNM3ChK+vcpLNCD86GI3JGdvwvJcRhQ5FAoOey58Xh6ISBXz7rkP17CiZ87CXrEsWeFD4drpogSYSe3HE2BLctjrDZEjLsU4viPxg7EaADdNieoUUXciZ4RWJjUujznNfrM1wxLzF5GlSbQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id f5-20020a17090aec8500b002970ce13bb9sm85149pjy.52.2024.02.08.20.06.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:33 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 05/20] bpf: Introduce bpf_arena. Date: Thu, 8 Feb 2024 20:05:53 -0800 Message-Id: <20240209040608.98927-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: o4kxsonwb7nswid7q5sw7d5hbk49zdwp X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 52FAC4000B X-Rspam-User: X-HE-Tag: 1707451595-279610 X-HE-Meta: U2FsdGVkX19R/GGJYznMZmSBVNUfmDnjjZpqnAixTyKt+X/YTulr6pQK56+7R9NiJi/zPIQ+v0nPn3lGXX1VHQzO/6poTAJgv0mxAmsVOcgMq/eNOVEMVRbJs7swFEXxqHuK4ymL/cS3sWb+1ZFV2RbJUaM44P/+2EkUKziSBEEMam0PfiFmwiHCT7rzjsJryCF1RWdl4PPOnJaq0tCuRS4nTW5y+RX/iIZtKzj6SH2XpbkyTyW+fNRAp1sKcc+o6Py2GQCu590dkZ8r3fB2aogNDt3iKIyUWMqC5SEO6c4FTw5wRVEGWTRhyXv0+8L2dVfvJVTXc7DewYdEptU34kQ+TOlouTfhWHGstRbmtvxG8yolf+RNMDcqr/f8c3pLfXasC67n5JJ1gCPeuQF7FC72+FXWIDcRZhZ1V7ZM4Nmws73zUZoVBlMpaUk1iT3k4qn8EX80Bb6wNEIadpg5J6WAzD3zBvvWkbzDsZPux8PjY/7x78b33CxX8D/cuKaRjtt+n9X1sFJ9lnThpzYLibXh7cMoHkl+x5Pua8JDUIGdOeTIbciMbh8AWURToqGY6R1HgFoNXOpWdrJ3IVaNxhf9llEMIiSy2gWfedKMVAzf8MDuTRES8Z+Fw4pUe9SJWcua40IDA5Qgy80AP9pC18rq11PMrns1yydd7DYUGBoeBOrVuXFj8e5n4UEHjE7Sr6VajmvGrl9UU7vRAON+XBGkDSkQ/spILkM8WrDtDaMuT1B7N3/BjHzSNMhD+3hnkjkBL984+p+5BCLDH1US3TtWnEmxoVO4yn5tnCIpTOd48tg6Uv/qh0foFELyEsIsxjg8T7p9SK30lGdL5NZZBwft6hPhorn+9d7WjcvhVTKsHzER9Gg3zgxi4RPRWWlDaLd+QQc7HVvbvZWfJFd/kMpsKnVgmrgPOYcpTpH7SHmpDjE1pOtgaoIYGF39sucwaNmIHBbIRJyKcXNRizI 5TSRpkNu XwDsgeerhbLzdRc+KYXk5TFq4F7sklsadhUwFO/zGkL2GTsREfubVwejqBA8BZRA+JxUmI4y6DPpRnQYo7JaCow924uC03INopqQfpIdCfhU9naz6pKY+O8Hm3yuM60wTSjMiwTFJFdCN0XYSpf6gqJwqCjjPy/8YaGH7SnfwF9vPvqJY+24MU18mX7ixp+fuq+ZZ3UAXWdQAJMXz5t1YTxGPCrZbMqWMW7KK/OI0y4+ZEDs/YuWyksFRJLO/wr/f/XfiJBkpR6sUQIrM51stkvFirjycBYShwKyaVSSdy9Tm//FO1pzP29/Tt7mFVbNuOqRx8HfFY39e+ZYhM/2SnYtwOWnV1QtmLj962FDMYiye2TuAV6SmWiN+SE8u0HCneg07aZH8pewgKN0m02dWA/UDRT9ISycwaJmATlMqGvP9fCvjjK/lpqmVJ+KKJ+m1QsHPj7jBB1ceQ+YwqF1PUXAMDFRC7Js1r7q1Nkb8VCufSOCeVXFJFi3gXvcNkYT7WqlGXftcxhXqsmmwNJqTo6aYUowm0yVeMd868vrNdHVTIpZfG03OiOiyXAKO/5nkhYmqjOBqvB96AcibWLmMS62h5oltju1A7Yqm55XCSTRsLEEv9YF1gzbz2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce bpf_arena, which is a sparse shared memory region between the bpf program and user space. Use cases: 1. User space mmap-s bpf_arena and uses it as a traditional mmap-ed anonymous region, like memcached or any key/value storage. The bpf program implements an in-kernel accelerator. XDP prog can search for a key in bpf_arena and return a value without going to user space. 2. The bpf program builds arbitrary data structures in bpf_arena (hash tables, rb-trees, sparse arrays), while user space consumes it. 3. bpf_arena is a "heap" of memory from the bpf program's point of view. The user space may mmap it, but bpf program will not convert pointers to user base at run-time to improve bpf program speed. Initially, the kernel vm_area and user vma are not populated. User space can fault in pages within the range. While servicing a page fault, bpf_arena logic will insert a new page into the kernel and user vmas. The bpf program can allocate pages from that region via bpf_arena_alloc_pages(). This kernel function will insert pages into the kernel vm_area. The subsequent fault-in from user space will populate that page into the user vma. The BPF_F_SEGV_ON_FAULT flag at arena creation time can be used to prevent fault-in from user space. In such a case, if a page is not allocated by the bpf program and not present in the kernel vm_area, the user process will segfault. This is useful for use cases 2 and 3 above. bpf_arena_alloc_pages() is similar to user space mmap(). It allocates pages either at a specific address within the arena or allocates a range with the maple tree. bpf_arena_free_pages() is analogous to munmap(), which frees pages and removes the range from the kernel vm_area and from user process vmas. bpf_arena can be used as a bpf program "heap" of up to 4GB. The speed of bpf program is more important than ease of sharing with user space. This is use case 3. In such a case, the BPF_F_NO_USER_CONV flag is recommended. It will tell the verifier to treat the rX = bpf_arena_cast_user(rY) instruction as a 32-bit move wX = wY, which will improve bpf prog performance. Otherwise, bpf_arena_cast_user is translated by JIT to conditionally add the upper 32 bits of user vm_start (if the pointer is not NULL) to arena pointers before they are stored into memory. This way, user space sees them as valid 64-bit pointers. Diff https://github.com/llvm/llvm-project/pull/79902 taught LLVM BPF backend to generate the bpf_cast_kern() instruction before dereference of the arena pointer and the bpf_cast_user() instruction when the arena pointer is formed. In a typical bpf program there will be very few bpf_cast_user(). From LLVM's point of view, arena pointers are tagged as __attribute__((address_space(1))). Hence, clang provides helpful diagnostics when pointers cross address space. Libbpf and the kernel support only address_space == 1. All other address space identifiers are reserved. rX = bpf_cast_kern(rY, addr_space) tells the verifier that rX->type = PTR_TO_ARENA. Any further operations on PTR_TO_ARENA register have to be in the 32-bit domain. The verifier will mark load/store through PTR_TO_ARENA with PROBE_MEM32. JIT will generate them as kern_vm_start + 32bit_addr memory accesses. The behavior is similar to copy_from_kernel_nofault() except that no address checks are necessary. The address is guaranteed to be in the 4GB range. If the page is not present, the destination register is zeroed on read, and the operation is ignored on write. rX = bpf_cast_user(rY, addr_space) tells the verifier that rX->type = unknown scalar. If arena->map_flags has BPF_F_NO_USER_CONV set, then the verifier converts cast_user to mov32. Otherwise, JIT will emit native code equivalent to: rX = (u32)rY; if (rY) rX |= clear_lo32_bits(arena->user_vm_start); /* replace hi32 bits in rX */ After such conversion, the pointer becomes a valid user pointer within bpf_arena range. The user process can access data structures created in bpf_arena without any additional computations. For example, a linked list built by a bpf program can be walked natively by user space. Signed-off-by: Alexei Starovoitov Reviewed-by: Barret Rhoden --- include/linux/bpf.h | 5 +- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 7 + kernel/bpf/Makefile | 3 + kernel/bpf/arena.c | 557 +++++++++++++++++++++++++++++++++ kernel/bpf/core.c | 11 + kernel/bpf/syscall.c | 3 + kernel/bpf/verifier.c | 1 + tools/include/uapi/linux/bpf.h | 7 + 9 files changed, 593 insertions(+), 2 deletions(-) create mode 100644 kernel/bpf/arena.c diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 8b0dcb66eb33..de557c6c42e0 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -37,6 +37,7 @@ struct perf_event; struct bpf_prog; struct bpf_prog_aux; struct bpf_map; +struct bpf_arena; struct sock; struct seq_file; struct btf; @@ -534,8 +535,8 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock); void bpf_rb_root_free(const struct btf_field *field, void *rb_root, struct bpf_spin_lock *spin_lock); - - +u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena); +u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena); int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size); struct bpf_offload_dev; diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index 94baced5a1ad..9f2a6b83b49e 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -132,6 +132,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) +BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint) BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index d96708380e52..f6648851eae6 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -983,6 +983,7 @@ enum bpf_map_type { BPF_MAP_TYPE_BLOOM_FILTER, BPF_MAP_TYPE_USER_RINGBUF, BPF_MAP_TYPE_CGRP_STORAGE, + BPF_MAP_TYPE_ARENA, __MAX_BPF_MAP_TYPE }; @@ -1370,6 +1371,12 @@ enum { /* BPF token FD is passed in a corresponding command's token_fd field */ BPF_F_TOKEN_FD = (1U << 16), + +/* When user space page faults in bpf_arena send SIGSEGV instead of inserting new page */ + BPF_F_SEGV_ON_FAULT = (1U << 17), + +/* Do not translate kernel bpf_arena pointers to user pointers */ + BPF_F_NO_USER_CONV = (1U << 18), }; /* Flags for BPF_PROG_QUERY. */ diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile index 4ce95acfcaa7..368c5d86b5b7 100644 --- a/kernel/bpf/Makefile +++ b/kernel/bpf/Makefile @@ -15,6 +15,9 @@ obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o +ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy) +obj-$(CONFIG_BPF_SYSCALL) += arena.o +endif obj-$(CONFIG_BPF_JIT) += dispatcher.o ifeq ($(CONFIG_NET),y) obj-$(CONFIG_BPF_SYSCALL) += devmap.o diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c new file mode 100644 index 000000000000..5c1014471740 --- /dev/null +++ b/kernel/bpf/arena.c @@ -0,0 +1,557 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include +#include + +/* + * bpf_arena is a sparsely populated shared memory region between bpf program and + * user space process. + * + * For example on x86-64 the values could be: + * user_vm_start 7f7d26200000 // picked by mmap() + * kern_vm_start ffffc90001e69000 // picked by get_vm_area() + * For user space all pointers within the arena are normal 8-byte addresses. + * In this example 7f7d26200000 is the address of the first page (pgoff=0). + * The bpf program will access it as: kern_vm_start + lower_32bit_of_user_ptr + * (u32)7f7d26200000 -> 26200000 + * hence + * ffffc90001e69000 + 26200000 == ffffc90028069000 is "pgoff=0" within 4Gb + * kernel memory region. + * + * BPF JITs generate the following code to access arena: + * mov eax, eax // eax has lower 32-bit of user pointer + * mov word ptr [rax + r12 + off], bx + * where r12 == kern_vm_start and off is s16. + * Hence allocate 4Gb + GUARD_SZ/2 on each side. + * + * Initially kernel vm_area and user vma are not populated. + * User space can fault-in any address which will insert the page + * into kernel and user vma. + * bpf program can allocate a page via bpf_arena_alloc_pages() kfunc + * which will insert it into kernel vm_area. + * The later fault-in from user space will populate that page into user vma. + */ + +/* number of bytes addressable by LDX/STX insn with 16-bit 'off' field */ +#define GUARD_SZ (1ull << sizeof(((struct bpf_insn *)0)->off) * 8) +#define KERN_VM_SZ ((1ull << 32) + GUARD_SZ) + +struct bpf_arena { + struct bpf_map map; + u64 user_vm_start; + u64 user_vm_end; + struct vm_struct *kern_vm; + struct maple_tree mt; + struct list_head vma_list; + struct mutex lock; +}; + +u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) +{ + return arena ? (u64) (long) arena->kern_vm->addr + GUARD_SZ / 2 : 0; +} + +u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) +{ + return arena ? arena->user_vm_start : 0; +} + +static long arena_map_peek_elem(struct bpf_map *map, void *value) +{ + return -EOPNOTSUPP; +} + +static long arena_map_push_elem(struct bpf_map *map, void *value, u64 flags) +{ + return -EOPNOTSUPP; +} + +static long arena_map_pop_elem(struct bpf_map *map, void *value) +{ + return -EOPNOTSUPP; +} + +static long arena_map_delete_elem(struct bpf_map *map, void *value) +{ + return -EOPNOTSUPP; +} + +static int arena_map_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + return -EOPNOTSUPP; +} + +static long compute_pgoff(struct bpf_arena *arena, long uaddr) +{ + return (u32)(uaddr - (u32)arena->user_vm_start) >> PAGE_SHIFT; +} + +static struct bpf_map *arena_map_alloc(union bpf_attr *attr) +{ + struct vm_struct *kern_vm; + int numa_node = bpf_map_attr_numa_node(attr); + struct bpf_arena *arena; + u64 vm_range; + int err = -ENOMEM; + + if (attr->key_size || attr->value_size || attr->max_entries == 0 || + /* BPF_F_MMAPABLE must be set */ + !(attr->map_flags & BPF_F_MMAPABLE) || + /* No unsupported flags present */ + (attr->map_flags & ~(BPF_F_SEGV_ON_FAULT | BPF_F_MMAPABLE | BPF_F_NO_USER_CONV))) + return ERR_PTR(-EINVAL); + + if (attr->map_extra & ~PAGE_MASK) + /* If non-zero the map_extra is an expected user VMA start address */ + return ERR_PTR(-EINVAL); + + vm_range = (u64)attr->max_entries * PAGE_SIZE; + if (vm_range > (1ull << 32)) + return ERR_PTR(-E2BIG); + + if ((attr->map_extra >> 32) != ((attr->map_extra + vm_range - 1) >> 32)) + /* user vma must not cross 32-bit boundary */ + return ERR_PTR(-ERANGE); + + kern_vm = get_vm_area(KERN_VM_SZ, VM_MAP | VM_USERMAP); + if (!kern_vm) + return ERR_PTR(-ENOMEM); + + arena = bpf_map_area_alloc(sizeof(*arena), numa_node); + if (!arena) + goto err; + + arena->kern_vm = kern_vm; + arena->user_vm_start = attr->map_extra; + if (arena->user_vm_start) + arena->user_vm_end = arena->user_vm_start + vm_range; + + INIT_LIST_HEAD(&arena->vma_list); + bpf_map_init_from_attr(&arena->map, attr); + mt_init_flags(&arena->mt, MT_FLAGS_ALLOC_RANGE); + mutex_init(&arena->lock); + + return &arena->map; +err: + free_vm_area(kern_vm); + return ERR_PTR(err); +} + +static int for_each_pte(pte_t *ptep, unsigned long addr, void *data) +{ + struct page *page; + pte_t pte; + + pte = ptep_get(ptep); + if (!pte_present(pte)) + return 0; + page = pte_page(pte); + /* + * We do not update pte here: + * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug + * 2. TLB flushing is batched or deferred. Even if we clear pte, + * the TLB entries can stick around and continue to permit access to + * the freed page. So it all relies on 1. + */ + __free_page(page); + return 0; +} + +static void arena_map_free(struct bpf_map *map) +{ + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + /* + * Check that user vma-s are not around when bpf map is freed. + * mmap() holds vm_file which holds bpf_map refcnt. + * munmap() must have happened on vma followed by arena_vm_close() + * which would clear arena->vma_list. + */ + if (WARN_ON_ONCE(!list_empty(&arena->vma_list))) + return; + + /* + * free_vm_area() calls remove_vm_area() that calls free_unmap_vmap_area(). + * It unmaps everything from vmalloc area and clears pgtables. + * Call apply_to_existing_page_range() first to find populated ptes and + * free those pages. + */ + apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), + KERN_VM_SZ - GUARD_SZ / 2, for_each_pte, NULL); + free_vm_area(arena->kern_vm); + mtree_destroy(&arena->mt); + bpf_map_area_free(arena); +} + +static void *arena_map_lookup_elem(struct bpf_map *map, void *key) +{ + return ERR_PTR(-EINVAL); +} + +static long arena_map_update_elem(struct bpf_map *map, void *key, + void *value, u64 flags) +{ + return -EOPNOTSUPP; +} + +static int arena_map_check_btf(const struct bpf_map *map, const struct btf *btf, + const struct btf_type *key_type, const struct btf_type *value_type) +{ + return 0; +} + +static u64 arena_map_mem_usage(const struct bpf_map *map) +{ + return 0; +} + +struct vma_list { + struct vm_area_struct *vma; + struct list_head head; +}; + +static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) +{ + struct vma_list *vml; + + vml = kmalloc(sizeof(*vml), GFP_KERNEL); + if (!vml) + return -ENOMEM; + vma->vm_private_data = vml; + vml->vma = vma; + list_add(&vml->head, &arena->vma_list); + return 0; +} + +static void arena_vm_close(struct vm_area_struct *vma) +{ + struct bpf_map *map = vma->vm_file->private_data; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + struct vma_list *vml; + + guard(mutex)(&arena->lock); + vml = vma->vm_private_data; + list_del(&vml->head); + vma->vm_private_data = NULL; + kfree(vml); +} + +#define MT_ENTRY ((void *)&arena_map_ops) /* unused. has to be valid pointer */ + +static vm_fault_t arena_vm_fault(struct vm_fault *vmf) +{ + struct bpf_map *map = vmf->vma->vm_file->private_data; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + struct page *page; + long kbase, kaddr; + int ret; + + kbase = bpf_arena_get_kern_vm_start(arena); + kaddr = kbase + (u32)(vmf->address & PAGE_MASK); + + guard(mutex)(&arena->lock); + page = vmalloc_to_page((void *)kaddr); + if (page) + /* already have a page vmap-ed */ + goto out; + + if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT) + /* User space requested to segfault when page is not allocated by bpf prog */ + return VM_FAULT_SIGSEGV; + + ret = mtree_insert(&arena->mt, vmf->pgoff, MT_ENTRY, GFP_KERNEL); + if (ret) + return VM_FAULT_SIGSEGV; + + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) { + mtree_erase(&arena->mt, vmf->pgoff); + return VM_FAULT_SIGSEGV; + } + + ret = vmap_pages_range(kaddr, kaddr + PAGE_SIZE, PAGE_KERNEL, &page, PAGE_SHIFT); + if (ret) { + mtree_erase(&arena->mt, vmf->pgoff); + __free_page(page); + return VM_FAULT_SIGSEGV; + } +out: + page_ref_add(page, 1); + vmf->page = page; + return 0; +} + +static const struct vm_operations_struct arena_vm_ops = { + .close = arena_vm_close, + .fault = arena_vm_fault, +}; + +static unsigned long arena_get_unmapped_area(struct file *filp, unsigned long addr, + unsigned long len, unsigned long pgoff, + unsigned long flags) +{ + struct bpf_map *map = filp->private_data; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + long ret; + + if (pgoff) + return -EINVAL; + if (len > (1ull << 32)) + return -E2BIG; + + /* if user_vm_start was specified at arena creation time */ + if (arena->user_vm_start) { + if (len > arena->user_vm_end - arena->user_vm_start) + return -E2BIG; + if (len != arena->user_vm_end - arena->user_vm_start) + return -EINVAL; + if (addr != arena->user_vm_start) + return -EINVAL; + } + + ret = current->mm->get_unmapped_area(filp, addr, len * 2, 0, flags); + if (IS_ERR_VALUE(ret)) + return 0; + if ((ret >> 32) == ((ret + len - 1) >> 32)) + return ret; + if (WARN_ON_ONCE(arena->user_vm_start)) + /* checks at map creation time should prevent this */ + return -EFAULT; + return round_up(ret, 1ull << 32); +} + +static int arena_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) +{ + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + guard(mutex)(&arena->lock); + if (arena->user_vm_start && arena->user_vm_start != vma->vm_start) + /* + * If map_extra was not specified at arena creation time then + * 1st user process can do mmap(NULL, ...) to pick user_vm_start + * 2nd user process must pass the same addr to mmap(addr, MAP_FIXED..); + * or + * specify addr in map_extra and + * use the same addr later with mmap(addr, MAP_FIXED..); + */ + return -EBUSY; + + if (arena->user_vm_end && arena->user_vm_end != vma->vm_end) + /* all user processes must have the same size of mmap-ed region */ + return -EBUSY; + + /* Earlier checks should prevent this */ + if (WARN_ON_ONCE(vma->vm_end - vma->vm_start > (1ull << 32) || vma->vm_pgoff)) + return -EFAULT; + + if (remember_vma(arena, vma)) + return -ENOMEM; + + arena->user_vm_start = vma->vm_start; + arena->user_vm_end = vma->vm_end; + /* + * bpf_map_mmap() checks that it's being mmaped as VM_SHARED and + * clears VM_MAYEXEC. Set VM_DONTEXPAND as well to avoid + * potential change of user_vm_start. + */ + vm_flags_set(vma, VM_DONTEXPAND); + vma->vm_ops = &arena_vm_ops; + return 0; +} + +static int arena_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off) +{ + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + if ((u64)off > arena->user_vm_end - arena->user_vm_start) + return -ERANGE; + *imm = (unsigned long)arena->user_vm_start; + return 0; +} + +BTF_ID_LIST_SINGLE(bpf_arena_map_btf_ids, struct, bpf_arena) +const struct bpf_map_ops arena_map_ops = { + .map_meta_equal = bpf_map_meta_equal, + .map_alloc = arena_map_alloc, + .map_free = arena_map_free, + .map_direct_value_addr = arena_map_direct_value_addr, + .map_mmap = arena_map_mmap, + .map_get_unmapped_area = arena_get_unmapped_area, + .map_get_next_key = arena_map_get_next_key, + .map_push_elem = arena_map_push_elem, + .map_peek_elem = arena_map_peek_elem, + .map_pop_elem = arena_map_pop_elem, + .map_lookup_elem = arena_map_lookup_elem, + .map_update_elem = arena_map_update_elem, + .map_delete_elem = arena_map_delete_elem, + .map_check_btf = arena_map_check_btf, + .map_mem_usage = arena_map_mem_usage, + .map_btf_id = &bpf_arena_map_btf_ids[0], +}; + +static u64 clear_lo32(u64 val) +{ + return val & ~(u64)~0U; +} + +/* + * Allocate pages and vmap them into kernel vmalloc area. + * Later the pages will be mmaped into user space vma. + */ +static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id) +{ + /* user_vm_end/start are fixed before bpf prog runs */ + long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT; + u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena); + long pgoff = 0, nr_pages = 0; + struct page **pages; + u32 uaddr32; + int ret, i; + + if (page_cnt > page_cnt_max) + return 0; + + if (uaddr) { + if (uaddr & ~PAGE_MASK) + return 0; + pgoff = compute_pgoff(arena, uaddr); + if (pgoff + page_cnt > page_cnt_max) + /* requested address will be outside of user VMA */ + return 0; + } + + /* zeroing is needed, since alloc_pages_bulk_array() only fills in non-zero entries */ + pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL); + if (!pages) + return 0; + + guard(mutex)(&arena->lock); + + if (uaddr) + ret = mtree_insert_range(&arena->mt, pgoff, pgoff + page_cnt - 1, + MT_ENTRY, GFP_KERNEL); + else + ret = mtree_alloc_range(&arena->mt, &pgoff, MT_ENTRY, + page_cnt, 0, page_cnt_max - 1, GFP_KERNEL); + if (ret) + goto out_free_pages; + + nr_pages = alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_ZERO, node_id, page_cnt, pages); + if (nr_pages != page_cnt) + goto out; + + uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE); + /* Earlier checks make sure that uaddr32 + page_cnt * PAGE_SIZE will not overflow 32-bit */ + ret = vmap_pages_range(kern_vm_start + uaddr32, + kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE, + PAGE_KERNEL, pages, PAGE_SHIFT); + if (ret) + goto out; + kvfree(pages); + return clear_lo32(arena->user_vm_start) + uaddr32; +out: + mtree_erase(&arena->mt, pgoff); +out_free_pages: + if (pages) + for (i = 0; i < nr_pages; i++) + __free_page(pages[i]); + kvfree(pages); + return 0; +} + +/* + * If page is present in vmalloc area, unmap it from vmalloc area, + * unmap it from all user space vma-s, + * and free it. + */ +static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) +{ + struct vma_list *vml; + + list_for_each_entry(vml, &arena->vma_list, head) + zap_page_range_single(vml->vma, uaddr, + PAGE_SIZE * page_cnt, NULL); +} + +static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) +{ + u64 full_uaddr, uaddr_end; + long kaddr, pgoff, i; + struct page *page; + + /* only aligned lower 32-bit are relevant */ + uaddr = (u32)uaddr; + uaddr &= PAGE_MASK; + full_uaddr = clear_lo32(arena->user_vm_start) + uaddr; + uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT)); + if (full_uaddr >= uaddr_end) + return; + + page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT; + + guard(mutex)(&arena->lock); + + pgoff = compute_pgoff(arena, uaddr); + /* clear range */ + mtree_store_range(&arena->mt, pgoff, pgoff + page_cnt - 1, NULL, GFP_KERNEL); + + if (page_cnt > 1) + /* bulk zap if multiple pages being freed */ + zap_pages(arena, full_uaddr, page_cnt); + + kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr; + for (i = 0; i < page_cnt; i++, kaddr += PAGE_SIZE, full_uaddr += PAGE_SIZE) { + page = vmalloc_to_page((void *)kaddr); + if (!page) + continue; + if (page_cnt == 1 && page_mapped(page)) /* mapped by some user process */ + zap_pages(arena, full_uaddr, 1); + vunmap_range(kaddr, kaddr + PAGE_SIZE); + __free_page(page); + } +} + +__bpf_kfunc_start_defs(); + +__bpf_kfunc void *bpf_arena_alloc_pages(void *p__map, void *addr__ign, u32 page_cnt, + int node_id, u64 flags) +{ + struct bpf_map *map = p__map; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt) + return NULL; + + return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id); +} + +__bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) +{ + struct bpf_map *map = p__map; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + if (map->map_type != BPF_MAP_TYPE_ARENA || !page_cnt || !ptr__ign) + return; + arena_free_pages(arena, (long)ptr__ign, page_cnt); +} +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(arena_kfuncs) +BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_KFUNCS_END(arena_kfuncs) + +static const struct btf_kfunc_id_set common_kfunc_set = { + .owner = THIS_MODULE, + .set = &arena_kfuncs, +}; + +static int __init kfunc_init(void) +{ + return register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set); +} +late_initcall(kfunc_init); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 71c459a51d9e..2539d9bfe369 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2970,6 +2970,17 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, { } +/* for configs without MMU or 32-bit */ +__weak const struct bpf_map_ops arena_map_ops; +__weak u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) +{ + return 0; +} +__weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) +{ + return 0; +} + #ifdef CONFIG_BPF_SYSCALL static int __init bpf_global_ma_init(void) { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8dd9814a0e14..6b9efb3f79dd 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -164,6 +164,7 @@ static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, if (bpf_map_is_offloaded(map)) { return bpf_map_offload_update_elem(map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_CPUMAP || + map->map_type == BPF_MAP_TYPE_ARENA || map->map_type == BPF_MAP_TYPE_STRUCT_OPS) { return map->ops->map_update_elem(map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH || @@ -1172,6 +1173,7 @@ static int map_create(union bpf_attr *attr) } if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER && + attr->map_type != BPF_MAP_TYPE_ARENA && attr->map_extra != 0) return -EINVAL; @@ -1261,6 +1263,7 @@ static int map_create(union bpf_attr *attr) case BPF_MAP_TYPE_LRU_PERCPU_HASH: case BPF_MAP_TYPE_STRUCT_OPS: case BPF_MAP_TYPE_CPUMAP: + case BPF_MAP_TYPE_ARENA: if (!bpf_token_capable(token, CAP_BPF)) goto put_token; break; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index db569ce89fb1..3c77a3ab1192 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18047,6 +18047,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_SK_STORAGE: case BPF_MAP_TYPE_TASK_STORAGE: case BPF_MAP_TYPE_CGRP_STORAGE: + case BPF_MAP_TYPE_ARENA: break; default: verbose(env, diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index d96708380e52..f6648851eae6 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -983,6 +983,7 @@ enum bpf_map_type { BPF_MAP_TYPE_BLOOM_FILTER, BPF_MAP_TYPE_USER_RINGBUF, BPF_MAP_TYPE_CGRP_STORAGE, + BPF_MAP_TYPE_ARENA, __MAX_BPF_MAP_TYPE }; @@ -1370,6 +1371,12 @@ enum { /* BPF token FD is passed in a corresponding command's token_fd field */ BPF_F_TOKEN_FD = (1U << 16), + +/* When user space page faults in bpf_arena send SIGSEGV instead of inserting new page */ + BPF_F_SEGV_ON_FAULT = (1U << 17), + +/* Do not translate kernel bpf_arena pointers to user pointers */ + BPF_F_NO_USER_CONV = (1U << 18), }; /* Flags for BPF_PROG_QUERY. */ From patchwork Fri Feb 9 04:05:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61E09C4829B for ; Fri, 9 Feb 2024 04:06:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E21818D0002; Thu, 8 Feb 2024 23:06:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DAA1F8D0001; Thu, 8 Feb 2024 23:06:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4A026B0092; Thu, 8 Feb 2024 23:06:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id ADC4A6B008A for ; Thu, 8 Feb 2024 23:06:41 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7A69B12017E for ; Fri, 9 Feb 2024 04:06:41 +0000 (UTC) X-FDA: 81770929002.29.CAB78AF Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf25.hostedemail.com (Postfix) with ESMTP id A25ACA0013 for ; Fri, 9 Feb 2024 04:06:39 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fuRKxMxu; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451599; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dj+O15ERlc2oNccgk7u96NynUdtw7IljkhhXG+cbct8=; b=XvDiOaEDafIF0Tu7ossazH3FFCHm6DtbU77oc0tWsEDF5r42UlZcfAIMk62L17tii/iOOn ZpXHmTalrFq1SM+mIyxoUWt7sC1eJT8EqYOILtEhadGkAXIYLuYcxJ73ussTJbCWu+1Gee alCu8dTLk9gC/pH/l9HxR7PwTeQkG1E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451599; a=rsa-sha256; cv=none; b=yXDAA7EI2v5hsYMaAonYayR1RKjiZVI2MekmA16/4ga9AJBGW2jLPVgGO6zy/s7kykKd39 zOxYVtnePjEnCKb8XI+PfjhKxiWjyGQgPUf3cpMaxjISci3Q+jhYmWphM8hcmy+dW0YJn+ aFCh6V9Cona72Q3CoSnTfKtEUCg4TIU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fuRKxMxu; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1d98fc5ebceso3216325ad.1 for ; Thu, 08 Feb 2024 20:06:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451598; x=1708056398; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dj+O15ERlc2oNccgk7u96NynUdtw7IljkhhXG+cbct8=; b=fuRKxMxupdvujWNcYPBuh+orlZGnmBqVZHRe0WVTX/FMQclTZQ7o+1Hu/aZgOn25O3 uID/thtdOSPn3S0OBPMNCZFE8HypzqSGhMAUQqDZf07jEKiV9Pc7tsfSVklfTLG7mbjT UO25UZNiiHtofcXiYe4JxRRVRx3zz4ck2FbT56I6VRJiJ4RZh0rbhSeTpzdadJ/20fER HGi3dwwOYnHEsgblrDB5qieif7L2IGZP924yoCLzkQNXKgea/n3C7POmvYlycvxKU6OS VS5eQoxXVFYIoJ03lwBmAg3dqS5KnCxGFSTPS2PVHdziSgMIzycgI1Y+iqHsv26RG5Qj 3CMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451598; x=1708056398; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dj+O15ERlc2oNccgk7u96NynUdtw7IljkhhXG+cbct8=; b=t4CsPILCfGYJySPKcKmhzsJQX0CilRjTjwHhpli4I8hbiy3mLi4GGyyI+KoZcxQcCR 8LaRGheM3HsXpzJWYV/2GRZhrTsgYeX2xcE8kYHCUtsbWr5q9bqq8/xXZ7BHjSbex1Ke A7JR6J3kMojIILL6ciG9jEw9EGjhf1AT+3S6WziBz5N2QkfXeXUc7gx1IFQde/45RAWQ 3njx3vEh4RazrOH93oVfavyimuZmq4FQxwKHianqavFRvQBNa43vDvhd5HZpVilxdMbh +cNizZDBk1dc75MNe+c8bN60uaPB+jfr+xciSgCRksbfXgvptMgI8k0OJgjh1cGo5J+U cZeA== X-Gm-Message-State: AOJu0YzJGuryfmYeqgXUqO0659+okUB1EQLjuEuLZYQBSCrHx9JqB2D0 2OEbLikzGTa7gOW+KSr/yiOOVjNQw5jTG2bMgQwSQNXUEUtTqHxd X-Google-Smtp-Source: AGHT+IH2IsgScbF5ztEZpq6NjR9/2tA7q8l7Ix0S1q4VHpCiUufQi2AhA2u+G2VsbGb/+uLf6rgQHA== X-Received: by 2002:a17:902:b68c:b0:1d8:fae3:2216 with SMTP id c12-20020a170902b68c00b001d8fae32216mr419187pls.35.1707451598204; Thu, 08 Feb 2024 20:06:38 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVCvhcfuaPez5OC/0K6lsZftyKKLtOFjodjSkRdPq0l1nidkiVoJHU0Trfc2t9CHoxVoGuR75BnUBtDC7gLBQ5y1kSWyuyg0x40itdsIGSi0vL6t/hDpiz3VGWvzx1E7jyVPl9K9LQX+7fbCFhxUC3CdIANDI84yBGyKa04JykPcZxl05c6A0hQNL7dkYjfPxslz4jAHOWN1SA3/EFocm4/Y4mzyEFTltE88QxdJlxI/+lGIt1LRaqh1R6CISaCy56IDDTUokGPy6rscJPCxBOK6mSWq+iGAF9teb5RqL2aAEQNZwCC6Nscq4hnc7lq7PODCpjmijWqNsxvLhCBkxJ8QMIxLowrFG/g1w/GsVw1Akxvfy/DcA== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id lh16-20020a170903291000b001d9537cf238sm538602plb.295.2024.02.08.20.06.36 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:37 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 06/20] bpf: Disasm support for cast_kern/user instructions. Date: Thu, 8 Feb 2024 20:05:54 -0800 Message-Id: <20240209040608.98927-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: k15sb1zjzp44s1fww45hi4txb9uwstez X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: A25ACA0013 X-Rspam-User: X-HE-Tag: 1707451599-314193 X-HE-Meta: U2FsdGVkX18aq2Ya9QzXqVUDfUsNRsjRU9S6xRk9V8ev5TBvy50b7uHY/HeXxVRmRccx32pEXKCr3LJ0eVsp8KiZna2H8cQuusbIsiyWWSFSga54ePaRH2/HCbFGu4ndG2Aom4BHqDJLF6PLlAJdKGOPxybmmolX3lRCGk7vELpRj1xmhQPmARFClkCVH2euIvFMTeAaOTRqL2Irs8cEYWSvOA7AA1NMi8iOdUhGRFAqj9aFY6bc8gqgVqqiX9Jo/zKXv92es+FCnqExJijwD0tYvAX16daXCEhMGl2xJw6+V5Hu0qJpGonO98sMctGVHV0riwHxAPRwW6dXOv2S4TblPo+46teDu+dQF8cTZwT2Y5rxVa06qqlL3bW6DAVxVEK1ij9+7vYDQVkDrZiyjxSLdlRJrmGsmzC0NNrsqcDSeDPGBVDKyl+BS7n97/HlRI1Fdn8N1wMBq/ftQrTk4u153Ls/Cq4rMwXmWvY4tFoUsM0BWFKKPw2XiqIQ6AG5KspWkxULf6TS0HzVftiFHMmgHytexO2qO/zm6wlF1Qeuvyy6M3rfTqrvV+1d56kjaR7c4r219dgLqiGgBM20wk+pnXZpB39EaxnC867IZyaqhPPgn4686XyBndWpAVy440QX7Ejo4jPDlrDpgym0RfCtd2G9yy3USqhyPoFCdbrJS3A6Gv+7oOQxZozHcx4M1OKC6RtgBjL89OowIXDjODS9CwB2A21mdhV0xWtuuLIdEcVeG2pH7CBlA0dd9HCs3RvPUqg8i65GVrdlmdLb2Fq732yIZlH+337JwhpUbCWxSjoLE4SUi5WJRreKSfO2XKx3mzmE67lnoch0NCsw9b5yWrCNeSbTHXE4xRBHTvuqpc9W5okK54FuoBCRUpm+sRsCq0Kb6G4JzuF0sTXygDK3zrq0JqKzsT2N8QDm1ZfOc8wcglyNptWwmCmPMv4j6dCe1+UKRAg8G312T2G Ttr7+fR2 /kDtaURqWtUqnNtENby7EPyQEOrN4uNTr8MK+wcyU7l6mBbq88mgfhAuIzVqkw2lcsGA8Fc/zWtKONI3vq3xGd8o7Bf/AGOSF1g8JNJqcpV2LyTrKX33Ma5lgiv6p0CiePzEI+zBUwRKQA4aRx4Qk2fRC1uIH6qsQy5I34BGI4gigR9eVOW2ISYM7tx185T9IIsfjXV170zRKWmQabH+YWbY1TTzw6KEhHErvDKQBvwZbJYxPCbuM6ADe/Nd92RwL78rjGLnq2X9I+jKC5BcTpMROXjeFJAbHLaiHXW9brwkaE0R89EXCbYUwJpaHjToz2BYztRbn64VfA61bQN16nJ5UFMcjCoXHS78s5KuWZTtvUWxpzMCDtzdLrrTVovMVdOfM0ohe9NmpPnUUNigYT9op6mqIsEv/04X1UmqxRqnd0O/vtfct1VsILzgT7E67eD0W0B8q/AgMgz9P0wWSKALwqWoa8KY9NLZe9QYvwulBJgs/QOwRglufsEAMHTPljdDtKHKa4B39SXxxyIvZ5b6lvnLKVYsczugP77qJJU3QMQSv5WinBDNzfiB+eWsmiyNhXNNObXwUVV6a1k6jt2h/uVB5uA9cPQe5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov LLVM generates rX = bpf_cast_kern/_user(rY, address_space) instructions when pointers in non-zero address space are used by the bpf program. Signed-off-by: Alexei Starovoitov --- include/uapi/linux/bpf.h | 5 +++++ kernel/bpf/disasm.c | 11 +++++++++++ tools/include/uapi/linux/bpf.h | 5 +++++ 3 files changed, 21 insertions(+) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index f6648851eae6..3de1581379d4 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1313,6 +1313,11 @@ enum { */ #define BPF_PSEUDO_KFUNC_CALL 2 +enum bpf_arena_cast_kinds { + BPF_ARENA_CAST_KERN = 1, + BPF_ARENA_CAST_USER = 2, +}; + /* flags for BPF_MAP_UPDATE_ELEM command */ enum { BPF_ANY = 0, /* create new element or update existing */ diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 49940c26a227..37d9b37b34f7 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -166,6 +166,12 @@ static bool is_movsx(const struct bpf_insn *insn) (insn->off == 8 || insn->off == 16 || insn->off == 32); } +static bool is_arena_cast(const struct bpf_insn *insn) +{ + return insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && + (insn->off == BPF_ARENA_CAST_KERN || insn->off == BPF_ARENA_CAST_USER); +} + void print_bpf_insn(const struct bpf_insn_cbs *cbs, const struct bpf_insn *insn, bool allow_ptr_leaks) @@ -184,6 +190,11 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->code, class == BPF_ALU ? 'w' : 'r', insn->dst_reg, class == BPF_ALU ? 'w' : 'r', insn->dst_reg); + } else if (is_arena_cast(insn)) { + verbose(cbs->private_data, "(%02x) r%d = cast_%s(r%d, %d)\n", + insn->code, insn->dst_reg, + insn->off == BPF_ARENA_CAST_KERN ? "kern" : "user", + insn->src_reg, insn->imm); } else if (BPF_SRC(insn->code) == BPF_X) { verbose(cbs->private_data, "(%02x) %c%d %s %s%c%d\n", insn->code, class == BPF_ALU ? 'w' : 'r', diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index f6648851eae6..3de1581379d4 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1313,6 +1313,11 @@ enum { */ #define BPF_PSEUDO_KFUNC_CALL 2 +enum bpf_arena_cast_kinds { + BPF_ARENA_CAST_KERN = 1, + BPF_ARENA_CAST_USER = 2, +}; + /* flags for BPF_MAP_UPDATE_ELEM command */ enum { BPF_ANY = 0, /* create new element or update existing */ From patchwork Fri Feb 9 04:05:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91DD2C4828F for ; Fri, 9 Feb 2024 04:06:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 213788D0003; Thu, 8 Feb 2024 23:06:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19D858D0001; Thu, 8 Feb 2024 23:06:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0B3F8D0003; Thu, 8 Feb 2024 23:06:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CF19C8D0001 for ; Thu, 8 Feb 2024 23:06:45 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A2D9D140FA7 for ; Fri, 9 Feb 2024 04:06:45 +0000 (UTC) X-FDA: 81770929170.07.A35563D Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf25.hostedemail.com (Postfix) with ESMTP id B453EA0008 for ; Fri, 9 Feb 2024 04:06:43 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IptkdW97; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451603; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r96UcfCBLUTrZkh0XKcSEOthU3NUKwaxo/e39vpFxwI=; b=a1Oq8brG/YREw6FI1rbD91NNCnaWyqJJEjnq3/0eZctzxEugDTCOfOJrriNkLM+rCq3S+g pFSR28db6y3UyqyC/SAA0hHac7qeZsfHgI5/sInWBiaLnUKuTiEr7CkZsTP4463HBq07bW +XJ1t31LFHFvXrJXmFFZKsy7NVGja04= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IptkdW97; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451603; a=rsa-sha256; cv=none; b=MalDOboXgVbBlw85PYQkVbyeNJhOofEc5vyc1ZuM8JuMc1iu+JOJGF7W6mNtRnckAvlrQu zM/GtGlSUAdTW/8thG2IOVuo7+1MLVD6EQ7FhUGnLd3hkhmPH+M736mweRPq32lryQLskn f8EkJZd4/PmdzUXJgIM8vuyQ3ICuwvo= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1d7354ba334so4475235ad.1 for ; Thu, 08 Feb 2024 20:06:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451602; x=1708056402; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r96UcfCBLUTrZkh0XKcSEOthU3NUKwaxo/e39vpFxwI=; b=IptkdW977u1kX2YMW/u6+kHlf/9W9cjt2bOgjEmN4SfP0LRfDDTt33a98Vx66snJ2X G09ckcuRflK6rUZm+njuARwgHmwILRPpMk2gXHWJbs5qRLqcPJLN8grAMvJNJcW/+71d 2ETuwWmRIO+hbEipIP8DEq7NkoBlePyjz5F83lrxBjRLH9+v8bqIZK2WVWYkBD8EvE++ YREvzSkr+/1nJdsDMnPvyqPQq82pF8Lba1INRRw9NE9L8ifRkTP97hSfI4RacKGa4KJj ifcttrOCkmOz//HOAapmu/H9WqHuWS/dV+nIViX3YJB8f0/WC+BUSSpwPzDJCSvpQmog DeZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451602; x=1708056402; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r96UcfCBLUTrZkh0XKcSEOthU3NUKwaxo/e39vpFxwI=; b=Ao7PSIGWWjPFFCAClev1CvRZKTh5dPYoi9m+v/oeXKNG/xpVFzbFtT/0hKceON2eJo k5FZxO0K8Rsssq/WPhHa9PXEoAY2WENOWbDZWEy/FcR4OsbvrpjFzW5S9D3Wg/4h6alu 6Fo/IzLgrNCeh6heUuGCuesCYKeSCF6RK62FV7SV/svMc/j3glFLaW1aylSB4WWeBD/Z w3dKLGpuv2r6fokiIPU22WNCxjiaJpSplCJtkDBaF22F3laScMiObtV+DooAX4RGJR/O qeEIqZPOTvdbmBMKWNnCNGipAU1ZM5ircORkWO7JCAlLWW6CCsEYZVY6g21kTYHFo19T fWqw== X-Forwarded-Encrypted: i=1; AJvYcCVK3H5ak2T9pnpz/nQ27LiXMtsH9IF3dM0mPw2h08Zba+jIYhMgSyGmUGiV6srgIAT/IABu0CtC/zgAwl9tCBGT3gs= X-Gm-Message-State: AOJu0YxjHAT9OnAPbXlViqfobHUGpObJzTshVu+7VR91pLU2CzbWNq9R GJJCHHUe+omidYbIBoefUb+8nIzckJl2NQuCIVVsPm7Yc+/HD+68ljnWutl4 X-Google-Smtp-Source: AGHT+IF1XX2MhSM/LO2CyghiHgp2M0a1axk082AxrJPPTkzUSm6fuDm951Y0QMtQRmbzz1hhWppIoA== X-Received: by 2002:a17:903:2287:b0:1d9:a29e:ff1a with SMTP id b7-20020a170903228700b001d9a29eff1amr536162plh.34.1707451602554; Thu, 08 Feb 2024 20:06:42 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCW7TFVefsK6YiLeFqnRQ+ngnAM3Iz9zc36ph1O0+RtguHwMKRjvVOYFFLDzlhm7GpceHe4QWZ/ukSU1k2THoo8Yh5tiJCf+N5NzpDvOH6PVcXVvr1DLsYcVrwwpgMureD9I4F9yvnBekkrpH4nQ5rAWXf+mvkPKcdk/4nhOKtazibWbS9zFESguuCqUyauUmA3uh3i7SXb3/y+uDn7yKkDIU3SU9QMGf6xHQZubzvwfv775vfmvAMkhvi34uCws6gTklKGRLS/S9Z1Xf7sGpXvGFH97phRnv3qgVxLHYnVzl5SX8QWnlFoVqxqpId9mbWPB5A7XDgRAeMZeixCaLHNbe6940Tr3mPQh4OCOoY2p4MV1oncWiw== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id kk7-20020a170903070700b001d8f99dbe4asm548739plb.4.2024.02.08.20.06.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:42 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 07/20] bpf: Add x86-64 JIT support for PROBE_MEM32 pseudo instructions. Date: Thu, 8 Feb 2024 20:05:55 -0800 Message-Id: <20240209040608.98927-8-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B453EA0008 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 6z8npnm4hbhnaumz8bhaoddmpk9n1kqx X-HE-Tag: 1707451603-1854 X-HE-Meta: U2FsdGVkX19B7HjZRm3Rhmwlt+u3XNYPBYyhMIBizQrre9XPqsTckCyQZXKJ9DB8CPxOWfJ4mrjWLUT7Aas/snIuX7Y+d3OYxVWqTtgpeMe7tX2rwhL2VxuZwEN19J14Cpp7lN0NdV8TEUqcKc1osB7enoMEfGQMl8eklKiyQKJ5tA3er3m4Ma71vS/qQRT/JzHNqMsouLbTm8mU4XaBjMDdX8pzBt8B7rWR0qe2PqbGwIESqGrMisTTZKD6QLPyHKfw0PNB1jRoDuypEFSdjuJX1gd1ew9Bk3RIwA9WL1SecSvJ54S3mhclaaLQQd+0c0aAbQjtJ+n8SnzxDcjpC7L1G9OzuJxczSQU7IqKz04acfI/H84lvZDbwhF0DWuehs1bqDVO3syR0/AQe0bC+cCaNtaO8OcFMYkXSMr43iV7qkTm8qGr0Btyr6t6CqXPyP+c4tWaG41Hetz+p6GlEaFsJuaQMgHdxr/iVhDwRdpnIYQl2GBz+04zwtfEvqUG8lyBsOX8YgyT/BAMbdrBBORO0yo24PqUKYlskLvcWRa88XPWRdgCz+ZAVbUiveLr6riWsQ/JwQvfwLXkorMTxsGK1BbSXoF0bM7oMGs3GlPPmTEwYfjELnNKDb/KpcntfP+qNDseeeK/ZGPazCTldwN1wzPzoTl95QtL6STVadyBxA4GqpBjoFjEgSqWUWRwxg9M2dT6VDanXJQ1VZ0idNMTrRUaqgO7SFN7bx46meODj71A6jPZlT8CPgyb5MUMTXjqKa2rCWY2Ba2QpySB/JhxLU8LRZRqfJBCd1z+H2r916G7IgZGwWW2x4MreswwC3jioq55iVHL/nFlvRirG19mkJZhKfea7L/PM3JeDx1aCnbcKp8CR/TvT8uNBxyxQz2wK54RciyyZlsFK6EgJ6PpAuxb/q0LoTw87jY6i/dgQwvzCQrb3W07AZOxf2kLat3t8QucnpkSoY9PRZv X8EMkd2W NOgQ/5hrLOHR+0gCKAsA2fU4XstoxYGsPPT1Icgp9fXOj2JCy0kNS069vEIbX7GBbsqCJqmxCeKUQq+iqMM0DRUk/6eP1qLcv/n073Hojf/DZZWi5aL50Df1MWwmbXOXxk6N3W6zfS36FANpw0lSplo/wyuDf5t0S5jNbLtD6vY5pTl5eS4Y2+j1ty2ZXfIXIA7FM9FzkYaVe3Xm9+87YEUHC3J3gvp8UPk5mXfqjvr+1GJfsQ+qNxb5yTtE3ZMzGHqettvZEgbE5ymJta+EFT7+LybFFAkb+nIlDz+f/eTJby3uVlwyhsOd0FGnKxURTbIiju9Q7RXJNXkz/vyBQqk0Pw9bL1geFH2MY6Y2w8mt2+bMuBrX3ZWDVtquqRnN7085uqXVYQOKvGpYlAE6NAkCvOBT3+SlPyHO0anzK8fkcYELuGLNVOf7klhDFsY1JDu7w4B3+ttTOyZ1k4s+2TlSjSfYP20bJFu731DkdYDiYpyrnXqxqb+/V4sRNYvIy8pxy2f5wwJMXhZ2/mHd2vLeK6/HOLIP6oIK4ab1iPMPpSYbDYfHG8KNtGUhodbZtPM2jYiPVwmugjDQiaDFanTp0G225Kezv3rD/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences: - PROBE_MEM has to check that the address is in the kernel range with src_reg + insn->off >= TASK_SIZE_MAX + PAGE_SIZE check - PROBE_MEM doesn't support store - PROBE_MEM32 relies on the verifier to clear upper 32-bit in the register - PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in %r12 in the prologue) Due to bpf_arena constructions such %r12 + %reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time. - PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed. Signed-off-by: Alexei Starovoitov --- arch/x86/net/bpf_jit_comp.c | 183 +++++++++++++++++++++++++++++++++++- include/linux/bpf.h | 1 + include/linux/filter.h | 3 + 3 files changed, 186 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e1390d1e331b..883b7f604b9a 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -113,6 +113,7 @@ static int bpf_size_to_x86_bytes(int bpf_size) /* Pick a register outside of BPF range for JIT internal work */ #define AUX_REG (MAX_BPF_JIT_REG + 1) #define X86_REG_R9 (MAX_BPF_JIT_REG + 2) +#define X86_REG_R12 (MAX_BPF_JIT_REG + 3) /* * The following table maps BPF registers to x86-64 registers. @@ -139,6 +140,7 @@ static const int reg2hex[] = { [BPF_REG_AX] = 2, /* R10 temp register */ [AUX_REG] = 3, /* R11 temp register */ [X86_REG_R9] = 1, /* R9 register, 6th function argument */ + [X86_REG_R12] = 4, /* R12 callee saved */ }; static const int reg2pt_regs[] = { @@ -167,6 +169,7 @@ static bool is_ereg(u32 reg) BIT(BPF_REG_8) | BIT(BPF_REG_9) | BIT(X86_REG_R9) | + BIT(X86_REG_R12) | BIT(BPF_REG_AX)); } @@ -205,6 +208,17 @@ static u8 add_2mod(u8 byte, u32 r1, u32 r2) return byte; } +static u8 add_3mod(u8 byte, u32 r1, u32 r2, u32 index) +{ + if (is_ereg(r1)) + byte |= 1; + if (is_ereg(index)) + byte |= 2; + if (is_ereg(r2)) + byte |= 4; + return byte; +} + /* Encode 'dst_reg' register into x86-64 opcode 'byte' */ static u8 add_1reg(u8 byte, u32 dst_reg) { @@ -887,6 +901,18 @@ static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off) *pprog = prog; } +static void emit_insn_suffix_SIB(u8 **pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off) +{ + u8 *prog = *pprog; + + if (is_imm8(off)) { + EMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off); + } else { + EMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off); + } + *pprog = prog; +} + /* * Emit a REX byte if it will be necessary to address these registers */ @@ -968,6 +994,37 @@ static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +{ + u8 *prog = *pprog; + + switch (size) { + case BPF_B: + /* movzx rax, byte ptr [rax + r12 + off] */ + EMIT3(add_3mod(0x40, src_reg, dst_reg, index_reg), 0x0F, 0xB6); + break; + case BPF_H: + /* movzx rax, word ptr [rax + r12 + off] */ + EMIT3(add_3mod(0x40, src_reg, dst_reg, index_reg), 0x0F, 0xB7); + break; + case BPF_W: + /* mov eax, dword ptr [rax + r12 + off] */ + EMIT2(add_3mod(0x40, src_reg, dst_reg, index_reg), 0x8B); + break; + case BPF_DW: + /* mov rax, qword ptr [rax + r12 + off] */ + EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B); + break; + } + emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off); + *pprog = prog; +} + +static void emit_ldx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +{ + emit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); +} + /* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) { @@ -1002,6 +1059,71 @@ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* STX: *(u8*)(dst_reg + index_reg + off) = src_reg */ +static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +{ + u8 *prog = *pprog; + + switch (size) { + case BPF_B: + /* mov byte ptr [rax + r12 + off], al */ + EMIT2(add_3mod(0x40, dst_reg, src_reg, index_reg), 0x88); + break; + case BPF_H: + /* mov word ptr [rax + r12 + off], ax */ + EMIT3(0x66, add_3mod(0x40, dst_reg, src_reg, index_reg), 0x89); + break; + case BPF_W: + /* mov dword ptr [rax + r12 + 1], eax */ + EMIT2(add_3mod(0x40, dst_reg, src_reg, index_reg), 0x89); + break; + case BPF_DW: + /* mov qword ptr [rax + r12 + 1], rax */ + EMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89); + break; + } + emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off); + *pprog = prog; +} + +static void emit_stx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +{ + emit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); +} + +/* ST: *(u8*)(dst_reg + index_reg + off) = imm32 */ +static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm) +{ + u8 *prog = *pprog; + + switch (size) { + case BPF_B: + /* mov byte ptr [rax + r12 + off], imm8 */ + EMIT2(add_3mod(0x40, dst_reg, 0, index_reg), 0xC6); + break; + case BPF_H: + /* mov word ptr [rax + r12 + off], imm16 */ + EMIT3(0x66, add_3mod(0x40, dst_reg, 0, index_reg), 0xC7); + break; + case BPF_W: + /* mov dword ptr [rax + r12 + 1], imm32 */ + EMIT2(add_3mod(0x40, dst_reg, 0, index_reg), 0xC7); + break; + case BPF_DW: + /* mov qword ptr [rax + r12 + 1], imm32 */ + EMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7); + break; + } + emit_insn_suffix_SIB(&prog, dst_reg, 0, index_reg, off); + EMIT(imm, bpf_size_to_x86_bytes(size)); + *pprog = prog; +} + +static void emit_st_r12(u8 **pprog, u32 size, u32 dst_reg, int off, int imm) +{ + emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm); +} + static int emit_atomic(u8 **pprog, u8 atomic_op, u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size) { @@ -1043,12 +1165,15 @@ static int emit_atomic(u8 **pprog, u8 atomic_op, return 0; } +#define DONT_CLEAR 1 + bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs) { u32 reg = x->fixup >> 8; /* jump over faulting load and clear dest register */ - *(unsigned long *)((void *)regs + reg) = 0; + if (reg != DONT_CLEAR) + *(unsigned long *)((void *)regs + reg) = 0; regs->ip += x->fixup & 0xff; return true; } @@ -1147,11 +1272,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image bool tail_call_seen = false; bool seen_exit = false; u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; + u64 arena_vm_start; int i, excnt = 0; int ilen, proglen = 0; u8 *prog = temp; int err; + arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); + detect_reg_usage(insn, insn_cnt, callee_regs_used, &tail_call_seen); @@ -1172,8 +1300,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image push_r12(&prog); push_callee_regs(&prog, all_callee_regs_used); } else { + if (arena_vm_start) + push_r12(&prog); push_callee_regs(&prog, callee_regs_used); } + if (arena_vm_start) + emit_mov_imm64(&prog, X86_REG_R12, + arena_vm_start >> 32, (u32) arena_vm_start); ilen = prog - temp; if (rw_image) @@ -1564,6 +1697,52 @@ st: if (is_imm8(insn->off)) emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); break; + case BPF_ST | BPF_PROBE_MEM32 | BPF_B: + case BPF_ST | BPF_PROBE_MEM32 | BPF_H: + case BPF_ST | BPF_PROBE_MEM32 | BPF_W: + case BPF_ST | BPF_PROBE_MEM32 | BPF_DW: + start_of_ldx = prog; + emit_st_r12(&prog, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm); + goto populate_extable; + + /* LDX: dst_reg = *(u8*)(src_reg + r12 + off) */ + case BPF_LDX | BPF_PROBE_MEM32 | BPF_B: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_H: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_W: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW: + case BPF_STX | BPF_PROBE_MEM32 | BPF_B: + case BPF_STX | BPF_PROBE_MEM32 | BPF_H: + case BPF_STX | BPF_PROBE_MEM32 | BPF_W: + case BPF_STX | BPF_PROBE_MEM32 | BPF_DW: + start_of_ldx = prog; + if (BPF_CLASS(insn->code) == BPF_LDX) + emit_ldx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + else + emit_stx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); +populate_extable: + { + struct exception_table_entry *ex; + u8 *_insn = image + proglen + (start_of_ldx - temp); + s64 delta; + + if (!bpf_prog->aux->extable) + break; + + ex = &bpf_prog->aux->extable[excnt++]; + + delta = _insn - (u8 *)&ex->insn; + /* switch ex to rw buffer for writes */ + ex = (void *)rw_image + ((void *)ex - (void *)image); + + ex->insn = delta; + + ex->data = EX_TYPE_BPF; + + ex->fixup = (prog - start_of_ldx) | + ((BPF_CLASS(insn->code) == BPF_LDX ? reg2pt_regs[dst_reg] : DONT_CLEAR) << 8); + } + break; + /* LDX: dst_reg = *(u8*)(src_reg + off) */ case BPF_LDX | BPF_MEM | BPF_B: case BPF_LDX | BPF_PROBE_MEM | BPF_B: @@ -2036,6 +2215,8 @@ st: if (is_imm8(insn->off)) pop_r12(&prog); } else { pop_callee_regs(&prog, callee_regs_used); + if (arena_vm_start) + pop_r12(&prog); } EMIT1(0xC9); /* leave */ emit_return(&prog, image + addrs[i - 1] + (prog - temp)); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index de557c6c42e0..26419a57bf9f 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1463,6 +1463,7 @@ struct bpf_prog_aux { bool xdp_has_frags; bool exception_cb; bool exception_boundary; + struct bpf_arena *arena; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/include/linux/filter.h b/include/linux/filter.h index fee070b9826e..cd76d43412d0 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -72,6 +72,9 @@ struct ctl_table_header; /* unused opcode to mark special ldsx instruction. Same as BPF_IND */ #define BPF_PROBE_MEMSX 0x40 +/* unused opcode to mark special load instruction. Same as BPF_MSH */ +#define BPF_PROBE_MEM32 0xa0 + /* unused opcode to mark call to interpreter with arguments */ #define BPF_CALL_ARGS 0xe0 From patchwork Fri Feb 9 04:05:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B06A2C48297 for ; Fri, 9 Feb 2024 04:06:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 40E8A8D0005; Thu, 8 Feb 2024 23:06:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BE608D0001; Thu, 8 Feb 2024 23:06:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 238BD8D0005; Thu, 8 Feb 2024 23:06:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 111C38D0001 for ; Thu, 8 Feb 2024 23:06:50 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D8DCB802DE for ; Fri, 9 Feb 2024 04:06:49 +0000 (UTC) X-FDA: 81770929338.14.9B36498 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf19.hostedemail.com (Postfix) with ESMTP id F15051A0011 for ; Fri, 9 Feb 2024 04:06:47 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BaQ1HbKh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GxmcZQXB6hILUuERStuFRZ0MPT4uFMATBLnu6rXBqLE=; b=a7NNsrQXKlR8rhOFWaF2NfYAzQSQVYCfqzniPfBCXaG+7uHn8FPeQwfdWMvwB9fS0ij7yp LvdyYoEtdhgARMOZTnQ4g6GO0UW31Jh6GNEqTxx9URrNSS1e7C/QDaJQBdORr52A1CoSYC dEBq3fsO7k8qiTAJCUOM2Ouyvq8bX94= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BaQ1HbKh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451608; a=rsa-sha256; cv=none; b=Rexb/6uYQMNCSShU0xtQXEWOileVf3/kBtcQgHDmsMO2cZdKgif7m5nzgX1yZtsoKHuGO1 TdllrYznbIP1v3bJFJIg5kOP5lRIgVc++Rmd9Ec5xY3ZVFIjqdWn78W7zs/Eq6uQmqr/aN HyNYqM0MqVQUZpCGNEIZHQkIbxiTExA= Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-5c66b093b86so1153443a12.0 for ; Thu, 08 Feb 2024 20:06:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451607; x=1708056407; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GxmcZQXB6hILUuERStuFRZ0MPT4uFMATBLnu6rXBqLE=; b=BaQ1HbKhYZ312r4b8R231508PO8kDj1aA0QeE0Q9SafO+lzm9Ke9qq4KbIoArkNTgZ 3vWe7w6qBGjb0jCd6YiIU5T/ToKTHRpuOLcGG2R3UWv1qsaZzg4nkDF7xJlQoXRRFLY7 GgrE4BPi+L/qUb0+sWiGTVp556H3i9oPkDbs+WAn31JxZPPK0wPKqgU+mHa3DhHFoQXb 38fax+SBlW+pc944kHRgyt/Q4pCght7cWMLZ6kDAP3AB3ROCyTF1GyEIGWMURRWkrcwn ETUSCNjDv/1AoSRXgBoI4kBJWNU1kEgmJVDXYYjUpbTcgXchDQ7JA/rqjIJoEeYlTOdP jcTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451607; x=1708056407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GxmcZQXB6hILUuERStuFRZ0MPT4uFMATBLnu6rXBqLE=; b=aPkYKgdAjEdmf3djJmWAFp5MZ0cUPQlg/fEPxDpCOGOj8ZJfTdfvqkbW0Bd4g4wirM zfWzqKQDpCVQaExyB3z97TEfOduudy/ywfFZkEg1v80ow2Q+BtkMgkV6YYaNTne8BvBz 853MUNaAhaG7TDZsiFjPD3hemi8KLXoOFTcq6adhwKgbMsKkL78xBpqnYiUYjnmREFz6 7IUWpbK/JHxxNmvQT6Nqxza0bSPW4I31CwTw1VejE7uKa2idpDxdG+uRHjxOEpzr0uIf ACaZ20wytAZ5JuGtsT63R8x6YiatRtmzDmhZZCd6ef9C8SMQmjLNheI268cnTE/RYvWN 6L6A== X-Gm-Message-State: AOJu0Yza3K2C2Hm2MXPMNp52yrBYBMG1BJaXSTTV9kE7uIEmyLTUDB9j iI4hcbD84DJYD8kjdE9pbR3afZwdq9sOAu7bV2Znwxz+QriIYQQv X-Google-Smtp-Source: AGHT+IHuqaEsk6hY3MyoaXGt78Tnlu+oap9lHuWJS3aqvunCJMIGEmoL01w5jyIJ2MTY+5Q0V0YZ4g== X-Received: by 2002:a17:90a:c291:b0:296:fff3:cf33 with SMTP id f17-20020a17090ac29100b00296fff3cf33mr1016964pjt.8.1707451606688; Thu, 08 Feb 2024 20:06:46 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWYw8ifD9te6TKOQruqSWHsfeyVWflUrl1SO2obE7M4KjqFDUY4tyAd7G+fW4weZ61H3tBeM7iE9QrtVUgUuqPkA3BqbcV/uqVtAYEMOdzBtNJ7moG+4HEo4Yrf8pUTvj0KhqQp2Ld60XtdpyNfIMydcJU0tnH+GgOO/Ir40yeps53JAqQY/ou32WGYozZ2TQUdJn/A04Fs+czxvy4plVwfCv1hwqMwa2KJUO7JBAlDbj2wEEWXbOZcNbE/o8UjskjgDrBXgRZh0EXhh2tx/MKhK2g1C+sdFMWpUb6Rk0/njLS/4TTplFl0/c9Po2lLk7OZzJFytqqZ5A6DCtSe9uqoRXyqYMZDfBGZ7eXuH9eKfEt9KVfxjg== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id gz21-20020a17090b0ed500b00296b90d93absm631254pjb.29.2024.02.08.20.06.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:46 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 08/20] bpf: Add x86-64 JIT support for bpf_cast_user instruction. Date: Thu, 8 Feb 2024 20:05:56 -0800 Message-Id: <20240209040608.98927-9-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: yq9s13nbnar6fg1g1x18kftbkz5ehs8p X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F15051A0011 X-HE-Tag: 1707451607-987369 X-HE-Meta: U2FsdGVkX19L7wEj9d6HREYxFWf26/dBYtvOkMXaMRpmBcSul0eIoKZHRpTiLGiwsb5diNrVOOHHXniQfFVF/gg33LsUB5BHm1ggwE+RS79xnMVuzMLsiTPTJjFgHeK1fO6eHVrMKuz1inp4cBrTUfnHFYoFGPUva7yysAhuJIgqytbWxrZvDpnY8VwLf/KT+zp+yoHjzOMZ0yTu9c48DgKkp4WfDCHMMhIb7tdTABSUK/h6hSK2sKj4hn25kr/4f4Y9jUZfYhE+tqohN0isHRivUCtY99L/vqZO+aCDcMM/Vpzaam4fqcevHqrT0Ncp7Yoo8Xy41aGktP+1gEhonfxkMNLwPBO5bSTxhIC0fk2MdX1QyeyC3Sss1/5lTR9vT2hlk2MRNLkNDM2xatq6n268YqolaXmX0qWSkmDI4ixXdGqnOmRv162j2p7tgkj9PrjPZdm3qgr3RCkJN5svjNGeStsIGEP0NAyK45ajAOUTRRltmKO4Fd+gJ9uxUpxz3MqxVeYtiInwMgu0avua9wGRikQaVQAml0batvw5RLJ9DAK0e5NNJsZ9cMPqLHmw1H1fiKa+H6y2xcDq/3m5b2OthYo3KszZKhgNYtGfDfPTNoLB1GaEVQPO3hfMaE+c2YfSjIDdu7nKf5v9kCYp05ck8qdk6oUIrx+igo3kC3j91Ug/g1NRthfGF6g4OJWlntKTNoynqcrP3eRF6FDBkMFJ9RdRCDq7CrrTXq9ezRXb0Fj2GobneYtm8RTgV2n9EU7AKr+yJfhobgpvXokyUIfvaZLse4aabVuGeE/7y3f7j0UpHoE7rl2HqIIHvwDziCRuHTppF53dVkesjqYHjFPG9SyikYNr9Zgx0havaFIobCAmhh+JASsmy0hKhQxp0+6gCSeoHeJ2mASvKo/+1alRQGrIEEWkPF1xUONBzZg2580vz+DFj2zxK6T51b7kea2OjUinHEGPgud3DU6 AX3+HrPg pBzzrlta3/TDTNFfsmcxPOKoZmvX2ggBH3tBNUKTsnHFAEDrR+EYuZZbVKwB+NSgrVA53kmtf2JyekAyS4TtPvrhC6JYP3a8ijKoSR/lAxe4sU4Pt74hK4v7XGcpIm2GLR7QHJFWqN2dvmeVlYgjA8LsylrlarhpyanMuy1lJ6QqZLNxWsTlo5ajcrQrDUGv1d+vVPi3bLzsPVcyzfPG0LLsx169MowE63mKpkC9zkCbKpGHHfmutrprQr0We6ZgzeM/Px2b5cw3BcPGcYsO1kYUz6VM4tmkgM1vkAgkaH+m2sCNcER4MTun4KCI57uv9BOuPR76IK0TR0zZStCS9hs2VKN56WQK5ef1aoQEG6X+08moxghWwbcaOmH83E2+tttdl0iWlVRhslMiSnKWJpGAfEjyW8Am3MgF6ttepgV7nNQ7WSVZxmOksfcGeatOuXHlsWYrKoPBH+zRvn0upTwzKguYWqvo1bs18DNslnvQHNtdhGRC8aQ4IczKTa/aFep5lZ/ix9j7SNIlK1tTkMUFo+pG/kgB6YUb2ct2vV1oxLzrgQzRqgWtHgu0/vMv+/UVnR5QR3QiJBo5FKT/bTSrTZKykMZBAQq/j56DlFFPG7qmahRB4Z2TptxSx9jEPtYL+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov LLVM generates bpf_cast_kern and bpf_cast_user instructions while translating pointers with __attribute__((address_space(1))). rX = cast_kern(rY) is processed by the verifier and converted to normal 32-bit move: wX = wY bpf_cast_user has to be converted by JIT. rX = cast_user(rY) is aux_reg = upper_32_bits of arena->user_vm_start aux_reg <<= 32 wX = wY // clear upper 32 bits of dst register if (wX) // if not zero add upper bits of user_vm_start wX |= aux_reg JIT can do it more efficiently: mov dst_reg32, src_reg32 // 32-bit move shl dst_reg, 32 or dst_reg, user_vm_start rol dst_reg, 32 xor r11, r11 test dst_reg32, dst_reg32 // check if lower 32-bit are zero cmove r11, dst_reg // if so, set dst_reg to zero // Intel swapped src/dst register encoding in CMOVcc Signed-off-by: Alexei Starovoitov Acked-by: Eduard Zingerman --- arch/x86/net/bpf_jit_comp.c | 41 ++++++++++++++++++++++++++++++++++++- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 46 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 883b7f604b9a..a042ed57af7b 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1272,13 +1272,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image bool tail_call_seen = false; bool seen_exit = false; u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; - u64 arena_vm_start; + u64 arena_vm_start, user_vm_start; int i, excnt = 0; int ilen, proglen = 0; u8 *prog = temp; int err; arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); + user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena); detect_reg_usage(insn, insn_cnt, callee_regs_used, &tail_call_seen); @@ -1346,6 +1347,39 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image break; case BPF_ALU64 | BPF_MOV | BPF_X: + if (insn->off == BPF_ARENA_CAST_USER) { + if (dst_reg != src_reg) + /* 32-bit mov */ + emit_mov_reg(&prog, false, dst_reg, src_reg); + /* shl dst_reg, 32 */ + maybe_emit_1mod(&prog, dst_reg, true); + EMIT3(0xC1, add_1reg(0xE0, dst_reg), 32); + + /* or dst_reg, user_vm_start */ + maybe_emit_1mod(&prog, dst_reg, true); + if (is_axreg(dst_reg)) + EMIT1_off32(0x0D, user_vm_start >> 32); + else + EMIT2_off32(0x81, add_1reg(0xC8, dst_reg), user_vm_start >> 32); + + /* rol dst_reg, 32 */ + maybe_emit_1mod(&prog, dst_reg, true); + EMIT3(0xC1, add_1reg(0xC0, dst_reg), 32); + + /* xor r11, r11 */ + EMIT3(0x4D, 0x31, 0xDB); + + /* test dst_reg32, dst_reg32; check if lower 32-bit are zero */ + maybe_emit_mod(&prog, dst_reg, dst_reg, false); + EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg)); + + /* cmove r11, dst_reg; if so, set dst_reg to zero */ + /* WARNING: Intel swapped src/dst register encoding in CMOVcc !!! */ + maybe_emit_mod(&prog, AUX_REG, dst_reg, true); + EMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg)); + break; + } + fallthrough; case BPF_ALU | BPF_MOV | BPF_X: if (insn->off == 0) emit_mov_reg(&prog, @@ -3424,6 +3458,11 @@ void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke, } } +bool bpf_jit_supports_arena(void) +{ + return true; +} + bool bpf_jit_supports_ptr_xchg(void) { return true; diff --git a/include/linux/filter.h b/include/linux/filter.h index cd76d43412d0..78ea63002531 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -959,6 +959,7 @@ bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_far_kfunc_call(void); bool bpf_jit_supports_exceptions(void); bool bpf_jit_supports_ptr_xchg(void); +bool bpf_jit_supports_arena(void); void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie); bool bpf_helper_changes_pkt_data(void *func); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2539d9bfe369..2829077f0461 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2926,6 +2926,11 @@ bool __weak bpf_jit_supports_far_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_arena(void) +{ + return false; +} + /* Return TRUE if the JIT backend satisfies the following two conditions: * 1) JIT backend supports atomic_xchg() on pointer-sized words. * 2) Under the specific arch, the implementation of xchg() is the same From patchwork Fri Feb 9 04:05:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1BFCC4828F for ; Fri, 9 Feb 2024 04:06:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F9908D0006; Thu, 8 Feb 2024 23:06:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4827F8D0001; Thu, 8 Feb 2024 23:06:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D4948D0006; Thu, 8 Feb 2024 23:06:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1C2448D0001 for ; Thu, 8 Feb 2024 23:06:54 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EC78CA0FA8 for ; Fri, 9 Feb 2024 04:06:53 +0000 (UTC) X-FDA: 81770929506.22.195839C Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf26.hostedemail.com (Postfix) with ESMTP id 1A3DF140007 for ; Fri, 9 Feb 2024 04:06:51 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FwGVEuny; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451612; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6orzNV0AjOboinNstixj6Czt+5FkF5rLiP2S91rGkng=; b=ZDTH4SGeWlJAkTqt13TI/V47H5PJxtZ5o9VDkMge1gRJbZsCXTJwITKRKQh+wVwpf7+/Nh vJZy3RRL9U6aRa/+7N3yOF7KATnPffYupEjfdW059pdKPvMIcfBc9iHDmjf6x6SBr6qznZ ffG6yC5DWDYhMPlSwfFZRngpsjDoUjg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FwGVEuny; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451612; a=rsa-sha256; cv=none; b=3/TP9ExRbTD53HjMiC6Dx1NyPBMp6PEX/3NWLErqQN7QybNDqP2z+iGBQt761zyxJeGBsl RPPuc7YjLUOlGIaQLo5Tw0vGu/g9P36UZxuzIsXodbkqNntH4REArDRo5sPgwDTfZM3mB1 akixXkdzoHGdZsf0+Vut1Ku6aadO0nc= Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-295d22bd625so431478a91.1 for ; Thu, 08 Feb 2024 20:06:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451611; x=1708056411; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6orzNV0AjOboinNstixj6Czt+5FkF5rLiP2S91rGkng=; b=FwGVEunyYiVoJvcTPCP7Qvbx89sVMZ2mCIxUUt3lUgvGh1YWFVuvbLg4oWDYAkMfZN XMjYimIyJiGnhilKCQrgnQNikKxzx7JYA67gglU/qHNZTsBPDZipA8b/vRUzpw69dFgN Sa+SfJfcfdR4LcdIV09wTFQwBVioeKccPuPTi4xVOILsyoPwuhdhuNw96WJ1iwi4J2fo s9m4u/dRqNXrbKzHsejnWuFyfrxNCuyRawf57R3LAeX7MH1U8qs/daeyrs342e68qsL2 2McvjM/5VaI+Vy1s8RDgWPbyVb5GNGfTbneUEOxvyLSMka2Gc/ZnWYIIazg0DAg8DOPv W/yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451611; x=1708056411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6orzNV0AjOboinNstixj6Czt+5FkF5rLiP2S91rGkng=; b=rE+xRfZEJ2KlbUUlnlA7QjGvpbuQzy6kbj+5Jn1RTobiDMg1I6aEbEj60xS5Zmuv9E xubdll7kxZEeL3Z6WQh7yn6Jj4YaEcmEOk8tj8h4xwBLFgcZudbBkQWFi1Br9esFpMGU 8A22/Lg338US5OkYSewEFdm20EhShtRhP3okgzeOfwzk8PQYJO+fVUw3TiJ8TtL+cCPp aLH/QzuqKy9Cs0OywxS/SwN3RQ36ABrwGbOFswv26/+RjO+i3ihPSHdCbn9Yw/YoYrP6 ZiBqZ7raiN31L86Q1kLnovNfl5B+cl4u+RzgP306TN8ORT9iBriP13ldJSeR6K1cyCIS TfdQ== X-Gm-Message-State: AOJu0YwpIA0sOg2jUYgVIoxVe+nF14tRIBULN6JEf3FBtQf2s3ImqcWh NhWcJJoIQfSwuP8PWLpZwyekr6fP0EJJQiIJkU8AnAIOMSby4LcxX8kepNK8 X-Google-Smtp-Source: AGHT+IELh8rBH0G9Lv9cd6+SlCU2HHOEdFHvhzYPN8nxJUwbCo29hXvQdLAj23YQAxaPQ6BGyTN6CQ== X-Received: by 2002:a17:90a:ec0a:b0:297:604:1ff7 with SMTP id l10-20020a17090aec0a00b0029706041ff7mr369707pjy.17.1707451610983; Thu, 08 Feb 2024 20:06:50 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVbmgFXdBFBjoBt3ptKvyA1cp6LgeIz61rrJcLFv5JUSuH38/oThZV2vN01bpNPpzSAQQzUFFLy5t84VFxFzRvu6fnUge0QkzDc5TzAwavQYHF/tskPhpRwU9tHxSE9TlxkDOuMxPxVDmYlVq+ESWxWRJAkVDkalPzwjCAUzmE8TAZSi0v2qvrRKnnIQ4mIciNs6exu3NF5zyf5m09UcymKjg5roaQ1+4O9u3IaSB1OjQcxE34V8P4E4nnCwSMCt8W3ZXoaNOZ/HHM2MBZ9HBuzyVwifcAlMUuCLYuZbK7iGpu3QO/o8FG5FX0lHxI0Y69dTchHVCLiMculML1U95RXRRI+BFCuced/EbT6d+8/wKzNVMVjsQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id x15-20020a17090a8a8f00b00296e2434e7esm608017pjn.53.2024.02.08.20.06.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:50 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 09/20] bpf: Recognize cast_kern/user instructions in the verifier. Date: Thu, 8 Feb 2024 20:05:57 -0800 Message-Id: <20240209040608.98927-10-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1A3DF140007 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 613sk3j44ai19g78pyam5hx8wb3oan7t X-HE-Tag: 1707451611-746604 X-HE-Meta: U2FsdGVkX19oMf28Gls5UvjEO/zY8QZrUBoq9spqks0B1KF6h+5cD8K/RtFwymrplAA6aaBrYna52KWd21oy0JMy5Ed435aNhICASyCXuhcn9DgRXbvE0JMLUrkVih844ETcwwMRLJZg2X1lQ+Bpmvq2FpbBTUsGJzW/3ZNoHBUD7/7EWboqfKHAOdlit/jkCndPC9cAwIyee6yEqh9ew7Mr59EKoyun5BOkJx2DIGy+z9dlN8lfmgnfwF9Sh+JusljnMSs25eLqlMVk6VdI3DHcIgC4UWhdu2+GUtjKTNPM2tjE/NWnCnQMQjJJreKIIUldI4YlPzchcZdBizDNsh50xbiZCtFry35ChXgUmvqpvrHnjxRomIX1QfD+0vG8mARFrGbwT5138/BWsSe2UDhNP8ad7C4xuIEQbDtm3BMZ1fAfFqMxXu0nHGsJowqg+8p2F92VbBhRIvvUAYsCk+GnyDIpV/70zQytU6YwVKxC/D5awVrmUqwmS4SQQ2Io+E6Z/mUjFywPCkQ18Ay8NyFHTqB9dJPGex6hy64IR3KrdIFs+L7lNDd2ZjoJhN78VRnJqAxNHkBZ6djwVIMVubXBMC6V5LPRq9ElFtT6+w/C2Jfy4V+Be4yNMeN+rTXIgL3hdH7X28IWmx51w9rBXq6LuXVyAh2z3IenLbf3aAlgXaAGujqfRJdOj8vCS9FieD4r69IRFVdIdBoFuZ5uDENNGG0B08VLZd4rCimJJqxHBghjxpZ06eMpx9kDQiYCVK3e3+mK7QAe9fw6OVZPWmgRlfj10gGbeTXNIxr+GyIODVQnUIdXp1wufR/s9zk2qXhVu7bo6xpAHbXllYoDBEvr9C37aYzF5CmcF7do03ZWMIqT1tmYjvrlxsMb5X7NRn33wh6flfPE1DQ3GdlZ5roOphcGkrZkXOWx7up3+W1etqan3JB1KkMMIFOI6Oh8R3k2dgOTijrnTosEzRK yE3GFkLa F6He1ADM+lEpCJxvy+TiEgww7R1G9K65nrawHtd34IXa85hojv90fhgPKENrX59FZ1bzi4NfuTH5+7femsvMOI9s2uFJm/bqQftLpJ2aGZkWH8J7aOswpN6rWn260tFd0HLTMoEzcG2qV3xY0J4kqkkM/qTB4+pbQ0ZFvZXsS3V9DKxI5jMuNjvTlTEDLERY7ugbBYWoaEFd+l1DeQh8iiCcNNbC+mkEIywbkGfon3qj/MPXuY4w1w6QEsZ+iTLD5b649CmwlIIkTEpa9idQp/5ZQxL2Xpl2zOrNI/8PE8IQ7nSQShcwwpe/djWOh6XuHWQmnrTJ5r+ozRpdOKpgmZqKSMC2mNz42jSZ6ImIK7X5NGIGrkR+7n2VRqJEdYwen7XNsFdvK/7qML7R2oIrk5xKTm4rFHWEBBh+G+WDgqoPVQl+bfwGvfoKlbFB5gVEUXi5pgx0+h7wWlOTyT225CdwHdCIRf1UUJ5Kc/iY75fMLz/5KWaZeZCiLX4bvPImw9646/YSnKy4MnLyk9u3SAC6h/4+fUDsAKtT1STZbzeFw/1YgpRPEg9QqmCvPs9b9QbXIX2ATA63g5fCLkwWoGsTBOY0nSvpGpNS+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov rX = bpf_cast_kern(rY, addr_space) tells the verifier that rX->type = PTR_TO_ARENA. Any further operations on PTR_TO_ARENA register have to be in 32-bit domain. The verifier will mark load/store through PTR_TO_ARENA with PROBE_MEM32. JIT will generate them as kern_vm_start + 32bit_addr memory accesses. rX = bpf_cast_user(rY, addr_space) tells the verifier that rX->type = unknown scalar. If arena->map_flags has BPF_F_NO_USER_CONV set then convert cast_user to mov32 as well. Otherwise JIT will convert it to: rX = (u32)rY; if (rX) rX |= arena->user_vm_start & ~(u64)~0U; Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 1 + kernel/bpf/log.c | 3 ++ kernel/bpf/verifier.c | 102 ++++++++++++++++++++++++++++++++--- 4 files changed, 100 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 26419a57bf9f..70d5351427e6 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -889,6 +889,7 @@ enum bpf_reg_type { * an explicit null check is required for this struct. */ PTR_TO_MEM, /* reg points to valid memory region */ + PTR_TO_ARENA, PTR_TO_BUF, /* reg points to a read/write buffer */ PTR_TO_FUNC, /* reg points to a bpf program function */ CONST_PTR_TO_DYNPTR, /* reg points to a const struct bpf_dynptr */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 84365e6dd85d..43c95e3e2a3c 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -547,6 +547,7 @@ struct bpf_insn_aux_data { u32 seen; /* this insn was processed by the verifier at env->pass_cnt */ bool sanitize_stack_spill; /* subject to Spectre v4 sanitation */ bool zext_dst; /* this insn zero extends dst reg */ + bool needs_zext; /* alu op needs to clear upper bits */ bool storage_get_func_atomic; /* bpf_*_storage_get() with atomic memory alloc */ bool is_iter_next; /* bpf_iter__next() kfunc call */ bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */ diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c index 594a234f122b..677076c760ff 100644 --- a/kernel/bpf/log.c +++ b/kernel/bpf/log.c @@ -416,6 +416,7 @@ const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type) [PTR_TO_XDP_SOCK] = "xdp_sock", [PTR_TO_BTF_ID] = "ptr_", [PTR_TO_MEM] = "mem", + [PTR_TO_ARENA] = "arena", [PTR_TO_BUF] = "buf", [PTR_TO_FUNC] = "func", [PTR_TO_MAP_KEY] = "map_key", @@ -651,6 +652,8 @@ static void print_reg_state(struct bpf_verifier_env *env, } verbose(env, "%s", reg_type_str(env, t)); + if (t == PTR_TO_ARENA) + return; if (t == PTR_TO_STACK) { if (state->frameno != reg->frameno) verbose(env, "[%d]", reg->frameno); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3c77a3ab1192..5eeb9bf7e324 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4370,6 +4370,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) case PTR_TO_MEM: case PTR_TO_FUNC: case PTR_TO_MAP_KEY: + case PTR_TO_ARENA: return true; default: return false; @@ -5805,6 +5806,8 @@ static int check_ptr_alignment(struct bpf_verifier_env *env, case PTR_TO_XDP_SOCK: pointer_desc = "xdp_sock "; break; + case PTR_TO_ARENA: + return 0; default: break; } @@ -6906,6 +6909,9 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ)) mark_reg_unknown(env, regs, value_regno); + } else if (reg->type == PTR_TO_ARENA) { + if (t == BPF_READ && value_regno >= 0) + mark_reg_unknown(env, regs, value_regno); } else { verbose(env, "R%d invalid mem access '%s'\n", regno, reg_type_str(env, reg->type)); @@ -8377,6 +8383,7 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, case PTR_TO_MEM | MEM_RINGBUF: case PTR_TO_BUF: case PTR_TO_BUF | MEM_RDONLY: + case PTR_TO_ARENA: case SCALAR_VALUE: return 0; /* All the rest must be rejected, except PTR_TO_BTF_ID which allows @@ -13837,6 +13844,21 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, dst_reg = ®s[insn->dst_reg]; src_reg = NULL; + + if (dst_reg->type == PTR_TO_ARENA) { + struct bpf_insn_aux_data *aux = cur_aux(env); + + if (BPF_CLASS(insn->code) == BPF_ALU64) + /* + * 32-bit operations zero upper bits automatically. + * 64-bit operations need to be converted to 32. + */ + aux->needs_zext = true; + + /* Any arithmetic operations are allowed on arena pointers */ + return 0; + } + if (dst_reg->type != SCALAR_VALUE) ptr_reg = dst_reg; else @@ -13954,16 +13976,17 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) } else if (opcode == BPF_MOV) { if (BPF_SRC(insn->code) == BPF_X) { - if (insn->imm != 0) { - verbose(env, "BPF_MOV uses reserved fields\n"); - return -EINVAL; - } - if (BPF_CLASS(insn->code) == BPF_ALU) { - if (insn->off != 0 && insn->off != 8 && insn->off != 16) { + if ((insn->off != 0 && insn->off != 8 && insn->off != 16) || + insn->imm) { verbose(env, "BPF_MOV uses reserved fields\n"); return -EINVAL; } + } else if (insn->off == BPF_ARENA_CAST_KERN || insn->off == BPF_ARENA_CAST_USER) { + if (!insn->imm) { + verbose(env, "cast_kern/user insn must have non zero imm32\n"); + return -EINVAL; + } } else { if (insn->off != 0 && insn->off != 8 && insn->off != 16 && insn->off != 32) { @@ -13993,7 +14016,12 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) struct bpf_reg_state *dst_reg = regs + insn->dst_reg; if (BPF_CLASS(insn->code) == BPF_ALU64) { - if (insn->off == 0) { + if (insn->imm) { + /* off == BPF_ARENA_CAST_KERN || off == BPF_ARENA_CAST_USER */ + mark_reg_unknown(env, regs, insn->dst_reg); + if (insn->off == BPF_ARENA_CAST_KERN) + dst_reg->type = PTR_TO_ARENA; + } else if (insn->off == 0) { /* case: R1 = R2 * copy register state to dest reg */ @@ -14059,6 +14087,9 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) dst_reg->subreg_def = env->insn_idx + 1; coerce_subreg_to_size_sx(dst_reg, insn->off >> 3); } + } else if (src_reg->type == PTR_TO_ARENA) { + mark_reg_unknown(env, regs, insn->dst_reg); + dst_reg->type = PTR_TO_ARENA; } else { mark_reg_unknown(env, regs, insn->dst_reg); @@ -15142,6 +15173,10 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) if (insn->src_reg == BPF_PSEUDO_MAP_VALUE || insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) { + if (map->map_type == BPF_MAP_TYPE_ARENA) { + __mark_reg_unknown(env, dst_reg); + return 0; + } dst_reg->type = PTR_TO_MAP_VALUE; dst_reg->off = aux->map_off; WARN_ON_ONCE(map->max_entries != 1); @@ -16519,6 +16554,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, * the same stack frame, since fp-8 in foo != fp-8 in bar */ return regs_exact(rold, rcur, idmap) && rold->frameno == rcur->frameno; + case PTR_TO_ARENA: + return true; default: return regs_exact(rold, rcur, idmap); } @@ -18235,6 +18272,31 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env) fdput(f); return -EBUSY; } + if (map->map_type == BPF_MAP_TYPE_ARENA) { + if (env->prog->aux->arena) { + verbose(env, "Only one arena per program\n"); + fdput(f); + return -EBUSY; + } + if (!env->allow_ptr_leaks || !env->bpf_capable) { + verbose(env, "CAP_BPF and CAP_PERFMON are required to use arena\n"); + fdput(f); + return -EPERM; + } + if (!env->prog->jit_requested) { + verbose(env, "JIT is required to use arena\n"); + return -EOPNOTSUPP; + } + if (!bpf_jit_supports_arena()) { + verbose(env, "JIT doesn't support arena\n"); + return -EOPNOTSUPP; + } + env->prog->aux->arena = (void *)map; + if (!bpf_arena_get_user_vm_start(env->prog->aux->arena)) { + verbose(env, "arena's user address must be set via map_extra or mmap()\n"); + return -EINVAL; + } + } fdput(f); next_insn: @@ -18799,6 +18861,18 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code == (BPF_ST | BPF_MEM | BPF_W) || insn->code == (BPF_ST | BPF_MEM | BPF_DW)) { type = BPF_WRITE; + } else if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && insn->imm) { + if (insn->off == BPF_ARENA_CAST_KERN || + (((struct bpf_map *)env->prog->aux->arena)->map_flags & BPF_F_NO_USER_CONV)) { + /* convert to 32-bit mov that clears upper 32-bit */ + insn->code = BPF_ALU | BPF_MOV | BPF_X; + /* clear off, so it's a normal 'wX = wY' from JIT pov */ + insn->off = 0; + } /* else insn->off == BPF_ARENA_CAST_USER should be handled by JIT */ + continue; + } else if (env->insn_aux_data[i + delta].needs_zext) { + /* Convert BPF_CLASS(insn->code) == BPF_ALU64 to 32-bit ALU */ + insn->code = BPF_ALU | BPF_OP(insn->code) | BPF_SRC(insn->code); } else { continue; } @@ -18856,6 +18930,14 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) env->prog->aux->num_exentries++; } continue; + case PTR_TO_ARENA: + if (BPF_MODE(insn->code) == BPF_MEMSX) { + verbose(env, "sign extending loads from arena are not supported yet\n"); + return -EOPNOTSUPP; + } + insn->code = BPF_CLASS(insn->code) | BPF_PROBE_MEM32 | BPF_SIZE(insn->code); + env->prog->aux->num_exentries++; + continue; default: continue; } @@ -19041,13 +19123,19 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->nr_linfo = prog->aux->nr_linfo; func[i]->aux->jited_linfo = prog->aux->jited_linfo; func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx; + func[i]->aux->arena = prog->aux->arena; num_exentries = 0; insn = func[i]->insnsi; for (j = 0; j < func[i]->len; j++, insn++) { if (BPF_CLASS(insn->code) == BPF_LDX && (BPF_MODE(insn->code) == BPF_PROBE_MEM || + BPF_MODE(insn->code) == BPF_PROBE_MEM32 || BPF_MODE(insn->code) == BPF_PROBE_MEMSX)) num_exentries++; + if ((BPF_CLASS(insn->code) == BPF_STX || + BPF_CLASS(insn->code) == BPF_ST) && + BPF_MODE(insn->code) == BPF_PROBE_MEM32) + num_exentries++; } func[i]->aux->num_exentries = num_exentries; func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable; From patchwork Fri Feb 9 04:05:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05304C4828F for ; Fri, 9 Feb 2024 04:06:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8562A8D0007; Thu, 8 Feb 2024 23:06:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8043E8D0001; Thu, 8 Feb 2024 23:06:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67FB18D0007; Thu, 8 Feb 2024 23:06:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 503958D0001 for ; Thu, 8 Feb 2024 23:06:58 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 28743120E3A for ; Fri, 9 Feb 2024 04:06:58 +0000 (UTC) X-FDA: 81770929716.28.0494956 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf02.hostedemail.com (Postfix) with ESMTP id 4848A80006 for ; Fri, 9 Feb 2024 04:06:56 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Nn31RBqR; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XZ9Snvkhjzxk5f60xE38IZQNIMkdSN5DBO4L+T7puE0=; b=YDrgiw0LemH/mlFCYgliIkF1CCMy6wqERADqFXASC5O1aM+V+AI40AnTPfrLu+KMNw/PC2 tFIwtudFTpbPJ7DLZiwZ0kqYO5E5AW2GthuVfYOnUUw5jcgagj55ZePbzf+DKFeiiIjcDV 3W4QUW3sfDYTNReZ1YM63PfUCaB86DU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451616; a=rsa-sha256; cv=none; b=vnCQ12yPtKadajmaCg0WyJ2hEuyHP+xynVHEOacg5lb1+QYjiTvi3Mhi573ROkZEI0sjkU u0B0/YEr/IFI2/KifOfZJrFp3Rfi56O9XUfOoGlggx0m1neKr76y1jfKdiO2zHjxeRIatF vIJl7paZmVB9Q86jcqkO4q9746jueRo= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Nn31RBqR; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1d7354ba334so4476065ad.1 for ; Thu, 08 Feb 2024 20:06:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451615; x=1708056415; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XZ9Snvkhjzxk5f60xE38IZQNIMkdSN5DBO4L+T7puE0=; b=Nn31RBqR873Jx8ukrPZVsfVGtDXH+cR44s3OxG1TvMHZ2Nnmc5Ah7IztAGLBgqlr3O 24x+Yg+aCou2BnREvpjaPxInSq4j7menFyEVrnT3kza/AVOaFNHdHAbzXfb7RRgmSxjJ iIe7RSL/A0S2flwtZ1wf5EAottEE2QGvVwMxDK71FfTwb++sEmL/V6qL82c1cKNaxCEg hEqOSvkPfr9XDnRm+iMMWo1gTNXeqYwHuW9h0JIQcZqe6Q/ptiuTLUdEczyVScp2L87o DhA8Af232J7bfYEB32M0m+RGKKXSdueQnXJLab+zl2892TBNs9jqbthpIRLFW6H39U4j PZ7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451615; x=1708056415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XZ9Snvkhjzxk5f60xE38IZQNIMkdSN5DBO4L+T7puE0=; b=pklb3X0zr1pe4VPiT1DiD/ku6MKk7YFzQFH2HrWlB7vYVh4BMWJ0QH4v+0//4RyHJr deL9BBRlL64O0fau0GwLxJFsVp446cN8pn2Zzps3tLlr8TVULNvbE5JHkJkc3liZQqAJ cOk+2miNSrv4YTTeC4H4v77QlV7nJmK2q8T+A+gEt8YQbYXqV6U5rPnUFYcSkH1jLPOC f74i14XsmdXiWhWoHzBXv2GjQFXyGHneqABU0JfNNoBI2xnjZLRQjHaw+FN8vvaNXyzQ YPcoC09zMFhNk0YgIMFYNQQMgKxzFcrp97l/Fi13RVnsnpYnia+5INYOpn2/sIV0Nt2o ul6g== X-Gm-Message-State: AOJu0YzbU0jVSc97Ghf8Bxokoc7OI9cdB3OPgqTx580UukZ8jJnBh1I5 bD4KSNgnyEtETH8Cw8RuuwDJRE/ZplN5wdETWDGUx7m1J6vLCTxF X-Google-Smtp-Source: AGHT+IGdKvGloD2x7T0TvljdUW1/xtPfwtVY4eu3Wz+SojJrERvkkNs762n7o/a8ViAyTbYVnq16Ig== X-Received: by 2002:a17:902:c403:b0:1d9:bbc2:87e7 with SMTP id k3-20020a170902c40300b001d9bbc287e7mr500885plk.36.1707451615176; Thu, 08 Feb 2024 20:06:55 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCU/x2cbpnz0BUBrwD3oynKGkbQWd49A9Q51EP6K/5fCnnF23ZUpv0qRDmHTlrR6yA0AtXPMi3FuK5K+KFjNWQZB5mJntsOxS8Pm1u0s8ziWoeQw2r37+Me/HIxl3XgxR4kF3y3DhvCArqcZd+TXsi4bpWVUbTSne3QN9pquK2NsflaDJazTaGUXoTDw5FqEjjU2zuYKegJHRBDw6WC7STY0O4c1TLAlrG4mfSz/8Qw8kYBzblb0HSXdBLKCCxn0P8d6aqm9/j/gIVQKJRkMiVlEC3V3y4n8PzaD9mBz4h5KNSVPY3TwerwIV4dK9wOn2J489jvaUMgPG6ksWBRa0Vjh8eFWxGtM4aKjqcZVj6h9pFRxYkRbAg== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id kw13-20020a170902f90d00b001d752c4f180sm560989plb.94.2024.02.08.20.06.53 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:54 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 10/20] bpf: Recognize btf_decl_tag("arg:arena") as PTR_TO_ARENA. Date: Thu, 8 Feb 2024 20:05:58 -0800 Message-Id: <20240209040608.98927-11-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4848A80006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 47uw9ueacot8xuuku5qfrc4rgfien1br X-HE-Tag: 1707451616-253340 X-HE-Meta: U2FsdGVkX19qlBM2T1OQIZgZM1xc/kpESDIPpDaQvakOm0DzE8tMRYlcoBvFPQ+kBOxDLffWTu8iAmeDiwEFwSR/OTJZcpgAX4DutMXzKgdzacQnQWS1o32cPUzo/+Z8UghRN8oK+t/bwLDb6SvmQo5EoMNjPuGbl8Vara5awm9+Y+oTiwf+rwxmQxrIMywi9bRwqNEXxEcsgRhmrfhaawnXAuxKaWlCyId7UE9N2Hc5jEpmhl0bqVhq+G/zXixQkcv/8b3K5OBtUFkmAcHIBHfTMVllhBgQB/6X+PNRPTatI5aoxxfojUyhjoQTppIH66oiYWlUNW/8rUpnSpUMEQo7HJpP2Mpt/ajE4UwHDNNM3iiKKOEdfO7+nu3WHcPKOXa/ztLjAihY576xpKXgDCmPlONCzhiS3pn1F0WXWqLjeeDVvN8Mq2wdR30bqMuArx59O6fX6zDZm2ULa53BF/1aRleD2pATuOTtxl0Izeazk/C0rR7hxQHE0qCiDAgEY59zVjoaYa026KZdh4GUWqDmqP6H2uttweDwgvL9R+kY0+oasF2SXN7cxAww7Ha1S3fBokQ98w0VKnWUVoFiyMedhRPbBIxaH7CgVPM+nd9dRoQ+r8UDnUERmNbwZP+zDJPV8jQFY1UFitwQj59QyE2TIK1u2qwy18L2SKzm0+bCE1nQsri2ySaB7IIay433QnzzRNwi1utC0hdkIIBc2xp4cvIR+gjgD9M1bRYY+FKxZBoxFIgb/PpX4C1hIxMaanJcWVrM82TYdWR208vPR7y2g1TuqDeW7TvbHvDsjHMwJ5xSLCCnQJPBsfbdvZ4T2GKZyB+Kfon3Uy9U3tZFGaiTY2vnzv9ZtpE8DVBg5nQOrh/25giepKgSqJpE7Od/dDmK4QXFM4FgcXpqVEgWyBhOhcUguujj86xpUFucJ14zad+20LzR/SZ+bOkic8X/KwUUTDfflm9afI2jYuT MS54eq// ReQey9ZIcKMY1J4Uixcdr6i/a0vWMD3SnMEGWFO/Uu09yPXNlKLlRdAuYy1Gnf9M285Vu32nP1gdynP2B6MWoovGg2JCS6zSg1+he+NqZaPDCQbduUNXAaKgIUyZPb4NpRHLopQUE7xeqGRf4ram5WjANW9ceqhvtPM+osLRKdIyoAPbbJf22xXaH/ytdLbpEcHxFtXmAeDfpOnnnujf3KODEUtVdlCQJFlMeIyDrSbhSDhaIF2eSMLdPtl9/JzoQy9NK/ZsHAC55ecmgJuwolvy+J+g2IW1VmY01l647FjWqGndd4uFCcqPlepVwEQhCUqNWoLPhAheFpD3xicu8cQ9Yp0KrjD54vII24q32nHJrNHY/tubuwPCreFaremEah1oxAfR6Z1ZqS18IjpdaXGz8TC9LkJbJ17iUzVNkxbg9SbFQ5jFERAtF/ieRapWZpFUxDdC9uZ0ef2fIKRZIGwktuOZ/o1tYoE7SURPnM5M5fKX0DsUs29fBVWhlDdyvFElwt/nrBVytdU04O5wCjgzOIuWdePjpm5qj0Gb/gm5oPpm4OxeWcQ+KOComE3se4FKp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov In global bpf functions recognize btf_decl_tag("arg:arena") as PTR_TO_ARENA. Note, when the verifier sees: __weak void foo(struct bar *p) it recognizes 'p' as PTR_TO_MEM and 'struct bar' has to be a struct with scalars. Hence the only way to use arena pointers in global functions is to tag them with "arg:arena". Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 1 + kernel/bpf/btf.c | 19 +++++++++++++++---- kernel/bpf/verifier.c | 15 +++++++++++++++ 3 files changed, 31 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 70d5351427e6..46a92e41b9d5 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -718,6 +718,7 @@ enum bpf_arg_type { * on eBPF program stack */ ARG_PTR_TO_MEM, /* pointer to valid memory (stack, packet, map value) */ + ARG_PTR_TO_ARENA, ARG_CONST_SIZE, /* number of bytes accessed from memory */ ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */ diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 8e06d29961f1..857059c8d56c 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7053,10 +7053,11 @@ static int btf_get_ptr_to_btf_id(struct bpf_verifier_log *log, int arg_idx, } enum btf_arg_tag { - ARG_TAG_CTX = 0x1, - ARG_TAG_NONNULL = 0x2, - ARG_TAG_TRUSTED = 0x4, - ARG_TAG_NULLABLE = 0x8, + ARG_TAG_CTX = BIT_ULL(0), + ARG_TAG_NONNULL = BIT_ULL(1), + ARG_TAG_TRUSTED = BIT_ULL(2), + ARG_TAG_NULLABLE = BIT_ULL(3), + ARG_TAG_ARENA = BIT_ULL(4), }; /* Process BTF of a function to produce high-level expectation of function @@ -7168,6 +7169,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) tags |= ARG_TAG_NONNULL; } else if (strcmp(tag, "nullable") == 0) { tags |= ARG_TAG_NULLABLE; + } else if (strcmp(tag, "arena") == 0) { + tags |= ARG_TAG_ARENA; } else { bpf_log(log, "arg#%d has unsupported set of tags\n", i); return -EOPNOTSUPP; @@ -7222,6 +7225,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) sub->args[i].btf_id = kern_type_id; continue; } + if (tags & ARG_TAG_ARENA) { + if (tags & ~ARG_TAG_ARENA) { + bpf_log(log, "arg#%d arena cannot be combined with any other tags\n", i); + return -EINVAL; + } + sub->args[i].arg_type = ARG_PTR_TO_ARENA; + continue; + } if (is_global) { /* generic user data pointer */ u32 mem_size; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5eeb9bf7e324..fa49602194d5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -9348,6 +9348,18 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, bpf_log(log, "arg#%d is expected to be non-NULL\n", i); return -EINVAL; } + } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { + /* + * Can pass any value and the kernel won't crash, but + * only PTR_TO_ARENA or SCALAR make sense. Everything + * else is a bug in the bpf program. Point it out to + * the user at the verification time instead of + * run-time debug nightmare. + */ + if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) { + bpf_log(log, "R%d is not a pointer to arena or scalar.\n", regno); + return -EINVAL; + } } else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) { ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0); if (ret) @@ -20329,6 +20341,9 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) reg->btf = bpf_get_btf_vmlinux(); /* can't fail at this point */ reg->btf_id = arg->btf_id; reg->id = ++env->id_gen; + } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { + /* caller can pass either PTR_TO_ARENA or SCALAR */ + mark_reg_unknown(env, regs, i); } else { WARN_ONCE(1, "BUG: unhandled arg#%d type %d\n", i - BPF_REG_1, arg->arg_type); From patchwork Fri Feb 9 04:05:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B341C4828F for ; Fri, 9 Feb 2024 04:07:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4C9D8D0008; Thu, 8 Feb 2024 23:07:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BFC508D0001; Thu, 8 Feb 2024 23:07:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A75F48D0008; Thu, 8 Feb 2024 23:07:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 94ABB8D0001 for ; Thu, 8 Feb 2024 23:07:02 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6590CC01E7 for ; Fri, 9 Feb 2024 04:07:02 +0000 (UTC) X-FDA: 81770929884.01.35B7724 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf03.hostedemail.com (Postfix) with ESMTP id 8480420010 for ; Fri, 9 Feb 2024 04:07:00 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DMNadKqm; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451620; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kyKA4wCImIZZ5xJE04KmfmUkjYhSBJW7+tRHKBcOaBI=; b=O21oo1KqPxremF0Eryh5q3R948tWBcjbvIFhHXAOTk8cXcAYmaUZEoNpSXu8ETDsarxuII 5fKNuU9h6fNzZSG9Yp1IqDwHswwmCPL5dv0Eep9cgt47YdV0HDK7eHoZv+1su8NPRC60mf O2IkhJcOBEA7OKG/8xKvctcd/qTKI9w= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DMNadKqm; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451620; a=rsa-sha256; cv=none; b=thRvYxgm/u1ne0DNFT528s3CR2PqAS/NNi0I8uBCS0V1q7AaxHBMWuf2LPUiA8pRXKzugH dap9d1yHP5mMGyp9NHGQmqyIRMqwkwkSM8NobSyNDAP8i/6f5vMcOhkMnnhStH+mQmdamR 6H6zrrCx1oytG4Vr2yvYF2udLUJ/k4Q= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e055c89dd9so1346712b3a.1 for ; Thu, 08 Feb 2024 20:07:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451619; x=1708056419; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kyKA4wCImIZZ5xJE04KmfmUkjYhSBJW7+tRHKBcOaBI=; b=DMNadKqmRZGbM14bT2gJ+lvbG3GSKGBDxmnnxYkLZNvsSq3bawscGNqLUumWX1T8FJ /NZG0o1xHdEYgWPAEdTFU0UClmeVD0apJP0VSu+O5jcCgyxAYRuU0xpFnKW1pIzOPKTF GvPROmnuvxMwYqkiBiYZk9VOH4uBONEbrSpEoHwQBuV3c/JS7B2tnXmdqo83NS4IkDHi hcFRe+zIvN+d3+iR7BTLCqKAxc3F2jhl3cRdBrirOCsSXtvKBO2JTYDO5+BkvYvWLZOb K4u+wRGN8s5Kg3Sj14aMI1wRB1I0YekXlilEMcc0hsASfjnz/MRAricb5XXdePo1fO3n frtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451619; x=1708056419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kyKA4wCImIZZ5xJE04KmfmUkjYhSBJW7+tRHKBcOaBI=; b=t38jG1ExAjRqWM7y9wPVBNtYKFvBPkMaztfufFBivf/AKkvERJHSgSQTSWLGoNmdcT 3sHrI2y3G0JhBtoN41avoCkkoQuAjmObd8OrGOwtRRSnDjRijggAWmw8WRBsUzZANmju m4dZJWwBHEAxZcu6PmyG7V+1QjgKqs33sECodrzl/oBgGdQCwd5kcmutkJRt7XmzBxh9 /9HMeUBNNGMTfQl90s9IT+dgShLj8zYf2CHXOjlp9Y6TGvm78xW5TFDTFTFwowcC3nBC dpvJiKEB/2VZn2EqrgNkJJlQS7JuzLL44OvXg6i7E6A9R0cJcUaLm2v9JI18DBrN5i9L EfQw== X-Forwarded-Encrypted: i=1; AJvYcCWoi5TFn6RBR4T/W06mMEtjy7dBP/CtVvbCxrsiMMcM0tf4HxGhMnOU8BN7vQjDt9aZBleQdUaAGrrgeYa35nphRdI= X-Gm-Message-State: AOJu0Yx3H8WUOlMUHAMV+uQsqVdTI6qi/+ozOVTruVuUQGQMt2rGd8AE MXmQcQZVfngkRZdt5Wyh4nn1s8IN2H4hvv/U6M3cflwUOaYbdA7V X-Google-Smtp-Source: AGHT+IEcq3u8IxQEchLTqYOdj8JCUTyhf++9XKhZV1VlMQZ/9+Av7nCVAWvySSMoNnMsdtUnppsIjA== X-Received: by 2002:a05:6a20:8c01:b0:19e:b477:33a4 with SMTP id j1-20020a056a208c0100b0019eb47733a4mr520269pzh.27.1707451619456; Thu, 08 Feb 2024 20:06:59 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWQEeNvrqoRwizmtNe7laAnu501WQNUUrnNc5bjAfQWP+sx+w5e2Fh+IdN7e3m/thOyA1PXmPbUrKFF2iv85lnNbw22sei0R1dMfM2b8sjF/scMfelPDzWIRLb3Rbmeu0nynfi0QRD9uaXnafRn3hsJTcHF8/wZcK3Qf+kv3Ni8kHO8NrzDQeHD4uLmFd/P0i5q1NYeju6EtjYm1j4OJY8+q7gyU4OR7agn4uOilcwA8qebcGY49lcc99jxuByats2X9fUO/pyRnbvi0DCQwCENKvViTtlTpGJaicvrQlE97J3YVk3FS629SYdh7Ivcfb+pS7C/9gwvPVJnwZqDz03apakQb62uN7S8jWHLS7hJgoSz3YlDeQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id f19-20020a056a00229300b006dbda7bcf3csm589192pfe.83.2024.02.08.20.06.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:06:59 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 11/20] libbpf: Add __arg_arena to bpf_helpers.h Date: Thu, 8 Feb 2024 20:05:59 -0800 Message-Id: <20240209040608.98927-12-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: sbbbjgetk659ywwfta3m6xyt16jzqcqb X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8480420010 X-HE-Tag: 1707451620-340762 X-HE-Meta: U2FsdGVkX192Ve/fVPuXIBiKqTUC6+O/BLJvcpKikIXgcYqW47PZ0FU4NcVk33W/pJBDMGxNY39S55b7LWCo+qGA6Dc6TaUR6tJzupsOY9WtY+pwj/ebQPJmxQJhcXCQ03NxWOZncd+H61Vt+J9vrlqTBpya36mKnzEsgdJDCPR0RvR2IBgzpxxnU+Bcauqv+2QuACll1kvis9J/B5FcEXMTgqIdqq3wpG6at15/mlIS6vwJA1raK9cKlQtjbWyOeGPOGgmI7KDMQBy8IVslhGVnkLfWTXrVm4ngm8KllvW0s8Asb495X5b4OOU7GM2/73Nkxo+tAEHPBOhjcPW6zQcZIEIsu7356Rorg7Jkot9cyQKL1cJh4KrJ5qocfCMFk0s6sQlzxcbxD5pZUu35ly98kVBLE+woT5fgQmdTio4IDs0gB5kUB3IQbctG9yRVNCA4Byz6QyDZhr7EbuiPcxI/LuNtNp13rjRvBpxMjuc4p7v8l0Rqn+XFCiNCiOzbkHsRVOeseJAx4NSUQLN1V5/lzazx3DKunRjm0s2i+gb6pIptJcOfv/V2nKOM8HXvELVVZycqa23dntztSKhtZjZ7P7flY/M4pAN7+0EbktoQ4ErsCjz8yz9QQEFyUh7S76dNIJRBb4rSDGUVHmQEojeSqsidYMR8zgaSoPzk9xLCUMsJFL00PCdFgqqyc3B1QRkOhPFwtK0jnkQ2bFEZMjcyn2br4fJmzRd/YWRbTeXErKWnE7byrlctglKUl5oXfE1zt70EOpj3Cpe3/lYH053FrJ3h87R9OmavQScA972KulPv73JOb2iIAtI7asrD6x3ULhciAR+LLA17glGi5x6r1tIbtdE0wO8Y67dn7fPV9WULMV+XfzKv6YcNR9kHTwokLrEHx1BXFAByccZgLCX1kPzHWUzEBHgx2SyJFQ+iYj6e0JJq/2i00PysIz4SedvTqlUMc7xJpPdIPVA 1l6OhubW BDRi0suJdWuR0yRokKa55EQsSq8JlgG6KQutT8yYLjcwTuQwolLXcXFJbXOGBd8JWrkEslyQa99+BcJgSZIvbwsnIH2wecmHPBRw+7LpcoXQEfGQ42tFCcFYFqwbCtcdzqRE78BghwOr8HG3spFyBkI0RMK2t7qaUqj/kNHa6JhRINRWFxUldwezg3paWmVCB8AVSx/ySVnNoo4hm7OvWqid7/NUGmnlRPReQ3yLZLlR44LUrP5OT6ji2uWjT5kJF8J+cQYXOVGL0VVOxO8hzsBY7ECPSOVBQtx3RnwQeAIHiBmMF+8UbmHIRt2g0U+Or7r/Kt3uGX7FyoiQ2UQkj8UQacPxdK4RWbArhUc4rfzrAQcpigDBSnkNxecQiICaAVXHdSOO0YOqf+ZFoJalHx6D2MExhHcYP7+fT6IeruUlrQALus9wCf9ezBT0J59RakUpQJOBBSVTvLpNfwdt5digX9FLosHx/+Pf3D1Z9pM2VivViFOQ4z6ZgqfOLxtr7DBd1fdb63EglqYxkgWdDMNkYBZt4i1y/TLNmQyVRutEAHDAgrbRhk/Jghrowv6KB30Yr5yoY7+jERue3sv4LIxZD5epewroT8fJX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000200, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Add __arg_arena to bpf_helpers.h Signed-off-by: Alexei Starovoitov Acked-by: Andrii Nakryiko --- tools/lib/bpf/bpf_helpers.h | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h index 79eaa581be98..9c777c21da28 100644 --- a/tools/lib/bpf/bpf_helpers.h +++ b/tools/lib/bpf/bpf_helpers.h @@ -192,6 +192,7 @@ enum libbpf_tristate { #define __arg_nonnull __attribute((btf_decl_tag("arg:nonnull"))) #define __arg_nullable __attribute((btf_decl_tag("arg:nullable"))) #define __arg_trusted __attribute((btf_decl_tag("arg:trusted"))) +#define __arg_arena __attribute((btf_decl_tag("arg:arena"))) #ifndef ___bpf_concat #define ___bpf_concat(a, b) a ## b From patchwork Fri Feb 9 04:06:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74FDDC48297 for ; Fri, 9 Feb 2024 04:07:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0167A8D0009; Thu, 8 Feb 2024 23:07:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE2C78D0001; Thu, 8 Feb 2024 23:07:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5C188D0009; Thu, 8 Feb 2024 23:07:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BF8598D0001 for ; Thu, 8 Feb 2024 23:07:06 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 944A8A066B for ; Fri, 9 Feb 2024 04:07:06 +0000 (UTC) X-FDA: 81770930052.06.FC4F8B2 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf14.hostedemail.com (Postfix) with ESMTP id BAE94100002 for ; Fri, 9 Feb 2024 04:07:04 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MSRpVZ0v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nd9ldHvznK5bBX0WOOiLjO64Z4v1t4JC5wlS+JmAlpg=; b=1cmGGv/hOwyIWYzTHl0W3hxCdMSu2LwLR6HuVPeHFvgXUBSxGyyAMNCScSJ2bhL67IQPIp MGRFqO2QfTiE4RXvhCtpYOT+hoY8HVRevmPNH7ddK5fOe/57xwl/qmrlBUSifZy3RGvJk5 TXZUJCy8Zv4gbuky6MMxSM0Ay/0/ejc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MSRpVZ0v; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451624; a=rsa-sha256; cv=none; b=xFLj6HKIiqRbCpG3FIduwhNvT63WwNV6hO8P2s76a5ErLY3WS/5UqvHEmfFWNAOAZt0Nks a4l666t8otlehFudKbAYfOKLJgFS3XPU8MM5Qg9I26+Aq98sbOhKvwB3SeSeWXYu1pX3DQ I9mKmSrhrRoW0LfUpkS8FdYClECon/s= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1d958e0d73dso4425525ad.1 for ; Thu, 08 Feb 2024 20:07:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451623; x=1708056423; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nd9ldHvznK5bBX0WOOiLjO64Z4v1t4JC5wlS+JmAlpg=; b=MSRpVZ0vknJGFoHewi2d3ikruX5c6ugOu+diDGVPdr5VHKfjKxl1HMYzDr2v/qKhAD KnLo+Nmq80LeMCKAetnFqMvB3RRAsCfA8vC0O89bisvWjByzvj7waVgJhibUOT7BDg3h HJS+4PSdRqLPMYsp2r8up89Kwm1M3zYuLE8GFkQy0mpG+lxSZKdO9byqX0NzHsKgZH/t 07P15b0hMvFR/1wdJ2WpokJQTzQV1+WFTewZXCHgrEpzP80O9PbYyRkcp9Uknq9XtCzr 9r1veqXt9WvyGcfrMpPdw+tEbhzH/XzZ0kgr+Uzasylv27qHxfPPJVDkP+zekVpWgV+Y JbvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451623; x=1708056423; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nd9ldHvznK5bBX0WOOiLjO64Z4v1t4JC5wlS+JmAlpg=; b=Tp3guQ0zx4SrWvl80r1fh9TB1GuvDu3k/We4/m+OnTLoPdbgf966UFDK8BqzBQrRBs kISSVkVlow6XkFOnycq1Rf79al/NOMLPJ+qq8H/TjqEyTxyjJ2Xp4Rf8oUFR5R2QfEta 2vdSi0TqB9nkkZgaRzBMrVdk62aAJR7w0TGJ1LSvcmjbMxrgJ5JExWbTw2plkVAdGMvr CufhQNKAYBboqjCl6yB243WpXpJu54ki/hskktm2gx22RTTuEXWVkKJtzPzjV+EAcbXD T0GDKrXVrdxr0hrapG0KbhEmRvr/Z8SD7uyYVofTcZd2fNnT1hg1r38pdgkdum9cKOgd 2Adg== X-Forwarded-Encrypted: i=1; AJvYcCU6DW/3oZdK52e/bMRy0Rg1AIkAhq5EfAyRFx9uLmXguk4eni02TeewnV0zeuYJ86Wvef2cEFBBlltMB7HCXP+hxvc= X-Gm-Message-State: AOJu0YyVWdojZDuB2ffiZ1Pg/HQ//QaywzOLo8rvI/sEZbfj/wL2VcnU roSoNodfyeiNS2Rn65GWoxuN+CMo2iwU5BGGdVQajX2hM35yFmyU X-Google-Smtp-Source: AGHT+IEoPNaoXNST3bY7EQ3Xy37Qb7fTssbofLabaCcz0TDvqlgviwuOkE+S47i6w80xgnTbu99/eA== X-Received: by 2002:a17:902:ce91:b0:1d9:34ff:f807 with SMTP id f17-20020a170902ce9100b001d934fff807mr727831plg.31.1707451623582; Thu, 08 Feb 2024 20:07:03 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCU4y6CYVGbOKIVod4G5Bg5GTdHTlGslPYpsljRN9o3Z+8SFl6eHgSF+Plus5zKv7ifiilN8+3Rjaqfp4tPFWlgiGm9U4WYTgptRd2cVgf0dvvrSyQcj/sI1ZPcywe5wsAK74e+moJ3nEcZD20bSvOl4UhYy9gcv8vpm3VgPJnpPhUu4irxBkQwSGw2QH6I3K6McMgvIFFbwqhzJxep1QZGhuyXkrRiSjJTp3dKWrYgJ878B2ObgfAyj29Ns+1kI/6tpzGl9I94oP1IN+y3mDfcH7Qk1OeqG6dNbq3FmYOFCEvdYqPdiXRYRMtkXLZqiyMVp1rjFBtFgEcyRZH6cMEDc1G3cXwnNMCz4r4xrV67tuUAMd7uW7g== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id g12-20020a170902f74c00b001d9ba3b2b33sm541375plw.163.2024.02.08.20.07.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:03 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 12/20] libbpf: Add support for bpf_arena. Date: Thu, 8 Feb 2024 20:06:00 -0800 Message-Id: <20240209040608.98927-13-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BAE94100002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: t8cjftzksyrmimqaqyzgeth9ojg9y9x6 X-HE-Tag: 1707451624-992582 X-HE-Meta: U2FsdGVkX18PhrS0nVVhiHOU4KPI/6bagD6yx0yipAL2zMUMDxmSM8hGWeiTQ5+MLARbr8xZURP9MNnd5eoYSLoOIUg2EK11GiHH+co8hNqbv6Pnu93jMC7sWJQdrPh7zmiM/ByYEaICJASu4ljcBaae5NT4A1pZulpwr59y7WiPshXwbUK7tavJpJCGxBe5T2fFYME5kHZgm7sJ+faSa2Tgjrzi0FxeeTafcKvSvV5bDMLBsb3TcjLzh8EopWf4p/uWoeT54l2w/Hd3Z1hwzzlSZOiHNs/286W/SRoMOBHKi2wrlBG/+1Kq5LgE0m+hYNzzfK32aSNo6KD09SqdIfrmak9+JieL4deMCHc5NXzXBCzM2wYwGJCDW/VezskJP6ltVtHsVlPj8Wo4fU/nr+zr0/Qj0HUurBzkoDflkHaJK1vrOTOjCtmMHbeKy7CIMVJHapFwBSq7bgi2s1FazqZLRQEuCjvbi9M46VVpuNi+AcqgiYrnweLn5awoN/q7auaa90onFdz8bZ9YHfkrozFUAnc1ha+kX+F5LLOaEuVFlMLbKRV96Oe6vU1mtjZ1yLL9T2zXgVMsW9AZ4rJS5i8e4H/LcEqUuK55UV+sVE5cC3fiwbH7fpQsLYqlkSd++jQ6lOvgd5nsL4P8p2FCDxn9Qu7wVtuvGGpF3Ybsri1UG60hAjF/ZnunibHo66bN7pQPMdnpicfCkG4ktdFSK3vAdxYeY20lNG2ug6H26vvqc2JOlABRcg9AGf2woGaJ4HMtiFR0Mi7cClHzFarmab+A+ahmtnNH2zmUBYYuh6ZoBAUiVmy6At+ciasIKaBurYdekdZzQDxBaRcyy7pZWx7RFGg/VePfvlXQhtC+Kl+zY2gv2zYjzeBmm0dURwt8LXSopAsbd06GuCC8FBKY75IXPw8DyxF76z64lLV5zM0w1jtGk1LnF/Zn1zYPfykZEsE6peOZpANtcRYF4JJ ZF/VkLX7 s2O7UbbKGcZMql0bOIXhXhyhxfYZIzFxVqZzJUtPWdV6x+P4D+hkHfu1YbJ8LQZdV6lyTa48F8S5YG5kj0hKWKjA+lCDjuIZplAj3nJoG9XIKEBpZah/hP/kJPlr9PVwxlVipBcRe1VimuCErstN4AMllWw+D2epZSymagYCo2e+dT8FxJf4TmY3nPxbVwhXc6AmCaYEvfGoXvZZXYjWpNPPSNxYsgBumA4DXDIyXK2uq/RutELwDc2wazgJmA0nKL1A7B/6k9U8DiyuBZNgj2l/DKr9bu74p8s5qlRcyHDzKJFNefQNMmnW5g4DVrUsbYzT8CDaUY2su6ieLigL4Xq1sWXrBTMPlc4q4bkpY8vdVAQ8uSfkpjK9Z8J8N3+QxSQI2tNp58qe2Rdzso2kSzDFbRFz98aVyl7de908didVCPkY2nCs6X+8sJJC8H73SBXScxyOr3pvpdfojFGaR7qUW1xvUkECVcLp2L5kYJzFq9qiLjJKVDstaoh5xBdvYh4CM91sOafHFbBTSFs6MmI2U0N+Tu3Zep1FDVkCCIyBddtSBkxexRTb1HSH+OVUljgQTTlaTJtKjAESkJpl2I0cWP6ijdQr3RNKn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov mmap() bpf_arena right after creation, since the kernel needs to remember the address returned from mmap. This is user_vm_start. LLVM will generate bpf_arena_cast_user() instructions where necessary and JIT will add upper 32-bit of user_vm_start to such pointers. Fix up bpf_map_mmap_sz() to compute mmap size as map->value_size * map->max_entries for arrays and PAGE_SIZE * map->max_entries for arena. Don't set BTF at arena creation time, since it doesn't support it. Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/libbpf.c | 43 ++++++++++++++++++++++++++++++----- tools/lib/bpf/libbpf_probes.c | 7 ++++++ 2 files changed, 44 insertions(+), 6 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 01f407591a92..4880d623098d 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -185,6 +185,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter", [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", + [BPF_MAP_TYPE_ARENA] = "arena", }; static const char * const prog_type_name[] = { @@ -1577,7 +1578,7 @@ static struct bpf_map *bpf_object__add_map(struct bpf_object *obj) return map; } -static size_t bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) +static size_t __bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) { const long page_sz = sysconf(_SC_PAGE_SIZE); size_t map_sz; @@ -1587,6 +1588,20 @@ static size_t bpf_map_mmap_sz(unsigned int value_sz, unsigned int max_entries) return map_sz; } +static size_t bpf_map_mmap_sz(const struct bpf_map *map) +{ + const long page_sz = sysconf(_SC_PAGE_SIZE); + + switch (map->def.type) { + case BPF_MAP_TYPE_ARRAY: + return __bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + case BPF_MAP_TYPE_ARENA: + return page_sz * map->def.max_entries; + default: + return 0; /* not supported */ + } +} + static int bpf_map_mmap_resize(struct bpf_map *map, size_t old_sz, size_t new_sz) { void *mmaped; @@ -1740,7 +1755,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", map->name, map->sec_idx, map->sec_offset, def->map_flags); - mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + mmap_sz = bpf_map_mmap_sz(map); map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); if (map->mmaped == MAP_FAILED) { @@ -4852,6 +4867,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b case BPF_MAP_TYPE_SOCKHASH: case BPF_MAP_TYPE_QUEUE: case BPF_MAP_TYPE_STACK: + case BPF_MAP_TYPE_ARENA: create_attr.btf_fd = 0; create_attr.btf_key_type_id = 0; create_attr.btf_value_type_id = 0; @@ -4908,6 +4924,21 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b if (map->fd == map_fd) return 0; + if (def->type == BPF_MAP_TYPE_ARENA) { + map->mmaped = mmap((void *)map->map_extra, bpf_map_mmap_sz(map), + PROT_READ | PROT_WRITE, + map->map_extra ? MAP_SHARED | MAP_FIXED : MAP_SHARED, + map_fd, 0); + if (map->mmaped == MAP_FAILED) { + err = -errno; + map->mmaped = NULL; + close(map_fd); + pr_warn("map '%s': failed to mmap bpf_arena: %d\n", + bpf_map__name(map), err); + return err; + } + } + /* Keep placeholder FD value but now point it to the BPF map object. * This way everything that relied on this map's FD (e.g., relocated * ldimm64 instructions) will stay valid and won't need adjustments. @@ -8582,7 +8613,7 @@ static void bpf_map__destroy(struct bpf_map *map) if (map->mmaped) { size_t mmap_sz; - mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + mmap_sz = bpf_map_mmap_sz(map); munmap(map->mmaped, mmap_sz); map->mmaped = NULL; } @@ -9830,8 +9861,8 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size) int err; size_t mmap_old_sz, mmap_new_sz; - mmap_old_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); - mmap_new_sz = bpf_map_mmap_sz(size, map->def.max_entries); + mmap_old_sz = bpf_map_mmap_sz(map); + mmap_new_sz = __bpf_map_mmap_sz(size, map->def.max_entries); err = bpf_map_mmap_resize(map, mmap_old_sz, mmap_new_sz); if (err) { pr_warn("map '%s': failed to resize memory-mapped region: %d\n", @@ -13356,7 +13387,7 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s) for (i = 0; i < s->map_cnt; i++) { struct bpf_map *map = *s->maps[i].map; - size_t mmap_sz = bpf_map_mmap_sz(map->def.value_size, map->def.max_entries); + size_t mmap_sz = bpf_map_mmap_sz(map); int prot, map_fd = map->fd; void **mmaped = s->maps[i].mmaped; diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index ee9b1dbea9eb..302188122439 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -338,6 +338,13 @@ static int probe_map_create(enum bpf_map_type map_type) key_size = 0; max_entries = 1; break; + case BPF_MAP_TYPE_ARENA: + key_size = 0; + value_size = 0; + max_entries = 1; /* one page */ + opts.map_extra = 0; /* can mmap() at any address */ + opts.map_flags = BPF_F_MMAPABLE; + break; case BPF_MAP_TYPE_HASH: case BPF_MAP_TYPE_ARRAY: case BPF_MAP_TYPE_PROG_ARRAY: From patchwork Fri Feb 9 04:06:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550854 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50349C4828F for ; Fri, 9 Feb 2024 04:07:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5DEE8D000A; Thu, 8 Feb 2024 23:07:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC0608D0001; Thu, 8 Feb 2024 23:07:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B393C8D000A; Thu, 8 Feb 2024 23:07:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9FF598D0001 for ; Thu, 8 Feb 2024 23:07:10 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7A09940010 for ; Fri, 9 Feb 2024 04:07:10 +0000 (UTC) X-FDA: 81770930220.16.07A9433 Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) by imf10.hostedemail.com (Postfix) with ESMTP id A1A4FC001D for ; Fri, 9 Feb 2024 04:07:08 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RAiKDjXP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.42 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F0RFXFHjgEipVAjjo7rYUK8PW9TKsuaRbrDEibnvW4k=; b=JuftfkVmtC8rObwjv2DrJ0ie/AZdXJQCYnbUo2teKKfdF6vjUcYnU/vJPxyKBPCT4wGYhZ 1htK67gea0qIsk8tguqpzAfsVyivM15rBJ9rWgJnfBzFRMdX2QoiBqCNY1WOKpbl77jQp1 gPmTZ0VtseIzsyFdhn8pYksD8FA5YCI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RAiKDjXP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.42 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451628; a=rsa-sha256; cv=none; b=czt1YiNvgzsuCfEEaZ1JUE2+bJd1N2RttA2UKFHnY30RgOXYqomjEg0J6OUE/rB2M5cEft EjtUapVyjJDiPJQOAWnTaG9A9iy2Fm59hrvm+cgRIcRv87OjNnRKQ7b5izUqWerPLzGV0v JVZkQAkgQAQulxZaLKkmzZmWyJXcTtQ= Received: by mail-oo1-f42.google.com with SMTP id 006d021491bc7-598699c0f1eso317140eaf.2 for ; Thu, 08 Feb 2024 20:07:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451628; x=1708056428; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=F0RFXFHjgEipVAjjo7rYUK8PW9TKsuaRbrDEibnvW4k=; b=RAiKDjXPHP36aCP4iXv2MQBTQb8zymn3F4bisRDwQBRSEkHmMxMotzPXleeVLRpLyX a1iB6dLjbubwnP4AjnJBAczK/fEbGnm7dhzm6flezYhzxr41LBnpTZnEMYNs2dneeVkz xxpe9wm5XIMxo8RRErAwYYzvNmQjC4UPwUtWlJx4qjc0nwGmqrilXH+JCkEvJe+R6B/C uqtd0sIn9aCeofFV6oEaaRz9hHepW1BFckvB+6NfaAcJia8uVjp/xTmIKCGKxcKPHmpW hrWYkDvOvYzH4cmFFOHqzudpC7GhXVuW08Eja+3KwJ5wafB0S16xL3627KsIlc7bzYrn 6u0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451628; x=1708056428; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F0RFXFHjgEipVAjjo7rYUK8PW9TKsuaRbrDEibnvW4k=; b=K/gcptUGZrICNr9m3+fdV9ip1eVxtbWWZrK8c+8Q4hQimZx4wQUAapX05tGx6Dw+Q3 rjmv9fwbEEuO2e64Vy34hg/j/cCmLYuqZP+MVje5qnvEHKIsJg+2INH/3GhxBP/g8HFx RxG19SB2MZguKBtdq3wxYBOCaE8Z18vASZw6P9Nm7w8OstW05q8Ytayfn9HJhAiA/Qpx taT3KNv8b7GOvXlTxdLyqY7rf6OoKHrOwJPaTbV5G6IayrelIKYCkt7ggwsHaXxw0r1a ICIu3UUsj0vzpqOeCFHTYbZQoFQQZgVczw+wSbnZTS90fXPLTA76Ep5Ruxgw8cNbK8M/ k3ZQ== X-Gm-Message-State: AOJu0YxMQYQUs9WzF7YzdJpLo5wM4TBTeylS6xssg6ckvJrnWNKQ+VUL 5bwimuZ0JLkrUbeExGHQsUR95pzsXqtPZ6CTGOsphyE/dR+fumRS X-Google-Smtp-Source: AGHT+IH3ELoemyowyx9wDHV3307l0ycjd0Y77bmRJBAVEdimNMeIuNSIlIwPFXHELajuyUjjlQuC3w== X-Received: by 2002:a05:6358:5923:b0:178:fcd3:c316 with SMTP id g35-20020a056358592300b00178fcd3c316mr369964rwf.19.1707451627656; Thu, 08 Feb 2024 20:07:07 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWoqc11kvj74nhgP0ItrhTalUSnkiKOdLRkeNXEUDkXxEiRPRf5d4bYuosF+6cOhfxzAMXVsl+t9+QOFHt5NPTVLFeWfXCYrBD7zGBhDwyKkAg83xUqbq16d4W438iTv/u4iqXlrxEfu5sU2ANBjPMqsLutqvxFcyLN7iF8AmK5czq/wJC1HhSd5mJgCo/vuHecDmQai1BM7uVYVxdm2xvH48dSiZe88KkqjcoFFxkEbZdmbNwrubW/g03c508mJ+Ep58KUOUKGi+0TXrw5efj3tEt/fIfNkCt6kmNdiEVmolW+mb3CmnCkJvU+Vf3d6lOpC4xDzA8R75HxOgkWVg0SLXn4HSflWlWByPP9oqC5Q5ltk7QVRA== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id r15-20020aa7988f000000b006e02da3a158sm610623pfl.17.2024.02.08.20.07.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:07 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 13/20] libbpf: Allow specifying 64-bit integers in map BTF. Date: Thu, 8 Feb 2024 20:06:01 -0800 Message-Id: <20240209040608.98927-14-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: r61zb7wozgx1mc7mowqmu7eqknu6gcqe X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A1A4FC001D X-HE-Tag: 1707451628-382653 X-HE-Meta: U2FsdGVkX1/5jKlKqJXP4zIsw5R/ueWtNDUELnA5i3mHhq+Ti3unK0tlwFbyeml+l70nJLvfqR1K+LmOy4OZCpL85Y7HyJ60dUCodtXMfQoOEdJhegw3SEJorR4wvYZP+sJhjxcrdcNO+QaauLFDZmPjSoDdkjG7LwHTF53qC8yDMue9EXUekB5QdVUqkoWHB9afr24AQhxLnPk8vMouJ8wH9g68Wq1bTs05CMQ5imRlAWLbT1ztneCTt1so4FwjBeXeUK1kyBdD1kRZWnMW7xVSE58jJXd7NmA7rcHPz2ck8p6Q6cHrFci34E2a5t5PdoD2sJfG3yyxlNA9piVrEHa+qQHJ8XTj+HaP+U2vtZKDOJisGqK3qaux+3s7MARJBYY4ApsurJbN1GuARMqJkxMaXL1PBX1aggyLZzimzznok+Aaweixy9F+Cl+BmMXDuWMXpmmh2JHfY7UFVnOfaTURGgzcIaAPXO39AUocMOTRSXeY6xizT2AKHqxCqObGiioBsQCVFZGJkPOYSHNpHzaDxljBy7bdGG/WyCd48cWS9uzSonRTFFomGc8qnnrLJHmr3R6TaLlYDlYxhW2KUo+U2sI/yhHTJAXsTOfQGVY/Z4fGl79Hl3c+9gEMUNCScitUF2Jjw+QNBaVbghgTyFt0lmOMGa2NqwD47JbLQOWl4EtFCz5F0MKIXhtl6PtsrzpNQDIZJY0di2OoHZBR0mAKdzb6IcEoDtsQZDciXJQm0jxNr8aImqizIZAh85MNsrZdoauP+FTmKA6CAG1x2szkPGDwUwEavtbaUTXJy3HCjKz3DRF4pI+CBi7p/SYfQS9HglWLM8Ip2wowAq9xIGTUGIAV39Jwcq8UdGrSXj523IY2uMxkcVIkpLcX36pPn8k5CsPr1dPljM/YigDt5UQTP7zUXyMtsLrx734/jBXQdAV4EnMXCL4Fh8ye5QytZYoHt9vaehBXzAp+l3h pbkc5C7X bHRQbxSToKPmO7ORnBDtUFF+6t8fkzplOIYYm1hmyTHr4yigGkEOYdtaQAHXRy9x3xGFxyCKZ+kJFxpcAycegGbyQB2Tb1EdU1UjNJtOoqFwADgVrJjWxFM7KjIB1X5MstVLrcUApiYWMpupeP/daoYQLG1F+GPoC6I2NuHcr4TrjaAMHkPwWnLA7pMuvRS5d1IUnF34xbFQZL/zmKYwawI8/lz0rGLOuAV5j2XPOQODnvlmSNyj2IOlWhPZ5e3V8ewEk1J/UMdY278xhMfKn+hbdMqMF59Gdl4tWgDMZy2vrDZaVA/qdn2lRGlFBhLO3j6fNAovoYvBWxThDFtfPabrvc6rHu8nvaUXsO2BckKyAtZQA3PXlVZSSsFh1Gnr0Slziz4ha7xJfjDC74aOW/lo8Fr3INarCYcG4sxIFf+U9eDSF9neSUVuw+3y6mevpCKAmRi4awfAqBTZ6o0VKfpjUHZ0hQqNgTCBsWkCyKyOYJIVYuJ7IHxo3Dgma828WVlnGB9ENFZWEHcud+mspRbAwYJQgm6gIaoa3yMq7UG9R8eyHRDz4Uj/L9w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov __uint() macro that is used to specify map attributes like: __uint(type, BPF_MAP_TYPE_ARRAY); __uint(map_flags, BPF_F_MMAPABLE); is limited to 32-bit, since BTF_KIND_ARRAY has u32 "number of elements" field. Introduce __ulong() macro that allows specifying values bigger than 32-bit. In map definition "map_extra" is the only u64 field. Signed-off-by: Alexei Starovoitov Acked-by: Eduard Zingerman --- tools/lib/bpf/bpf_helpers.h | 5 +++++ tools/lib/bpf/libbpf.c | 44 ++++++++++++++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 3 deletions(-) diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h index 9c777c21da28..0aeac8ea7af2 100644 --- a/tools/lib/bpf/bpf_helpers.h +++ b/tools/lib/bpf/bpf_helpers.h @@ -13,6 +13,11 @@ #define __uint(name, val) int (*name)[val] #define __type(name, val) typeof(val) *name #define __array(name, val) typeof(val) *name[] +#ifndef __PASTE +#define ___PASTE(a,b) a##b +#define __PASTE(a,b) ___PASTE(a,b) +#endif +#define __ulong(name, val) enum { __PASTE(__unique_value, __COUNTER__) = val } name /* * Helper macro to place programs, maps, license in diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 4880d623098d..f8158e250327 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -2243,6 +2243,39 @@ static bool get_map_field_int(const char *map_name, const struct btf *btf, return true; } +static bool get_map_field_long(const char *map_name, const struct btf *btf, + const struct btf_member *m, __u64 *res) +{ + const struct btf_type *t = skip_mods_and_typedefs(btf, m->type, NULL); + const char *name = btf__name_by_offset(btf, m->name_off); + + if (btf_is_ptr(t)) + return false; + + if (!btf_is_enum(t) && !btf_is_enum64(t)) { + pr_warn("map '%s': attr '%s': expected enum or enum64, got %s.\n", + map_name, name, btf_kind_str(t)); + return false; + } + + if (btf_vlen(t) != 1) { + pr_warn("map '%s': attr '%s': invalid __ulong\n", + map_name, name); + return false; + } + + if (btf_is_enum(t)) { + const struct btf_enum *e = btf_enum(t); + + *res = e->val; + } else { + const struct btf_enum64 *e = btf_enum64(t); + + *res = btf_enum64_value(e); + } + return true; +} + static int pathname_concat(char *buf, size_t buf_sz, const char *path, const char *name) { int len; @@ -2476,10 +2509,15 @@ int parse_btf_map_def(const char *map_name, struct btf *btf, map_def->pinning = val; map_def->parts |= MAP_DEF_PINNING; } else if (strcmp(name, "map_extra") == 0) { - __u32 map_extra; + __u64 map_extra; - if (!get_map_field_int(map_name, btf, m, &map_extra)) - return -EINVAL; + if (!get_map_field_long(map_name, btf, m, &map_extra)) { + __u32 map_extra_u32; + + if (!get_map_field_int(map_name, btf, m, &map_extra_u32)) + return -EINVAL; + map_extra = map_extra_u32; + } map_def->map_extra = map_extra; map_def->parts |= MAP_DEF_MAP_EXTRA; } else { From patchwork Fri Feb 9 04:06:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2ABDC48297 for ; Fri, 9 Feb 2024 04:07:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EB4A8D000B; Thu, 8 Feb 2024 23:07:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 371328D0001; Thu, 8 Feb 2024 23:07:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EB288D000B; Thu, 8 Feb 2024 23:07:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0A8B58D0001 for ; Thu, 8 Feb 2024 23:07:15 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D9D7D40351 for ; Fri, 9 Feb 2024 04:07:14 +0000 (UTC) X-FDA: 81770930388.18.C664690 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf30.hostedemail.com (Postfix) with ESMTP id ED28580009 for ; Fri, 9 Feb 2024 04:07:12 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LOiWMzI+; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451633; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kLcdghajJR/QGdT0y2Nv6AwpiR5Q3wh3RT5GphLP8Vw=; b=hOpVQsBLoMWapmtrCLH9YpLvNSzIv0kJl7h4lrihplsg0zan8qxxlxvjpML8JzbGkeMi2L n/OnmOZ44b9n1zj3AgsJttXF6kjfZ2icr16oUFKMj7uSEg0QXSpd34e2Q0VZKq2vttCDdJ 5g24uDH63HsDSigkmvemeK+SMVXvFxg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451633; a=rsa-sha256; cv=none; b=cYSYRTVWZ3oDfqr5QXgMlndN6UeVq3ap2ITvd08XUCHiU1YxTowzeKMvHho+/OcB48JPei NpZZ1jttI9qEUEHs0DP+kocUrQxmsbRwUy763L27VZXGLYC8JYpQ0xm21L4ljO4Roodb/0 DvBK5DdZgwFtehiLdj61jkMMnjF8rjs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LOiWMzI+; spf=pass (imf30.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e02597a0afso364973b3a.1 for ; Thu, 08 Feb 2024 20:07:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451632; x=1708056432; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kLcdghajJR/QGdT0y2Nv6AwpiR5Q3wh3RT5GphLP8Vw=; b=LOiWMzI+DozuJbxDtnp6HAy2IPL4xBaR7+4EiwwXisP8ZBYrWAzm8OrAnO6fgcsW8R F3ajr2d9pKAuRrllx4ayYf9w8B32CbwsQO7wPJZYt5hKjHTFA1wm13WVKgVupdaKDP4y zOq/ivzgrwXYrNl0flgCGwFZOK8Mn0yrCOyHINlaCOUjK6EhclEGq7iQ5eFcV/19DIFY jbCgI1+CObd2BFT/ZhaeDwBzmdN96sR46PxFVI+IlhJThO5L/uTYVv+MaOKqHj5fndvM bawizbRsrrTUYST90A+g36PYuoar5dO+NZ9ux4tqCSBV17SmyVSzGGpdR+dXzUcSWn0K oKBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451632; x=1708056432; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kLcdghajJR/QGdT0y2Nv6AwpiR5Q3wh3RT5GphLP8Vw=; b=azFrbo61lceVRI94lag8P99Qp1Y3jVRWh+9XR02BRfyeSmVY1V3YP0hNV+bxyTb6Qw RpT7I0Hmh0aR+c5mOCcsYHoyTkjHea+NPgY2rpSqXTy9/akr8f/ZNrjyNxrYVTSHjO11 9iw4sLYmTofz1R4s5zMRI7oC6fbCvKySdrztKu00wrPdD/NTwxs6pXhQ1rYIZhFfieMg y/K8oM22+lNVet2dvzoM64MMK6ABuy8o5+8e4v1etIQbFUxoT7v4/Ykb+B/4jRM6GA9n id8KTAIk7MZTFuWcBN3BUePubbGTB4PChIEqqElLvXhUV1IkoSQ4TZTWyDoxYwf9Wftb Zhtg== X-Forwarded-Encrypted: i=1; AJvYcCXkhAwV2MgU7/VWU6VTlj39PlXiqb52H0RzuhFyIF6OWLKLr80gklr/CqGRhI7TLr3TtDC/OnZz1I3s9E9L4iDwgeU= X-Gm-Message-State: AOJu0Ywy251URa8Je4h77iNCK746ArR5F50FHAHVlTg8+etXc8epGI4F 9EYK2/j9HfLdD88CkNdu8H0Givo3jdycp0/kxVZzdocCF3kg13o9 X-Google-Smtp-Source: AGHT+IFyT2HEc4ssBsDGMa4TA3yrF8aBqF74Rmstz83H0u4hHry+ndHS7WzzE7gccPrmnj/Cj736wQ== X-Received: by 2002:a05:6a20:4c96:b0:19a:4418:1e86 with SMTP id fq22-20020a056a204c9600b0019a44181e86mr530941pzb.58.1707451631872; Thu, 08 Feb 2024 20:07:11 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCUMVJtQVx+RHJh+uTVXziJuPTI/HF31SiR4SNYP6BHTsrkAvpTU1sc62DC21f13kRH9Dye4YDzYfba3LqMoAKSi+zMvfbZxGIUdYE4eQTBaHACABp5HjD9YYkJ7KoGVldB8/5CPr+2TEHW8cuKCioB9EIZV+kQ5CCdT4rZWQSNjINRK7wk8vRN2AufUOUbG4S0F8ZI5Fh4BtKBORfHn9I/5Du1L3rrfFgBPPTnFwtR37nrQi7pimC+mbyRHJbo3bH7RKBioHKKpzRZiTlPSUxx5A2rr+cKgpBI7AI6TMyhVRxJ0Nu5ITYRkBCWEAd6OiF/vq72Jxx7rz7KGQuubEf/0nS2k6bgxYL+WyM67sSGSZp7txMlxWQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id r1-20020a17090a438100b0029464b5fcdbsm715206pjg.42.2024.02.08.20.07.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:11 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 14/20] libbpf: Recognize __arena global varaibles. Date: Thu, 8 Feb 2024 20:06:02 -0800 Message-Id: <20240209040608.98927-15-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: ED28580009 X-Rspam-User: X-Stat-Signature: un3cz1zw1tkyiqkg7uz7rtja6gne3rnd X-Rspamd-Server: rspam03 X-HE-Tag: 1707451632-745006 X-HE-Meta: U2FsdGVkX18hRpagpYeXc+Q1ZOX+ZbXSHDJmRiB/FLj04e32SnvvPw3ichJu5Ldq9kk0rocZV78bhaDDNfs5lDqtn4oJwPrG6rhGC0ODWcgQ3yH1G+ZKqMKSehs4qewJIxEYEnUrLoQ2tJJNt5MPRDPN4ISuzK+bBYdp/mH5O/rGZFu9zuF5nHXI1U083EYHxNZxTLKLaROkv6bW4CCWfytKVZBDFdZ4UdgN31DoS6RjzPDr193J8D6jYa4EDjJ/c8AlBc2sMg4kmBKvYbi77dowVtHK0zvDd377cbEDd5UIIZLDXDsFAJFb9UkrfKbRnRIh7eszk4MCg3AEzhZWmXS0oSawwBc+Z3daDchqYZ7Ib+orbmvdkaM26LGA1zQgEi1sKFODytzNhltrBDb0Pam97FWe8VaRksh5cCoDj1H+sYzlKRaa5oOr5YN2swawSFtYeZsNoS0+yreXxvfD46j0GV19jggNhCgrKh2J5nSjhmG8+MDmHsYLMKSoDfuFVaTlOCFGiLmvrMgEwWlhM5kcLE17YLcpZy+3LlJPps9JzVzDi/XZLlcyikxXUbAdx+MpIZoca9XFdiTPQ+WdIcyBvR1B9CbUsIUL/RptVaLQxgM/RVklAbYgQicd+DQEeCVnIMJ2yM4hlrG6T92zi4MJ24z1fgyIjtrPR0Uh27VHgV5QdAjx7iAMUBQe/FzWuniflEIF6/gt7v0SAoMTJWHUIk8qHMd7gdMD3c37bJTCru1wFNp1SWLsod8GxOqmegpM3Z+PPa6KVSVUnF+Ub2L3MhCIk5ZrFoXCp810Cqzq+fPicmxjB89O5NLO/4TFki62bssyoIOBgZKXyoXSn2JSBpimx+HmUTOxXR5Cdac/i47pCxWpNQZnPnmbVgnYHm4iA0K897E+lOLGxrr3aLVEzxilg+yzyUOG/J0psHD5X7/dTOJjfu8+9gyAheKtfAh5jT/oFCVbf4Xtu61 MO/270x1 /YH7GJS3jqgu91dyCig55OIvyPomPPpPhgIl6UyHHeIu8JZFp6ebyb7jq+EDB5c5VPISd8MhhRmQBlUwyh80c584bNWkENzLYnazvOi+lKapMTbd8hvLaiulNlPdKscalVye0dMYFmlotgeXM/y1UtxNW7pzkNZg8KrPv12WkdYEyG6chhumJPvr1KQ3drsgNDG3YszTM8BaCR1s+lMu6dPdCqnaLl0b58esqeLotKd4cXWbSMlZrzJrYUpqpj+oyCg8+Uy7hWFz7d60WnkJm9b8tEI99dQKembehb0eHdvrdaoXYO3oa68n45W8CD2MxecaWohztnXeIxolwZRMstwHhudYiqYJX7Xy8ABlQ8xOIkF7rBeA+AxYh3A2sG7dVC2wKMcEjHcKz/wAgySw86J+WaxIpNYgSqIS4C7XUSCqFA5l6YZ8hg5DMDgx0Lte5kXSW2mvdEvix73QFkMCCi8PaeG4+FCNBwmfNwST7SdLqJfC+cwAmPoAWuEb7JgLlQk99aJiFP77lAA+d/RBVJVZVwoS4fIFNnEHsQ0ieVwQJMkaCqcE+wSyXOa1Yr3s1gdx+NSUPzjPYGOy91DwLMvSRbD6BsFoT3sTF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov LLVM automatically places __arena variables into ".arena.1" ELF section. When libbpf sees such section it creates internal 'struct bpf_map' LIBBPF_MAP_ARENA that is connected to actual BPF_MAP_TYPE_ARENA 'struct bpf_map'. They share the same kernel's side bpf map and single map_fd. Both are emitted into skeleton. Real arena with the name given by bpf program in SEC(".maps") and another with "__arena_internal" name. All global variables from ".arena.1" section are accessible from user space via skel->arena->name_of_var. For bss/data/rodata the skeleton/libbpf perform the following sequence: 1. addr = mmap(MAP_ANONYMOUS) 2. user space optionally modifies global vars 3. map_fd = bpf_create_map() 4. bpf_update_map_elem(map_fd, addr) // to store values into the kernel 5. mmap(addr, MAP_FIXED, map_fd) after step 5 user spaces see the values it wrote at step 2 at the same addresses arena doesn't support update_map_elem. Hence skeleton/libbpf do: 1. addr = mmap(MAP_ANONYMOUS) 2. user space optionally modifies global vars 3. map_fd = bpf_create_map(MAP_TYPE_ARENA) 4. real_addr = mmap(map->map_extra, MAP_SHARED | MAP_FIXED, map_fd) 5. memcpy(real_addr, addr) // this will fault-in and allocate pages 6. munmap(addr) At the end look and feel of global data vs __arena global data is the same from bpf prog pov. Another complication is: struct { __uint(type, BPF_MAP_TYPE_ARENA); } arena SEC(".maps"); int __arena foo; int bar; ptr1 = &foo; // relocation against ".arena.1" section ptr2 = &arena; // relocation against ".maps" section ptr3 = &bar; // relocation against ".bss" section Fo the kernel ptr1 and ptr2 has point to the same arena's map_fd while ptr3 points to a different global array's map_fd. For the verifier: ptr1->type == unknown_scalar ptr2->type == const_ptr_to_map ptr3->type == ptr_to_map_value after the verifier and for JIT all 3 ptr-s are normal ld_imm64 insns. Signed-off-by: Alexei Starovoitov --- tools/bpf/bpftool/gen.c | 13 ++++- tools/lib/bpf/libbpf.c | 102 +++++++++++++++++++++++++++++++++++----- 2 files changed, 101 insertions(+), 14 deletions(-) diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c index a9334c57e859..74fabbdbad2b 100644 --- a/tools/bpf/bpftool/gen.c +++ b/tools/bpf/bpftool/gen.c @@ -82,7 +82,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz) const char *name = bpf_map__name(map); int i, n; - if (!bpf_map__is_internal(map)) { + if (!bpf_map__is_internal(map) || bpf_map__type(map) == BPF_MAP_TYPE_ARENA) { snprintf(buf, buf_sz, "%s", name); return true; } @@ -106,6 +106,12 @@ static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz) static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" }; int i, n; + /* recognize hard coded LLVM section name */ + if (strcmp(sec_name, ".arena.1") == 0) { + /* this is the name to use in skeleton */ + strncpy(buf, "arena", buf_sz); + return true; + } for (i = 0, n = ARRAY_SIZE(pfxs); i < n; i++) { const char *pfx = pfxs[i]; @@ -239,6 +245,11 @@ static bool is_internal_mmapable_map(const struct bpf_map *map, char *buf, size_ if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE)) return false; + if (bpf_map__type(map) == BPF_MAP_TYPE_ARENA) { + strncpy(buf, "arena", sz); + return true; + } + if (!get_map_ident(map, buf, sz)) return false; diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index f8158e250327..d5364280a06c 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -498,6 +498,7 @@ struct bpf_struct_ops { #define KSYMS_SEC ".ksyms" #define STRUCT_OPS_SEC ".struct_ops" #define STRUCT_OPS_LINK_SEC ".struct_ops.link" +#define ARENA_SEC ".arena.1" enum libbpf_map_type { LIBBPF_MAP_UNSPEC, @@ -505,6 +506,7 @@ enum libbpf_map_type { LIBBPF_MAP_BSS, LIBBPF_MAP_RODATA, LIBBPF_MAP_KCONFIG, + LIBBPF_MAP_ARENA, }; struct bpf_map_def { @@ -547,6 +549,7 @@ struct bpf_map { bool reused; bool autocreate; __u64 map_extra; + struct bpf_map *arena; }; enum extern_type { @@ -613,6 +616,7 @@ enum sec_type { SEC_BSS, SEC_DATA, SEC_RODATA, + SEC_ARENA, }; struct elf_sec_desc { @@ -1718,10 +1722,34 @@ static int bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, const char *real_name, int sec_idx, void *data, size_t data_sz) { + const long page_sz = sysconf(_SC_PAGE_SIZE); + struct bpf_map *map, *arena = NULL; struct bpf_map_def *def; - struct bpf_map *map; size_t mmap_sz; - int err; + int err, i; + + if (type == LIBBPF_MAP_ARENA) { + for (i = 0; i < obj->nr_maps; i++) { + map = &obj->maps[i]; + if (map->def.type != BPF_MAP_TYPE_ARENA) + continue; + arena = map; + real_name = "__arena_internal"; + mmap_sz = bpf_map_mmap_sz(map); + if (roundup(data_sz, page_sz) > mmap_sz) { + pr_warn("Declared arena map size %zd is too small to hold" + "global __arena variables of size %zd\n", + mmap_sz, data_sz); + return -E2BIG; + } + break; + } + if (!arena) { + pr_warn("To use global __arena variables the arena map should" + "be declared explicitly in SEC(\".maps\")\n"); + return -ENOENT; + } + } map = bpf_object__add_map(obj); if (IS_ERR(map)) @@ -1732,6 +1760,7 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, map->sec_offset = 0; map->real_name = strdup(real_name); map->name = internal_map_name(obj, real_name); + map->arena = arena; if (!map->real_name || !map->name) { zfree(&map->real_name); zfree(&map->name); @@ -1739,18 +1768,32 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, } def = &map->def; - def->type = BPF_MAP_TYPE_ARRAY; - def->key_size = sizeof(int); - def->value_size = data_sz; - def->max_entries = 1; - def->map_flags = type == LIBBPF_MAP_RODATA || type == LIBBPF_MAP_KCONFIG - ? BPF_F_RDONLY_PROG : 0; + if (type == LIBBPF_MAP_ARENA) { + /* bpf_object will contain two arena maps: + * LIBBPF_MAP_ARENA & BPF_MAP_TYPE_ARENA + * and + * LIBBPF_MAP_UNSPEC & BPF_MAP_TYPE_ARENA. + * The former map->arena will point to latter. + */ + def->type = BPF_MAP_TYPE_ARENA; + def->key_size = 0; + def->value_size = 0; + def->max_entries = roundup(data_sz, page_sz) / page_sz; + def->map_flags = BPF_F_MMAPABLE; + } else { + def->type = BPF_MAP_TYPE_ARRAY; + def->key_size = sizeof(int); + def->value_size = data_sz; + def->max_entries = 1; + def->map_flags = type == LIBBPF_MAP_RODATA || type == LIBBPF_MAP_KCONFIG + ? BPF_F_RDONLY_PROG : 0; - /* failures are fine because of maps like .rodata.str1.1 */ - (void) map_fill_btf_type_info(obj, map); + /* failures are fine because of maps like .rodata.str1.1 */ + (void) map_fill_btf_type_info(obj, map); - if (map_is_mmapable(obj, map)) - def->map_flags |= BPF_F_MMAPABLE; + if (map_is_mmapable(obj, map)) + def->map_flags |= BPF_F_MMAPABLE; + } pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", map->name, map->sec_idx, map->sec_offset, def->map_flags); @@ -1814,6 +1857,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj) NULL, sec_desc->data->d_size); break; + case SEC_ARENA: + sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx)); + err = bpf_object__init_internal_map(obj, LIBBPF_MAP_ARENA, + sec_name, sec_idx, + sec_desc->data->d_buf, + sec_desc->data->d_size); + break; default: /* skip */ break; @@ -3646,6 +3696,10 @@ static int bpf_object__elf_collect(struct bpf_object *obj) } else if (strcmp(name, STRUCT_OPS_LINK_SEC) == 0) { obj->efile.st_ops_link_data = data; obj->efile.st_ops_link_shndx = idx; + } else if (strcmp(name, ARENA_SEC) == 0) { + sec_desc->sec_type = SEC_ARENA; + sec_desc->shdr = sh; + sec_desc->data = data; } else { pr_info("elf: skipping unrecognized data section(%d) %s\n", idx, name); @@ -4148,6 +4202,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj, case SEC_BSS: case SEC_DATA: case SEC_RODATA: + case SEC_ARENA: return true; default: return false; @@ -4173,6 +4228,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx) return LIBBPF_MAP_DATA; case SEC_RODATA: return LIBBPF_MAP_RODATA; + case SEC_ARENA: + return LIBBPF_MAP_ARENA; default: return LIBBPF_MAP_UNSPEC; } @@ -4326,7 +4383,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog, reloc_desc->type = RELO_DATA; reloc_desc->insn_idx = insn_idx; - reloc_desc->map_idx = map_idx; + reloc_desc->map_idx = map->arena ? map->arena - obj->maps : map_idx; reloc_desc->sym_off = sym->st_value; return 0; } @@ -4813,6 +4870,9 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) bpf_gen__map_freeze(obj->gen_loader, map - obj->maps); return 0; } + if (map_type == LIBBPF_MAP_ARENA) + return 0; + err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0); if (err) { err = -errno; @@ -5119,6 +5179,15 @@ bpf_object__create_maps(struct bpf_object *obj) if (bpf_map__is_internal(map) && !kernel_supports(obj, FEAT_GLOBAL_DATA)) map->autocreate = false; + if (map->libbpf_type == LIBBPF_MAP_ARENA) { + size_t len = bpf_map_mmap_sz(map); + + memcpy(map->arena->mmaped, map->mmaped, len); + map->autocreate = false; + munmap(map->mmaped, len); + map->mmaped = NULL; + } + if (!map->autocreate) { pr_debug("map '%s': skipped auto-creating...\n", map->name); continue; @@ -9735,6 +9804,8 @@ static bool map_uses_real_name(const struct bpf_map *map) return true; if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0) return true; + if (map->libbpf_type == LIBBPF_MAP_ARENA) + return true; return false; } @@ -13437,6 +13508,11 @@ int bpf_object__load_skeleton(struct bpf_object_skeleton *s) continue; } + if (map->arena) { + *mmaped = map->arena->mmaped; + continue; + } + if (map->def.map_flags & BPF_F_RDONLY_PROG) prot = PROT_READ; else From patchwork Fri Feb 9 04:06:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2F87C4828F for ; Fri, 9 Feb 2024 04:07:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39C528D000C; Thu, 8 Feb 2024 23:07:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 327578D0001; Thu, 8 Feb 2024 23:07:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19CB98D000C; Thu, 8 Feb 2024 23:07:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 04BF98D0001 for ; Thu, 8 Feb 2024 23:07:19 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DAA2E803B9 for ; Fri, 9 Feb 2024 04:07:18 +0000 (UTC) X-FDA: 81770930556.01.6C3637A Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf25.hostedemail.com (Postfix) with ESMTP id 0B6F2A0013 for ; Fri, 9 Feb 2024 04:07:16 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=amwYJBQe; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451637; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mCIDfn6Q/Sj02IoZMnsUeC+cwQq+R0K4Vy6avN3EcgA=; b=nXNm7UHki+33ORcCQjny1cWYHWE/CuH6RLGVE8UaXzfocQyJ3IpVL3TMPyS1J0IvxHWOO0 3lqRcg7JkurPjFakZBgZtBbIqnTSwA2Zq9LETI2m3dzGKNWBjYiKa5zvdI96u62ovD74SK rmzYyCDBJX34dPz3o6YUyOSjwInbvzs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=amwYJBQe; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451637; a=rsa-sha256; cv=none; b=OatCgVEo/xRsaIt01UEukqw14jM9tpd/nw9cu1Ocwo3hhPzS89pXHNu7xrzdP/ovekUHCa HJ1PmvbLabCgrrPK6f9ObmOSdpRczeE4WZhYGbv1LqthZIcCF5FRtugNJRCTymH3zVJSAp AYmnzWMCG2wkg2yOgJtyY6Ns5vx40DQ= Received: by mail-pg1-f175.google.com with SMTP id 41be03b00d2f7-5d4d15ec7c5so369916a12.1 for ; Thu, 08 Feb 2024 20:07:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451636; x=1708056436; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mCIDfn6Q/Sj02IoZMnsUeC+cwQq+R0K4Vy6avN3EcgA=; b=amwYJBQeVYT9dKsPjxn5xV3us4ScqqVadDYqlbmJBirlfJuwOTiejKypE8p7hb5UzN Ispn+NyG3Owj/hu6azyxULxaQtOaTjFuHnebj1lqcSepgAbjBcKGArE/iigeIB2c3Nuo +zqCkr8Ulvq9KFGcnXg5qriBW8yReFjcxkbK4VOA+vYjA/8ems6P44Vjb/KFkSew7Zjd ptpeYyx6Vvxi4p/9ZWUBQnPbZT+v+RTRT9IhsIhaR76ezGPa0DzmYBTTzSr44PYbZgBM sIDhAz895aS5ZmqhrwyF9r0LnGHqlM8fHOwtXc+BpTPu196cNo7sEjs01YbJ0nV1BOYQ lmvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451636; x=1708056436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mCIDfn6Q/Sj02IoZMnsUeC+cwQq+R0K4Vy6avN3EcgA=; b=wtt6c1k6QgZsxnixPt5ntIr0AxEdSG8Pkj7DDH9Nt+AVGRdFx+SXWHsDTaO37kbqFr uzdsQSSD+e7UFn/Coo4Z+UlI66eLRDE9AeyTi7aTbZs0PNOHqRzQp7fncCtbopHjbqOW l0V+oMl2GLVgP6HbB2dw3pHtWkOFia0ViwkJm/6/1uJhddpGYU2X4dLBuwcXGEZrPm44 0nfsJu4Ud8nu/rNBdJlOlBVgyO0Tl7h/ivncka10+WzqN04C0TwiYpvyh7Znd14KeeA5 pbTmb7Lp6fSGYb1K2eLh+NNtHhCri5sIPXKeJYnlnPkBephVqf2dkv889UmewGLkiX8B Wrhg== X-Forwarded-Encrypted: i=1; AJvYcCUW40XbxPxiCC4ZksYcGi/N/W53bfO7K48ub6b5eUJ9TWsW20Nmp+aVh31BoDbu6TNdxdZ9lo7LYyJaX3OgreL6LRA= X-Gm-Message-State: AOJu0YyGsn/ZzgeuiYTj7M5susRRB3/PlxU22xaE01RozDHV5ZoCGb9x 6kmYNd1+g1YR5itDm56zbmci3c6tJtbWdyr3YpYG5TmLSr7OEXS+ X-Google-Smtp-Source: AGHT+IEi6h+ceTfRMJ5dI0p3X7XJYh4ZzKQVqNqt1uF5ux0MvuhWLqAFaBeMLSjm/CCs2wr1Y3GeYQ== X-Received: by 2002:a05:6a21:6711:b0:19c:9c2e:7860 with SMTP id wh17-20020a056a21671100b0019c9c2e7860mr691997pzb.13.1707451635981; Thu, 08 Feb 2024 20:07:15 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXL44286flKbPuiC5jJpQL4r9tnFLyhpDpeoUK86uHyxm1KmiZjhC6iWst+l0Yj3ohxbkOTv9OMTtkRZouflvj/p0qsmCfCCNN+RU3re5UYu0eZbWx8Sm3Trzrg2OIR5wSWop8s/7FQFUH7FL46mVTYTOUwdUDu4w4XDY8IA5NbPjR3F59YeYpxDdlZ6qaYXgatcmUWxkbTYM+4Q6SW/7jYYzybmfjOPjllVmPAAYXvZ0YIqXV+dVZ+aEPOeAxlLZHGJST60otvvVynWglcMCjg34Qy05PuwmX05Am2ndC0Gyx9LCDF3GGY+BLLY5ra/6Typc56SRoEhQjNtaZfxlsYSSewLkgZjqAGJ0j4Ju5PQ7l/N2P/Sg== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id t8-20020a170902bc4800b001d9a40e204bsm551470plz.21.2024.02.08.20.07.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:15 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 15/20] bpf: Tell bpf programs kernel's PAGE_SIZE Date: Thu, 8 Feb 2024 20:06:03 -0800 Message-Id: <20240209040608.98927-16-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0B6F2A0013 X-Rspam-User: X-Stat-Signature: wfstsinmnafd91eem9a68g3x4njygqo8 X-Rspamd-Server: rspam01 X-HE-Tag: 1707451636-239406 X-HE-Meta: U2FsdGVkX1/tpbpljVzbMaKz91sxCnkaL/ttDX+m6us66zKtlnWsQkhs9W76QFK2YH2c8Dvy0Cp7uDWECfaBlx5d6FK/iUFqyYdkK9hDpYN9fgsjexVG3rO4QeDzbTSKkz4HfoBtL1NqVbToiUwMU4BKagDfBC+J/UUVy6K43H8J+ujqzU7rI2JAAwmuUzevO+oqsAWuxDos1aiZBpPd1Jv7PCN+U027nt4yfVTFv01cp8Y6W4NEhil1eQvOdQUoKTqzu7y+2hG7pL4M2Row2b+ujz13cDxEeaJCj23TEbk9vkNjVLiG1THTJuRgL7shGig6fu7KxnAVWScwHMaPaNcRpXTZcehraNsKV9vAdw3xs+w2nC6MmLGA2iDJ6vPkAbCwb6iB3RQSmGl8PRccZvEO2GsTTuBwB4ZpTWtRI6ulIHL5bwFv4vdbIikRr+Od83zWTHhxsEHmpjs54dhCpN3VyLTisJ/0uw8TYoiUoxVmF4m/4+vgEFCQTIP7RVrRHBSvEGvVxf7mlcNpE7YHS0HHsR7lctORvEGdD9J5JU/sgYx/sgOd3mkwU+4iIQq4Qnb5lpo6GArGS+z1JLsariFYI0qlk91g2PDMJOQDISFB/jUprMsrzOAFRsmI0iYf3UWy/bKMnnd090DO8i+VhxKJeWxki36J48ueASksLcNzDIO/vdqvAvXxhk9gS/8vkj6WMBUM9bBMDyj1l+8afekcGPmcM6NUa4nXRep6yB/63+iLCHmeasbQR3apcpmi7xxyUEVfwhdg5anevwNPt9ykBEBdtIA3tSohMb96vxqAGjQQ0T3/WMaHsZBrki8KrzGzzRWPNtNapbLSxJSpOCRlxZZ59yiZFXxMA28bBFBKZJ/f+r5FxxWwYub48zhZ9Ec4OlbvRkFSBb/7ZVC3VQH31ah1VNNS1jpPlmtaWj4CiSe2rzlFORuPCgrYFar1WnaLsUEdzjgWMC/WTE4 QikPkoMj fQ7XXlI/E4d/6OGkyaQKPBmN1CGbFXVQXjAAlFnR1a+AyvHhUH4digb9ecQL4h2+52yY5Id74W1fTsCcDGC8ceDzwOFCHd14BKXok4CEcTJqNuZgVexUUuZ5LMn7mvI1wKZ1UEVDldbWKxRm9dpwBUJ/zs5u8V7Z/wOyBZt5z8lv0atsjbP7pDtY54nLFw0YLxYPhfwx1G7tzyvG532RFJiumvWtUf/+bzlxAeQ/5sSJdaw/WIV2VyNDFxwfZ0bT6zVQX3PwM8iimjjFIDl+XvX0IkTpRezfpA83ga9eRpPrmCb5h1JJy/Mdc/uxQlRwr/ahaSAUhmafk4EhDD0KtK7wSZFWTwZUWlhrxEYvKzCs1X6fTV/pl4BCXH02wserSsudHt4LoiQrZ2pM+6zEkEc61WSexwV/cvRlwzSvTfHs2uQiEYBrVExLtP/MXmP5BXk6I4KFV+dcPA6Sue63fVhO1hjlS+ycFL9F95ovonhYcI5ksScfjy0+rXKH+bLliGljQpHMyXsqOubJ44UPCvK6LKOT5Xgsgjhrpfa9SfSiASq/RbS+oq3fIga0IFpg4N51SXaEf0Tn2PfP+o7bRQAuqtz40jx2gbrESwuf8QlvmJ8OnN5lK6HHscA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000690, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov vmlinux BTF includes all kernel enums. Add __PAGE_SIZE = PAGE_SIZE enum, so that bpf programs that include vmlinux.h can easily access it. Signed-off-by: Alexei Starovoitov --- kernel/bpf/core.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2829077f0461..3aa3f56a4310 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -88,13 +88,18 @@ void *bpf_internal_load_pointer_neg_helper(const struct sk_buff *skb, int k, uns return NULL; } +/* tell bpf programs that include vmlinux.h kernel's PAGE_SIZE */ +enum page_size_enum { + __PAGE_SIZE = PAGE_SIZE +}; + struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags) { gfp_t gfp_flags = bpf_memcg_flags(GFP_KERNEL | __GFP_ZERO | gfp_extra_flags); struct bpf_prog_aux *aux; struct bpf_prog *fp; - size = round_up(size, PAGE_SIZE); + size = round_up(size, __PAGE_SIZE); fp = __vmalloc(size, gfp_flags); if (fp == NULL) return NULL; From patchwork Fri Feb 9 04:06:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED7FBC4828F for ; Fri, 9 Feb 2024 04:07:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7931D8D000D; Thu, 8 Feb 2024 23:07:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 741A48D0001; Thu, 8 Feb 2024 23:07:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E2B68D000D; Thu, 8 Feb 2024 23:07:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4B9038D0001 for ; Thu, 8 Feb 2024 23:07:23 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3248C1603DC for ; Fri, 9 Feb 2024 04:07:23 +0000 (UTC) X-FDA: 81770930766.29.20E0799 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf07.hostedemail.com (Postfix) with ESMTP id 4E9F940009 for ; Fri, 9 Feb 2024 04:07:21 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bYhs3uvS; spf=pass (imf07.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451641; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MsJBrUUPtruHWtQsS6zgKVxqPHg8IzuE7qrpmwhQpng=; b=kfu4M+rQY4YDPeuu76esZ0TIxVCJzZWGNUp4khdft2esLwsTmhvMYDePqL4H3r5MWuObB8 aQweItRxkvjl89KDjDhb2blXLD238+M8+7e5jy/P7+ScUjNAg3GSgnN/DJAX+S8X5tpjo2 LBZT8KjW6aZvYMEgjW5Y2rP3WM6I4vI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451641; a=rsa-sha256; cv=none; b=i1+e78kJOX9mVEmSWN/fW2e6BXzH7tbcr/o2Zu6fHo75fdzqXB6rVaM+u0z90XzKmK6ZKA O4drR6h3ft0msYkOOb5mPJiJSpRzj8rE6oCfdeiAdfT0MYa+iGXkzr7trhOptMSqvdgwmQ g6IEL6hYncrl8NlRKcizYzvOM6/P6ks= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bYhs3uvS; spf=pass (imf07.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d93f2c3701so3167145ad.3 for ; Thu, 08 Feb 2024 20:07:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451640; x=1708056440; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MsJBrUUPtruHWtQsS6zgKVxqPHg8IzuE7qrpmwhQpng=; b=bYhs3uvSAexCEhBut9bw41bkJaR8YX+JCeKuNxqLbKSdj2GnKos1+tLROgPPuID4Hl +IcBbc9w9Z3wOq6aXUdTQkWsUK+1ZFUNhxFunuTlz3zIkZvrEk8hnpQsAX8ogWF7NhiS n3beBT981/8p4Q6jlD9c2S6I7cdmKuNf9/4h7INjupIVB37+OvZ24MKhDG6qIG+9UNiT tokLpnV96Kh9Gn8rdUj3HWduKMxwitSlIznRjWY9ZfTBfEIYT8BUFWcwQcn37jcqX1RS oOz3D5lLZuegTy2aufO4hjBBlH9IGZzCm9SG68nOBaTwjgxxpCEJhVNv0GehcENSZFsC ImNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451640; x=1708056440; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MsJBrUUPtruHWtQsS6zgKVxqPHg8IzuE7qrpmwhQpng=; b=TsgL8wC57W/Z8oBdBALkqps19JQS9kc3qmTyNR57/WazRhUFGNvXMnT+bv+uoiJcTZ t8OVMfysODqnvCvNJsemHvkzFS8v94jDsc8VA54De/oDtv0YZP4KWeJDei2eeQwGEL4T 9O+znI+GJoGSHO0nSwfdvQUSzOiFG71VFlN/Ax78ATWNU+fycBvJ+RoA1bzviuQ5ut/Q ng0dP8XQ7XXonNSj614+jkXnDbq20YXeDwwS9ZoZnbOeZmWsSpv2VIutUyOMAceT83vg yqxyOGgGqzXkyTn6vwVJzLRZ3v52VvI0Z+z9QchI5Borkmgh5fpjU6R4zzRVHKbVPsP/ FHzA== X-Forwarded-Encrypted: i=1; AJvYcCX6rf1t2zk/6sOh3oIpln8A0pHuiLNMj0yj3M2xFH3KIjMfp2/1rX52vkRJ+F+OWLx9+GcehqLY6erRANZ9RntS+lQ= X-Gm-Message-State: AOJu0Yz4sktRhHjxLPIPBq/CJfRjHLiB7b2Ud6ufjh6boePek7FUIaSZ RfqnAWrf2eTx98Xew7j5U/g+CFiYrI6i8ciEaLJHsEjK7rmShSs9 X-Google-Smtp-Source: AGHT+IFFu/goz1FrmbpZ66r+D/WoNAbblyTq352/gs8sP6e/6QY7jw8AvLvp16EdF9oF+hm9bcXL7A== X-Received: by 2002:a17:902:ced1:b0:1d9:a2d3:8127 with SMTP id d17-20020a170902ced100b001d9a2d38127mr470046plg.52.1707451640097; Thu, 08 Feb 2024 20:07:20 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWP7HPoDU+RIV0hbhhVZtaTKZ52ozQSAI2pmi/5/+M4R+kofDRBWlviHowAmZf4KAFWB00xVtEVtRwexebBtYOr5AunnGUHJHavtyq1fO76hzOsbjlXb6ahRaPTVniCLqPQVTDLcmGtA+rqFV9qPMYDXBfd1cJ61lPC4OPXYwCm56ZKxVkulgzpnfoB2DhozBGuIbcQ/lmUZzREPwYFIWjmTN+9/y2JuUZspdbHfxqeXh60l+d35X6YqgaPJBYzRA+h7ETU6QLl1X17oZ7YJWJ35m4Mn4X9nbYlY0JqtdjgCe50tzxXbXK82HqDvWSALZFykpj6UqRgzuzDJqpHvP6D7mmZ/eNM3hPd+94XK6JgFgmhKcw1oA== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id i19-20020a170902eb5300b001d8f3f91a23sm535557pli.258.2024.02.08.20.07.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:19 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 16/20] bpf: Add helper macro bpf_arena_cast() Date: Thu, 8 Feb 2024 20:06:04 -0800 Message-Id: <20240209040608.98927-17-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4E9F940009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: qkaiys8qr7h3nyyxkt1mt1e577mzqkeh X-HE-Tag: 1707451641-345264 X-HE-Meta: U2FsdGVkX18wmtLP74DCYm/1ytWs6K4jnZY66WoIAKSuStMRiYBCK14IcUgfvKBxlMaf4W/5dC+by8T7p1Va1tJxWj9oCs0zAbO7vOgFAbaBNpHCFFswkcGBCCOLnG6llwbC3Gw0PhmQLaShDYre1aGKckD7LRscSpIgDeqCROW5bFkpFOZDta6RIFSY8faZzmR5mlE7aN9C2nIRSd17fSHc8w7ELXJStljNhQoME8alNwngvXnD1pTH0V25Wi+5F16flvWtAo3PGHDHHz56fY8grCpC5JM5f1RtklMmkq62U4lQJig1MG6EX8JzbPqmUe7fIAjlVBJLNImACX+M0bk3pODfgm2n8QTHq4cpI+xQYxyGhNZ/xb8OTp93ZmK5vcHeOIdYaCdSbv1mCt0jFbkysv/KQznelXPZULT71y4/BHWs8W5FuXUggr2WEHSy6ONk6HOAmZEIn3mnsgBaA2RpkNwe7ygMw6tGcyS0fChGV1mR6gExfTa76DdbfxDA4ZYgS+jCe9qd9vMpPgZkOAaw2FGzEF+WDhMEMFbKXwj0/T7Ct6uuWjP6i4JZxRde0bT/AbLZSzFjZM6tjnIBZ5RLP2R5vux25T9uOQbWRBKflwxSBdGHEG3UGd5L8Stv98fux5MCf4DLBD+4lyG9ohcUBTSJV2m5WUcRMmOef9VKrbd1xNCAQYK0tGNd8o169ZCuw3+ePh83V1oj06CHmtDY+zjcRnG9Ddd8eXoxh9wg+x43+yfmJR4QkQlsCQDK0Nu4eMxXEq/efeJcmH0e5oWeS0CsfHUyT3jXpvKLAuDJgvFhCqJhxi2eTmAESFyrEZDOg3qyO3w5hLnbuieGy7q4EMUK2XnKN1LpW6skApCo89mhsiLgk6U8uHbDO+AsHafeF6xA4AIxXR52aFp4TSLwPvXhAwu49TleRw/kBMffJ9xZX0B1m9y2Omaxt3DvNx6m8uj7XBkW7+0iatL vvpfzZCO eMjqdB7tZEnu9kPdXahDtYHNbnGsH1mAJPqApgiCrDgGA+2TKgSB8lsKm8eY+qQjR3WEOK15QikO0V12f3pjtkZyo9Ooa/qpCQ+NTYLLIFUciBA3rNdYsJSxSbEiKacOd5v0nMM36reC3QN40pkRbyXtNelGfhDgtXnRzlg3x63ngj9wDEnVOEMgsZTwCUR6nu9B/JYMXfuFwp/JSGtqnG4qvYKgnm2nEZfF7kD0irCDTpfRi0Fw0fQPmoHCO6v72yAXtGujHnrxyFXUG8ZpjYBf5JRvWicaZL5CGSObe+99kmdod8+ja9hszWsberj+aRkx/B8+RkVdA46h/p1huEcoiKWVi4nqilP1nGwmuem8pQyYhMRRuQqtTcEon+7Fn1nXyiASFA+O8wCIgs/0pcVL6RrV947K9fUZ9VNYBL2WU9D5cqTsOLflVlJu7QMh/p5gbWfY4U6ddb/IjSxKCyAcE5cpz65n72EMQm8dtxkBwbNpjbhpB+ih/dCUoVZ1AkniktTMDwoLtZ21ThT3M4JCKRUa5aOHpEc0PNc3o6YwhQLNh+K6nBr9Cezjy8e6M/+AXdIXxDDX1lwQ288JL08QXpDq62UJk+Unz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce helper macro bpf_arena_cast() that emits: rX = rX instruction with off = BPF_ARENA_CAST_KERN or off = BPF_ARENA_CAST_USER and encodes address_space into imm32. It's useful with older LLVM that doesn't emit this insn automatically. Signed-off-by: Alexei Starovoitov --- .../testing/selftests/bpf/bpf_experimental.h | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 0d749006d107..e73b7d48439f 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -331,6 +331,47 @@ l_true: \ asm volatile("%[reg]=%[reg]"::[reg]"r"((short)var)) #endif +/* emit instruction: rX=rX .off = mode .imm32 = address_space */ +#ifndef bpf_arena_cast +#define bpf_arena_cast(var, mode, addr_space) \ + ({ \ + typeof(var) __var = var; \ + asm volatile(".byte 0xBF; \ + .ifc %[reg], r0; \ + .byte 0x00; \ + .endif; \ + .ifc %[reg], r1; \ + .byte 0x11; \ + .endif; \ + .ifc %[reg], r2; \ + .byte 0x22; \ + .endif; \ + .ifc %[reg], r3; \ + .byte 0x33; \ + .endif; \ + .ifc %[reg], r4; \ + .byte 0x44; \ + .endif; \ + .ifc %[reg], r5; \ + .byte 0x55; \ + .endif; \ + .ifc %[reg], r6; \ + .byte 0x66; \ + .endif; \ + .ifc %[reg], r7; \ + .byte 0x77; \ + .endif; \ + .ifc %[reg], r8; \ + .byte 0x88; \ + .endif; \ + .ifc %[reg], r9; \ + .byte 0x99; \ + .endif; \ + .short %[off]; .long %[as]" \ + :: [reg]"r"(__var), [off]"i"(mode), [as]"i"(addr_space)); __var; \ + }) +#endif + /* Description * Assert that a conditional expression is true. * Returns From patchwork Fri Feb 9 04:06:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D799C48297 for ; Fri, 9 Feb 2024 04:07:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98D878D000E; Thu, 8 Feb 2024 23:07:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 916BE8D0001; Thu, 8 Feb 2024 23:07:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 769268D000E; Thu, 8 Feb 2024 23:07:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 61E838D0001 for ; Thu, 8 Feb 2024 23:07:28 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 461D9A265F for ; Fri, 9 Feb 2024 04:07:28 +0000 (UTC) X-FDA: 81770930976.06.8E61D61 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf27.hostedemail.com (Postfix) with ESMTP id 3FD0A4001E for ; Fri, 9 Feb 2024 04:07:25 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=knk6lra0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cygxMwXpUKU81RFmqoqePBosJeNZ00y1PK/p5gN4GLo=; b=AqzI/5oqtqhJtrWRJqfc0PXiI98UHRrD7S3Z24p/ZBn+nD30nZ1fLYkGzqyYsoN6txPSFz hJiy8t63S8nghPd0F7z+A1Kcidrh88B6B1m1eP1u9iZ0ga2xA/GUV8KFEEQSVVN+AqcjZh 4qJxSRqH1KevZGipbD58Qf7HW8wfL6I= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=knk6lra0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451646; a=rsa-sha256; cv=none; b=xxDbIKhO64+GOc0nS4rrogVp36hJNhONUqP9VlgkUXevzqv/7PYjPb0WIvZJ/UAGiYPuAB XiB4d9Bj1RbiLP+7BI/Yx8cCmhTtLQNkIqjv7KmQI/8l1y0mDuDde0Vo6ihNGt6hFWH86i g+BwqZh35RULGW5xOcqhZVNjipvqt4g= Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6e04fd5e05aso440875b3a.0 for ; Thu, 08 Feb 2024 20:07:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451645; x=1708056445; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cygxMwXpUKU81RFmqoqePBosJeNZ00y1PK/p5gN4GLo=; b=knk6lra0qujXKcs/FbTt47TGViFp0CPHs3lKyf7z3AfBOJOkIi6ZkZOfiKfJ19RF+l giXdKkXH5F5tBoIju6dK0z5ibKk7sRmlQa+mNRyPXteDFqopqVgTfEhPtG1WdZ6jnLLK UbSqx/xA/vvtmfmuNpP6GEDPvAIyCjwiROtWeKfxA1pG7LqtbWvkm2fHO4940Ej9hUdU ygymofC/HxInly/oW5owRXyyWhZ9Ozng27Zb4hmAWOvTsr7JY0fZJGP+UdSTz62fXFkz TK1AQxkSUPRR1eBNqwefr6aaPnuTQSP1LhCbPg1YUmBtKCTv9vGEcOpHxgOBWEzlg1DV cx0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451645; x=1708056445; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cygxMwXpUKU81RFmqoqePBosJeNZ00y1PK/p5gN4GLo=; b=WuCFVXXUhQCOGYXbM/GKEqN+nXghzfm+AWjs1VYPzXR05Q7c6kY8TEolWRBzxhNaxj 0Tv9jGSQbmsKUYdXd381qWxGp7cgSaAdUbE2HCnc7lGP/0pccaqfvv+Pqi2iWL/weIcg QvLyKy6A7+tj5FWWxVAOvn+xsAqIRMUe6HNlwTTvCFE0fJLnRHr6sC2TP6DoxOmQNU1c 8oLmTvEMOrdHcZ8fYWPMbfwGcHnMq9LXHS2qrV3py5x60UapPGy8jJPQ7iDFrDRR1Ej5 yQr9YDIqWsMASQ/5dU2/Q8mC+ZsY62erCNQ4YzVVb1ZO9qLxHnpUpcOdGK7WWmj3vw+m 9n0g== X-Gm-Message-State: AOJu0YwUe2uF6jZZl6auA8KFDDUQWr6j/KQSvn79WIvqozqXXj5ccipq GHJQmgACe1t2Y52v/S/MZBg+7yNuVcmiAGQQlKMx9mmwYUwfBQk/ X-Google-Smtp-Source: AGHT+IHaHqKC2gjT6eMMFa5TCSDFBnv1yNEkB/syWL6KEWt6hbk/NU6aBcqi1y0pyAXerAeLHX9DIA== X-Received: by 2002:a62:f901:0:b0:6da:bcea:4cd4 with SMTP id o1-20020a62f901000000b006dabcea4cd4mr650234pfh.16.1707451645004; Thu, 08 Feb 2024 20:07:25 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXKa7AoYoSTpHf4QY1MuRhoeHy/otD5YbJxf7F1nq3Kf80HMJe7T0JPldiANHIWK0RPvaLH7ALpvryppvlFYP3mvzADeCO+6W0Df2OEfEyZCy1EthA0awtTR8laxGBdXgAA+oT4niKbJetncz2Fps4xueuExRoXrm/pX3Gcvt7rdxSEJ2rWTFSq+W25Wl3Drj8BYjUFe9Ck6okd8ulBbCOHQJIDRJ8dlA6f3o17pQoFudGpIsq00Zyk6sAn17YSRtSDazpAvv0earkDd4DsTwoFLj0GNPgNctSu+ObZYiRoMJJK+zmn/wkbO9+fovsxt3aeg00hIDWktHbfbs2uiBwomv1vCaKMYx0lfUcmtBHD7mvIUD0euQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id v14-20020aa7850e000000b006e0825acbc3sm590230pfn.77.2024.02.08.20.07.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:24 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 17/20] selftests/bpf: Add unit tests for bpf_arena_alloc/free_pages Date: Thu, 8 Feb 2024 20:06:05 -0800 Message-Id: <20240209040608.98927-18-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 4yrsjbjuse5rhw7drrxj78e6xc7xwxim X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3FD0A4001E X-HE-Tag: 1707451645-440812 X-HE-Meta: U2FsdGVkX1/t3vs+sEfNTG3SnftZf0y55KacwDl9akPrxkbM4gMYcVxLob3mW1tHOAVD/XOyF7raa3SRY6TjVO9I0TaV8ElTyh8OppjKXFD171z9Fn29lkFIUav8K5SDvonAURwMgdM+84w2c0jUcEsdKl92uR6gqzGih4tRNbyQL7iILNzpNs01BCgShxBcpRgAaFvYAoa69kWMoh3jRCw/54Lh5LJpFZhRZJiWJup8PzfVeGlDX3wR21e6m2F186edxK8dvaudr3gGwU+ONS7BGHZY0EuNDql+rYYGmYu3jMmEiZ427iZtpVdqXp+WzXz+fiUGposPF0gWRaO+H0PbL6VUSSdAZoz8Jp+GdFnXgyU9nyFPisf55Pc39NW/qxLe284z3qzQHxO+9wYJbMQF6ekrnfaA2z6oNWhWgc/uEUbbe2+bKbXGXucfF9b1J9enKreyg5r0vB9lQUKTOtomJO25xKJtB24VzHuwtAO540si1baZ26+fIlXAOJBRvfd21i3GyJAy2hB24zQsq6PZ0w57gozsMREN+VrGrtG/DB+cl//OZcWWn/bqxjKLV3trLy/pDIazwALdjlj8NGSAgK34ELEKe8LxNW5CSiIO8BDbvCONIv7lzUgHlpc1qk8ROVqpMXv5fZeocmmpO/UGRpyeUrOFZ2zZ0Kc92/SwRvDHHIEntGW8Mnz9H5dT2/i4P6cj+Iry8hLdPKpgUF52TfPa8qerudnKYa1OQuE1XppxH8spWgv3/FZuCwNwIFKDdzU5IoiRrI5bRLWGAX/mTpR29mzmX2+m9YphWAEebXb0XELvxsA3Grtgb7D1JWu7jVgYD5rsts3VnSQtYazCVg/QJqbfTP9A0rz9A8zmAHfKfqCTvAtR8o6n7Rp8gRe1YzHKOajT0iuf3/TYywmpZvdVVzC83EWq2uNv6FVs9kFJYfs3YKlJWDG2e/kBOgXW/R88ZEtr/uAQoCY XHGh5pr7 bExr0l3dK34s11K9KdyXn06HwGxJ6soYJBlEW3lihI5j4wxwWfMxMwvVBGvjosAWBvPl5I6DroWjfcR3J8/G7HfsA+uiubeaCi2YJ512/4pIVnB62gLJPbfm4k/pG/u1QEZFHlb7ge8PVGmOTUHR6Ti69ct5z26s7MerRuuGffRCZWi70Gor48+Tf0uaWZq2fzE8DSwqSFfM+rdW8pgqDA5rz31BFqjGqLILoEPj5jtbITWexZzDKXEDSpIGN3JobDRWnPwioc+BnG6ryWyFZHIX6yPib/5tWrIfUxxi7jqIR+ZizLjUY9GZKSfk+tSr/w0/j1IgsNoR2SMIi+1yUrEA7BjP5hE2cVhcneY6ygeF34MhWJCM9aOIfXv/Ml/XJY67K5tN4fznZdaOflLXn7JQxmlkqubu2NXhMYi/+MBRTCcv/aIcN04YahJUI05HEMrh7WHMpN7PH/Egd//yXwIkuxwsEbeH1LBNv4DWyDgqOpEJ5JlA90uAlHH1qqiI0Ts4WRqTI8nzdXXLIYzoqN2cy2fAYWLbegkJa0fP50BUmocEhJAgL+r+eeIerW7H/v4YvF03VFrm0ot9DHJ0sLeO5dkDEHxS5l8MK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Add unit tests for bpf_arena_alloc/free_pages() functionality and bpf_arena_common.h with a set of common helpers and macros that is used in this test and the following patches. Also modify test_loader that didn't support running bpf_prog_type_syscall programs. Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../testing/selftests/bpf/bpf_arena_common.h | 70 ++++++++++++++ .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_arena.c | 91 +++++++++++++++++++ tools/testing/selftests/bpf/test_loader.c | 9 +- 6 files changed, 172 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/bpf/bpf_arena_common.h create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena.c diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 5c2cc7e8c5d0..8e70af386e52 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -11,3 +11,4 @@ fill_link_info/kprobe_multi_link_info # bpf_program__attach_kprobe_mu fill_link_info/kretprobe_multi_link_info # bpf_program__attach_kprobe_multi_opts unexpected error: -95 fill_link_info/kprobe_multi_invalid_ubuff # bpf_program__attach_kprobe_multi_opts unexpected error: -95 missed/kprobe_recursion # missed_kprobe_recursion__attach unexpected error: -95 (errno 95) +verifier_arena # JIT does not support arena diff --git a/tools/testing/selftests/bpf/DENYLIST.s390x b/tools/testing/selftests/bpf/DENYLIST.s390x index 1a63996c0304..ded440277f6e 100644 --- a/tools/testing/selftests/bpf/DENYLIST.s390x +++ b/tools/testing/selftests/bpf/DENYLIST.s390x @@ -3,3 +3,4 @@ exceptions # JIT does not support calling kfunc bpf_throw (exceptions) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?) +verifier_arena # JIT does not support arena diff --git a/tools/testing/selftests/bpf/bpf_arena_common.h b/tools/testing/selftests/bpf/bpf_arena_common.h new file mode 100644 index 000000000000..07849d502f40 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_common.h @@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once + +#ifndef WRITE_ONCE +#define WRITE_ONCE(x, val) ((*(volatile typeof(x) *) &(x)) = (val)) +#endif + +#ifndef NUMA_NO_NODE +#define NUMA_NO_NODE (-1) +#endif + +#ifndef arena_container_of +#define arena_container_of(ptr, type, member) \ + ({ \ + void __arena *__mptr = (void __arena *)(ptr); \ + ((type *)(__mptr - offsetof(type, member))); \ + }) +#endif + +#ifdef __BPF__ /* when compiled as bpf program */ + +#ifndef PAGE_SIZE +#define PAGE_SIZE __PAGE_SIZE +/* + * for older kernels try sizeof(struct genradix_node) + * or flexible: + * static inline long __bpf_page_size(void) { + * return bpf_core_enum_value(enum page_size_enum___l, __PAGE_SIZE___l) ?: sizeof(struct genradix_node); + * } + * but generated code is not great. + */ +#endif + +#if defined(__BPF_FEATURE_ARENA_CAST) && !defined(BPF_ARENA_FORCE_ASM) +#define __arena __attribute__((address_space(1))) +#define cast_kern(ptr) /* nop for bpf prog. emitted by LLVM */ +#define cast_user(ptr) /* nop for bpf prog. emitted by LLVM */ +#else +#define __arena +#define cast_kern(ptr) bpf_arena_cast(ptr, BPF_ARENA_CAST_KERN, 1) +#define cast_user(ptr) bpf_arena_cast(ptr, BPF_ARENA_CAST_USER, 1) +#endif + +void __arena* bpf_arena_alloc_pages(void *map, void __arena *addr, __u32 page_cnt, + int node_id, __u64 flags) __ksym __weak; +void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) __ksym __weak; + +#else /* when compiled as user space code */ + +#define __arena +#define __arg_arena +#define cast_kern(ptr) /* nop for user space */ +#define cast_user(ptr) /* nop for user space */ +__weak char arena[1]; + +#ifndef offsetof +#define offsetof(type, member) ((unsigned long)&((type *)0)->member) +#endif + +static inline void __arena* bpf_arena_alloc_pages(void *map, void *addr, __u32 page_cnt, + int node_id, __u64 flags) +{ + return NULL; +} +static inline void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) +{ +} + +#endif diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 9c6072a19745..985273832f89 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -4,6 +4,7 @@ #include "cap_helpers.h" #include "verifier_and.skel.h" +#include "verifier_arena.skel.h" #include "verifier_array_access.skel.h" #include "verifier_basic_stack.skel.h" #include "verifier_bitfield_write.skel.h" @@ -118,6 +119,7 @@ static void run_tests_aux(const char *skel_name, #define RUN(skel) run_tests_aux(#skel, skel##__elf_bytes, NULL) void test_verifier_and(void) { RUN(verifier_and); } +void test_verifier_arena(void) { RUN(verifier_arena); } void test_verifier_basic_stack(void) { RUN(verifier_basic_stack); } void test_verifier_bitfield_write(void) { RUN(verifier_bitfield_write); } void test_verifier_bounds(void) { RUN(verifier_bounds); } diff --git a/tools/testing/selftests/bpf/progs/verifier_arena.c b/tools/testing/selftests/bpf/progs/verifier_arena.c new file mode 100644 index 000000000000..0e667132ef92 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_arena.c @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" +#include "bpf_arena_common.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 2); /* arena of two pages close to 32-bit boundary*/ + __ulong(map_extra, (1ull << 44) | (~0u - __PAGE_SIZE * 2 + 1)); /* start of mmap() region */ +} arena SEC(".maps"); + +SEC("syscall") +__success __retval(0) +int basic_alloc1(void *ctx) +{ + volatile int __arena *page1, *page2, *no_page, *page3; + + page1 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page1) + return 1; + *page1 = 1; + page2 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page2) + return 2; + *page2 = 2; + no_page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (no_page) + return 3; + if (*page1 != 1) + return 4; + if (*page2 != 2) + return 5; + bpf_arena_free_pages(&arena, (void __arena *)page2, 1); + if (*page1 != 1) + return 6; + if (*page2 != 0) /* use-after-free should return 0 */ + return 7; + page3 = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page3) + return 8; + *page3 = 3; + if (page2 != page3) + return 9; + if (*page1 != 1) + return 10; + return 0; +} + +SEC("syscall") +__success __retval(0) +int basic_alloc2(void *ctx) +{ + volatile char __arena *page1, *page2, *page3, *page4; + + page1 = bpf_arena_alloc_pages(&arena, NULL, 2, NUMA_NO_NODE, 0); + if (!page1) + return 1; + page2 = page1 + __PAGE_SIZE; + page3 = page1 + __PAGE_SIZE * 2; + page4 = page1 - __PAGE_SIZE; + *page1 = 1; + *page2 = 2; + *page3 = 3; + *page4 = 4; + if (*page1 != 1) + return 1; + if (*page2 != 2) + return 2; + if (*page3 != 0) + return 3; + if (*page4 != 0) + return 4; + bpf_arena_free_pages(&arena, (void __arena *)page1, 2); + if (*page1 != 0) + return 5; + if (*page2 != 0) + return 6; + if (*page3 != 0) + return 7; + if (*page4 != 0) + return 8; + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index ba57601c2a4d..524c38e9cde4 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -501,7 +501,7 @@ static bool is_unpriv_capable_map(struct bpf_map *map) } } -static int do_prog_test_run(int fd_prog, int *retval) +static int do_prog_test_run(int fd_prog, int *retval, bool empty_opts) { __u8 tmp_out[TEST_DATA_LEN << 2] = {}; __u8 tmp_in[TEST_DATA_LEN] = {}; @@ -514,6 +514,10 @@ static int do_prog_test_run(int fd_prog, int *retval) .repeat = 1, ); + if (empty_opts) { + memset(&topts, 0, sizeof(struct bpf_test_run_opts)); + topts.sz = sizeof(struct bpf_test_run_opts); + } err = bpf_prog_test_run_opts(fd_prog, &topts); saved_errno = errno; @@ -649,7 +653,8 @@ void run_subtest(struct test_loader *tester, } } - do_prog_test_run(bpf_program__fd(tprog), &retval); + do_prog_test_run(bpf_program__fd(tprog), &retval, + bpf_program__type(tprog) == BPF_PROG_TYPE_SYSCALL ? true : false); if (retval != subspec->retval && subspec->retval != POINTER_VALUE) { PRINT_FAIL("Unexpected retval: %d != %d\n", retval, subspec->retval); goto tobj_cleanup; From patchwork Fri Feb 9 04:06:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B529C48297 for ; Fri, 9 Feb 2024 04:07:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 918878D000F; Thu, 8 Feb 2024 23:07:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C8378D0001; Thu, 8 Feb 2024 23:07:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71C5E8D000F; Thu, 8 Feb 2024 23:07:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5C9D88D0001 for ; Thu, 8 Feb 2024 23:07:32 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3B39580512 for ; Fri, 9 Feb 2024 04:07:32 +0000 (UTC) X-FDA: 81770931144.08.449E45E Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf07.hostedemail.com (Postfix) with ESMTP id 5B9E240009 for ; Fri, 9 Feb 2024 04:07:30 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PZ8+8lj9; spf=pass (imf07.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eb+bGqWXAT7EYhVLB+w+aV2CuyvwLgJLPbtGe0eOYJ0=; b=oezT8YOr+UxjBdGn9kd577V0wN7+cf72+ObkUNf2T+5aXjdJLk4gxyzUdRkFup+TRpKjj8 ThWkLoNVitwjvtuxvKh4nD4b2ZyMUd+595GCITiJ9BnHrjXYL0rzySTXP8Rh9xOfebznrt xm53UZTxhHrZINW7jU3YMuNocp8cG28= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451650; a=rsa-sha256; cv=none; b=qOf1FXZL6SIQhDyXrVANGe+O3xxPGm9fY1R7p0FZ1jf+bDEty1M7OUCyAue1YbOHay+MBa RO5glC/MEYQhmRRFe76UzcUU//7k77T0EkzYXRTbyJJpCxuR0y7DDczltMsSplQJU3PUeK F1bnGWepwGgTtyL+e9htHyL6EXEd10Y= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PZ8+8lj9; spf=pass (imf07.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6e067ececb1so348143b3a.3 for ; Thu, 08 Feb 2024 20:07:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451649; x=1708056449; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eb+bGqWXAT7EYhVLB+w+aV2CuyvwLgJLPbtGe0eOYJ0=; b=PZ8+8lj9c+dnJnTHmRrycI/t84FhgrUpvDbHJq0qNabTvPwIfKNwrsGtecMybB1UGu mz8RaHD3q16eKyxIwM+hqJi9CggvhPVlYuJlmsiyGqNQ4sHBqjZ22G3TH0+gRcU0oLxT yc64SdEzh2efSj11zaY7QrcxIq9SStTz5yayGcBfWPitXjDmy8jzhMCOgNxPXdsQxPHP hWWyAdTuu94RUmUgU2yDyWt1L1L9THSTPqK/zMS3Ui7yiMvFylV+GEQwq59YCJRtMPvE rvH16uxqdqzNpxTZfgZPnwvmudTYrZLk49FDzOADC8UANA+iO/9sviMbywfOe9YMM/fo /k6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451649; x=1708056449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eb+bGqWXAT7EYhVLB+w+aV2CuyvwLgJLPbtGe0eOYJ0=; b=REbNyqhEzJTobnFlCau6VZy/HeqtD/wZoCavtX9Ass+Yl9CTKaO1RUHbSAEcrXvdYy 3oNQySjIe+8O0rEBQcwGFAgAMWP6mdU1Eekbk1A2i9YiaRM3gDoT0BiG4RjG19dx2g0N IZnMC2mVMiLuVT0Uu8zIq7QgdUakIOsx0I5XelgmGhn1Svue4BjT+7uTpLVb9Wb4DSLz qHcfV/mjfq7U7SbsKTAcD3rznnf+rHt9dK9zZ+fEBTgCSixwVEVtmGjZX99j36eec4Fz Zg/RR91xEIjC9JjjyvHYrLxttz99vaOuiJGNS5f/t8W4q/x0D9jVRbwOPz7BVvhAfhe/ qYDA== X-Gm-Message-State: AOJu0YzTs4srjdodwtqwFJcePNhTGvYXe5xlUGxrxcdtelEyzhBjGDaR 2gaLZ424fWprBzmvfwINBS6MTHs8KX2+I966jG08ZGPccyYguEaM X-Google-Smtp-Source: AGHT+IGG5TNU//6saVXjiaDd3768cg95qj6j0b0wUL2WN3QIR5+8d+0UOYlVtQ8UWRPRvc2Z7Z+eHQ== X-Received: by 2002:aa7:8058:0:b0:6d9:bb2f:3a69 with SMTP id y24-20020aa78058000000b006d9bb2f3a69mr491139pfm.28.1707451649186; Thu, 08 Feb 2024 20:07:29 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCUjtlzegU4md9eHPzOxI0W1dNxIUvJ3ODioqbSz+ZgQBvX4dmxdiUbBS7O0w8vBr9vAaA6uqWY3U+OSwiXqgbz757P/uWzG9Bl/sXQ7GtnQIvtx20f9N7sMwORHdhPk1jSJQUGn+dV9YbQOAp9CA6N76SE+oDBaw4b2sQToICMcKVjX46jljS+JQMlD//n/vyjn+4CQctUA2dDpLFz731lYn5YNR6kHFjmOpnCVsqCimcT4OO69W6fLm3kO0bz103wJeDiTKCf2y2EhDA8TpqLgZ0/vRkNtoRuBNvX+x8dmjKsHDq6+BYewLzKAphfVEbgNcG8GaOwoAFr2ClxgPEgiiSFHz8ev2mjVD2N/VkgGqLN8yyjFNg== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id z19-20020aa785d3000000b006ddcf5d5b0bsm582570pfn.153.2024.02.08.20.07.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:28 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 18/20] selftests/bpf: Add bpf_arena_list test. Date: Thu, 8 Feb 2024 20:06:06 -0800 Message-Id: <20240209040608.98927-19-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ww7ekx9omwqjueyxpg7a5xi8ep1fgbab X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5B9E240009 X-Rspam-User: X-HE-Tag: 1707451650-717709 X-HE-Meta: U2FsdGVkX188AMqyEQT9SkhQDny+UKLXSeXZ3H2oKQGCp1C0YiGeg8wB7v6SG3bNACMUoPiSakreffPtqZje4Zx4B+ghmuUhnL9s6ApULAa3OwOF7vKmU8VjNoVLRB+hDYWUnTmZ1QDk8eOA/DQV3oVnwzebQvvUcIKnpgCoE/Eu8TrTFCvncxvIwu3stAOuOSMLUAQuHT9fNhpXVUGrrKouqKhtxFg8i8AX1HslibEnuDioZMGRLLXiBx5tDfNr/PVt4kkAIE/+o6uQpPVCv0UeRWs1Gs43SDgFbT3CV0shF2SPqRsr51I+pzT5JaKunJEP6jCDmI3nLk23qqSBcqH6cjlhV1i7EUrwu42upD0jZmuIpEg1wwMhj1nwXeyexxVt5h4W99mvMSZnjd6S5jml4mklCe2iwVzj3jUXKtvAOfLUBWYzDFLMKdExD5un8o30m/wvAVgN/4aNYBq+zUWkpI2w/n4QVQEprmfCaNf5571tw26D3Or0nt7eF6gQy2WgLtexHTp4lDjmL4NyNGOcT9/eI691/eMalkDfuuabEVtrStqGkbI176jWySfXEWYNKXUOAth0CR+1UeFZLFLMZkA+jnTglxiFFTHqbEx4cjF0ZF/PE9lOhuW8tO6heddCX2n0Ob02oQVpYCCOxOMx7aROg4W/UbC9ZCu984A5lr1eA+bclqHysdmlYXfNTAAphb9X4FSf9c6d2GsSV8TIDh1pg7BgeOyIix3nvxxN/yNfli6sZTdkn9NmZ7Sd/cLBapLLPtPH4hlEovWOVddLV/MmLYDuFndsQ/u+Tu6zNZW29SNCEtGB+RblNCM4wpmzCshd3YCVXm4tXS7UKBv2+Jl14C657dWElZ8itr9Z1A9gNGAx4to7KdUncx/pDpI8eO8vm/4LQUEZyHivUEHltZarmv1EtwgAVm6C5KGlkTgN+i0/vMVTJP4rlGAxlVuTH3ZkuZRtG52kZbH Mwd5R4H8 YVA2IeP/sex1FVFduORmL2rI62Qo5MFMX8NKqKGxUAWgCLII62yVUrsVgsoVHNFGC5z2AINv63WNzIePuNO80jibulAm5lPKTwRSjfHgKlHL89hH5X4fvd27cYw8QriS0l3aRKQP46CI6jOFd02iv5DeVJG7L26jCblaVvqzn/hQlAzV/sqetRKJ3p3tpxNGYyCIz9MXtV2L8ZkaqN9qLGHXGW2I4NRz0DIzK5KlT9DUREsftuHfT0grtoUTf7SeksXgJ1Cq1SLwlOWu/K6Seo6I2lAjoAB/dALTg0Yf4l0YcN6zAu6cQH7l3UnDjolG3RTrQhlLKz7orJhybiCmC6tq5s8Cv9EaavuEhZIAKLvMCKEJfdK5N0iaEBbaStVSiJpfM3XGVf08msTRitOLEaXt4947WhxusRgVHFWhLrXkY02BXAO0SvDiWJGYhWQ2+Da8YpOEtmduUW77fa/iaZiHDXVOEjVxxvcwo7vD56Bdpo4M1MZQKHTOPmz49tynj1qJIZKF2lg2/K/k1ALfMVWoqNRtm2FPgJZqlPUFPTFF+Myy5q/mSiY9xpenvWbPsXKPyD3ARXEl93aFC/6gWt24Nf4SvD/hxnv96 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov bpf_arena_alloc.h - implements page_frag allocator as a bpf program. bpf_arena_list.h - doubly linked link list as a bpf program. Compiled as a bpf program and as native C code. Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + tools/testing/selftests/bpf/bpf_arena_alloc.h | 58 +++++++++++ tools/testing/selftests/bpf/bpf_arena_list.h | 95 +++++++++++++++++++ .../selftests/bpf/prog_tests/arena_list.c | 68 +++++++++++++ .../testing/selftests/bpf/progs/arena_list.c | 76 +++++++++++++++ 6 files changed, 299 insertions(+) create mode 100644 tools/testing/selftests/bpf/bpf_arena_alloc.h create mode 100644 tools/testing/selftests/bpf/bpf_arena_list.h create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_list.c create mode 100644 tools/testing/selftests/bpf/progs/arena_list.c diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 8e70af386e52..83a3d9bee59c 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -12,3 +12,4 @@ fill_link_info/kretprobe_multi_link_info # bpf_program__attach_kprobe_mu fill_link_info/kprobe_multi_invalid_ubuff # bpf_program__attach_kprobe_multi_opts unexpected error: -95 missed/kprobe_recursion # missed_kprobe_recursion__attach unexpected error: -95 (errno 95) verifier_arena # JIT does not support arena +arena # JIT does not support arena diff --git a/tools/testing/selftests/bpf/DENYLIST.s390x b/tools/testing/selftests/bpf/DENYLIST.s390x index ded440277f6e..9293b88a327e 100644 --- a/tools/testing/selftests/bpf/DENYLIST.s390x +++ b/tools/testing/selftests/bpf/DENYLIST.s390x @@ -4,3 +4,4 @@ exceptions # JIT does not support calling kfunc bpf_throw (excepti get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?) verifier_arena # JIT does not support arena +arena # JIT does not support arena diff --git a/tools/testing/selftests/bpf/bpf_arena_alloc.h b/tools/testing/selftests/bpf/bpf_arena_alloc.h new file mode 100644 index 000000000000..0f4cb399b4c7 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_alloc.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once +#include "bpf_arena_common.h" + +#ifndef __round_mask +#define __round_mask(x, y) ((__typeof__(x))((y)-1)) +#endif +#ifndef round_up +#define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1) +#endif + +void __arena *cur_page; +int cur_offset; + +/* Simple page_frag allocator */ +static inline void __arena* bpf_alloc(unsigned int size) +{ + __u64 __arena *obj_cnt; + void __arena *page = cur_page; + int offset; + + size = round_up(size, 8); + if (size >= PAGE_SIZE - 8) + return NULL; + if (!page) { +refill: + page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page) + return NULL; + cast_kern(page); + cur_page = page; + cur_offset = PAGE_SIZE - 8; + obj_cnt = page + PAGE_SIZE - 8; + *obj_cnt = 0; + } else { + cast_kern(page); + obj_cnt = page + PAGE_SIZE - 8; + } + + offset = cur_offset - size; + if (offset < 0) + goto refill; + + (*obj_cnt)++; + cur_offset = offset; + return page + offset; +} + +static inline void bpf_free(void __arena *addr) +{ + __u64 __arena *obj_cnt; + + addr = (void __arena *)(((long)addr) & ~(PAGE_SIZE - 1)); + obj_cnt = addr + PAGE_SIZE - 8; + if (--(*obj_cnt) == 0) + bpf_arena_free_pages(&arena, addr, 1); +} diff --git a/tools/testing/selftests/bpf/bpf_arena_list.h b/tools/testing/selftests/bpf/bpf_arena_list.h new file mode 100644 index 000000000000..31fd744dfb72 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_list.h @@ -0,0 +1,95 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once +#include "bpf_arena_common.h" + +struct arena_list_node; + +typedef struct arena_list_node __arena arena_list_node_t; + +struct arena_list_node { + arena_list_node_t *next; + arena_list_node_t * __arena *pprev; +}; + +struct arena_list_head { + struct arena_list_node __arena *first; +}; +typedef struct arena_list_head __arena arena_list_head_t; + +#define list_entry(ptr, type, member) arena_container_of(ptr, type, member) + +#define list_entry_safe(ptr, type, member) \ + ({ typeof(*ptr) * ___ptr = (ptr); \ + ___ptr ? ({ cast_kern(___ptr); list_entry(___ptr, type, member); }) : NULL; \ + }) + +#ifndef __BPF__ +static inline void *bpf_iter_num_new(struct bpf_iter_num *it, int i, int j) { return NULL; } +static inline void bpf_iter_num_destroy(struct bpf_iter_num *it) {} +static inline bool bpf_iter_num_next(struct bpf_iter_num *it) { return true; } +#endif + +/* Safely walk link list of up to 1M elements. Deletion of elements is allowed. */ +#define list_for_each_entry(pos, head, member) \ + for (struct bpf_iter_num ___it __attribute__((aligned(8), \ + cleanup(bpf_iter_num_destroy))), \ + * ___tmp = ( \ + bpf_iter_num_new(&___it, 0, (1000000)), \ + pos = list_entry_safe((head)->first, \ + typeof(*(pos)), member), \ + (void)bpf_iter_num_destroy, (void *)0); \ + bpf_iter_num_next(&___it) && pos && \ + ({ ___tmp = (void *)pos->member.next; 1; }); \ + pos = list_entry_safe((void __arena *)___tmp, typeof(*(pos)), member)) + +static inline void list_add_head(arena_list_node_t *n, arena_list_head_t *h) +{ + arena_list_node_t *first = h->first, * __arena *tmp; + + cast_user(first); + cast_kern(n); + WRITE_ONCE(n->next, first); + cast_kern(first); + if (first) { + tmp = &n->next; + cast_user(tmp); + WRITE_ONCE(first->pprev, tmp); + } + cast_user(n); + WRITE_ONCE(h->first, n); + + tmp = &h->first; + cast_user(tmp); + cast_kern(n); + WRITE_ONCE(n->pprev, tmp); +} + +static inline void __list_del(arena_list_node_t *n) +{ + arena_list_node_t *next = n->next, *tmp; + arena_list_node_t * __arena *pprev = n->pprev; + + cast_user(next); + cast_kern(pprev); + tmp = *pprev; + cast_kern(tmp); + WRITE_ONCE(tmp, next); + if (next) { + cast_user(pprev); + cast_kern(next); + WRITE_ONCE(next->pprev, pprev); + } +} + +#define POISON_POINTER_DELTA 0 + +#define LIST_POISON1 ((void __arena *) 0x100 + POISON_POINTER_DELTA) +#define LIST_POISON2 ((void __arena *) 0x122 + POISON_POINTER_DELTA) + +static inline void list_del(arena_list_node_t *n) +{ + __list_del(n); + n->next = LIST_POISON1; + n->pprev = LIST_POISON2; +} diff --git a/tools/testing/selftests/bpf/prog_tests/arena_list.c b/tools/testing/selftests/bpf/prog_tests/arena_list.c new file mode 100644 index 000000000000..e61886debab1 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/arena_list.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include + +#define PAGE_SIZE 4096 + +#include "bpf_arena_list.h" +#include "arena_list.skel.h" + +struct elem { + struct arena_list_node node; + __u64 value; +}; + +static int list_sum(struct arena_list_head *head) +{ + struct elem __arena *n; + int sum = 0; + + list_for_each_entry(n, head, node) + sum += n->value; + return sum; +} + +static void test_arena_list_add_del(int cnt) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct arena_list *skel; + int expected_sum = (u64)cnt * (cnt - 1) / 2; + int ret, sum; + + skel = arena_list__open_and_load(); + if (!ASSERT_OK_PTR(skel, "arena_list__open_and_load")) + return; + + skel->bss->cnt = cnt; + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_add), &opts); + ASSERT_OK(ret, "ret_add"); + ASSERT_OK(opts.retval, "retval"); + if (skel->bss->skip) { + printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__); + test__skip(); + goto out; + } + sum = list_sum(skel->bss->list_head); + ASSERT_EQ(sum, expected_sum, "sum of elems"); + ASSERT_EQ(skel->arena->arena_sum, expected_sum, "__arena sum of elems"); + ASSERT_EQ(skel->arena->test_val, cnt + 1, "num of elems"); + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_del), &opts); + ASSERT_OK(ret, "ret_del"); + sum = list_sum(skel->bss->list_head); + ASSERT_EQ(sum, 0, "sum of list elems after del"); + ASSERT_EQ(skel->bss->list_sum, expected_sum, "sum of list elems computed by prog"); + ASSERT_EQ(skel->arena->arena_sum, expected_sum, "__arena sum of elems"); +out: + arena_list__destroy(skel); +} + +void test_arena_list(void) +{ + if (test__start_subtest("arena_list_1")) + test_arena_list_add_del(1); + if (test__start_subtest("arena_list_1000")) + test_arena_list_add_del(1000); +} diff --git a/tools/testing/selftests/bpf/progs/arena_list.c b/tools/testing/selftests/bpf/progs/arena_list.c new file mode 100644 index 000000000000..04ebcdd98f10 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_list.c @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include "bpf_experimental.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 1000); /* number of pages */ + __ulong(map_extra, 2ull << 44); /* start of mmap() region */ +} arena SEC(".maps"); + +#include "bpf_arena_alloc.h" +#include "bpf_arena_list.h" + +struct elem { + struct arena_list_node node; + __u64 value; +}; + +struct arena_list_head __arena *list_head; +int list_sum; +int cnt; +bool skip = false; + +long __arena arena_sum; +int __arena test_val = 1; +struct arena_list_head __arena global_head; + +SEC("syscall") +int arena_list_add(void *ctx) +{ +#ifdef __BPF_FEATURE_ARENA_CAST + __u64 i; + + list_head = &global_head; + + bpf_for(i, 0, cnt) { + struct elem __arena *n = bpf_alloc(sizeof(*n)); + + test_val++; + n->value = i; + arena_sum += i; + list_add_head(&n->node, list_head); + } +#else + skip = true; +#endif + return 0; +} + +SEC("syscall") +int arena_list_del(void *ctx) +{ +#ifdef __BPF_FEATURE_ARENA_CAST + struct elem __arena *n; + int sum = 0; + + arena_sum = 0; + list_for_each_entry(n, list_head, node) { + sum += n->value; + arena_sum += n->value; + list_del(&n->node); + bpf_free(n); + } + list_sum = sum; +#else + skip = true; +#endif + return 0; +} + +char _license[] SEC("license") = "GPL"; From patchwork Fri Feb 9 04:06:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C34FC48297 for ; Fri, 9 Feb 2024 04:07:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B13EB8D0010; Thu, 8 Feb 2024 23:07:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AC3838D0001; Thu, 8 Feb 2024 23:07:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93CED8D0010; Thu, 8 Feb 2024 23:07:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7FDA18D0001 for ; Thu, 8 Feb 2024 23:07:36 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 66945A0DAB for ; Fri, 9 Feb 2024 04:07:36 +0000 (UTC) X-FDA: 81770931312.21.F04CE3F Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf08.hostedemail.com (Postfix) with ESMTP id 7A889160002 for ; Fri, 9 Feb 2024 04:07:34 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ko2v8iBr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ax1i6Xx3KClBMGyRsPQIF9NlJODAQM+auENr+G97oYA=; b=yUVdBfuVPyalWL3vRflBMG1Jg5cseEckxzHESmUWcds9W8kU1++0mlgQigGJn+eJKPAzk/ Q2m67BZS2j01DRoRKDu7zoBlECoUu8KY1DB7T5HbIZCdzmcYaLyLLeMbajDH0/2fnMZCuY HhXccTTDvRoGKVzJuiCFYfPVajorvTo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ko2v8iBr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451654; a=rsa-sha256; cv=none; b=kNLpRlvYtNdzK43UoqWGauP60Ws/olHUIp5GWhibfjrHcUdqZEDBZwedTzfB2FtwBm7CzE yM8/jJA+88Z/BxBNHUn4UCctvvKHyCJlhFGSGa/FpS0SCPPsRJSDXt23y/RCdldmTYYnap X5kPK4hwzHam6z1GDR1ZjkYl6kuEDp4= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1d74045c463so4471685ad.3 for ; Thu, 08 Feb 2024 20:07:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451653; x=1708056453; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ax1i6Xx3KClBMGyRsPQIF9NlJODAQM+auENr+G97oYA=; b=Ko2v8iBrEVfv+eXSI2BRv2liTQhh60VvJJznoDUcqZUQxrnT1HLWivE3zRa02HNRqB OVRZXk1hvyvIw5QdIu8caJe5lqX3yUwBdK7BKnQ5PxLnMWdQafo0VM/NY+vcCWNjV4WM ntqp8EHxAnupTK12wnsSgkFa3nW461xjLCzFk6nKxpdPZiT4zG0nt5FBmAGftYs2yTqg nKb+ktWtRIzOF2xiiqWV7ATj+PaleqQ93F05i/LUXGPjugCgCdtRsonxT2KXRMUEzfn3 BBkmf5zpjdrq0VvkdFtQKw8kuREZQZrrJG1OLbrrtr3cmuVNioBDHr6yitRw4kHAztUJ Jj6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451653; x=1708056453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ax1i6Xx3KClBMGyRsPQIF9NlJODAQM+auENr+G97oYA=; b=w3NIuC5P7924B6i1zoxDaXnUK1I52wasFU8+yK2Zf8kGfEymvzjrZjXWZ18UPOT9L0 YL5uza13oFwtut3BBu4b+6Q73YhyYm5e0kAcz7TB4kICF0ZMdVYV9z+dixCqV4FodbBd X6CcAkdFRl9yG4rrfel6xo8TIYDdJoxUuiiRZvJmr8BS0vzyD7gKoSPCEGT7OgdOsClO uZUXIqdNJLQCGP49GEHgIQpAggwCl88vO1HadAcBSUMGmy7E1WTQxEqLk6u0sJRiFhMS 3X4A8WVzrM3yFQ46Q1uaINi2I37lwSZO+JuAJRox4RrRVN03ix6JyIdMfUha5Dv0VcNR rk3w== X-Forwarded-Encrypted: i=1; AJvYcCUCuvCnxzwiKDDjCvhn9Arm+4B+KbUgYcTCLdVkU2Z8hnMhOjKCrhYE7tR9j0M8z/uAzDyrX6SbSd3hqaSCMzEgJbI= X-Gm-Message-State: AOJu0YwfzcJ/n2O34oBUGjP6Yt0/YYOktuBZHj4hAr+ClQmhi8FuIO8I Rtg5MC4kbsvyqVHj09R2NVUu3OZ6Tdg99INv/XLVN7JkRI2oQZUFg/3gYWAp X-Google-Smtp-Source: AGHT+IEGpYswlstUcGLeCwP8YO/u0UAf0+3x8WGh941h8RI1JhGiemicOLthwo5sUBqPjsP/3SF5Yw== X-Received: by 2002:a17:902:ec86:b0:1d8:b798:dfe3 with SMTP id x6-20020a170902ec8600b001d8b798dfe3mr539249plg.43.1707451653406; Thu, 08 Feb 2024 20:07:33 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVs1Kr+GkxG5gGSe9IAqpYlusa71Kdv6s5vkhdFG/fLIWePlY2qWKxLMvnRUrNz4H/DAJGLF26mTDtKKjD8WWDBIctDQZzbsST9F4Ln+bPNO0wK+o8/lgyIJDSDX9j0UVSTG00H0rEVqkxYJfTYelhcv9T3cgo6Q3OyKabdF+pCiDx0TAFrFONzZioUBqDHvXhS2oIgVjsslE6CRoZx9oQ36cxG82cPnMKVzWA3lxyVdGIsESj7DqQMgkA+fPZOW4nam9SwLlwJAN1KWtzKdp0P8+g79UkteFT1wAcUGkp8s/fBeG9eZbxbKgPEzBihaEPIHFM4epKG+jmwpn4Na/ARvgwu54fm4mUdB5o9tfeL1v7IVfQwXQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id q13-20020a170902a3cd00b001d963d963aasm561477plb.308.2024.02.08.20.07.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:33 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 19/20] selftests/bpf: Add bpf_arena_htab test. Date: Thu, 8 Feb 2024 20:06:07 -0800 Message-Id: <20240209040608.98927-20-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7A889160002 X-Stat-Signature: qtzdp584bc19z3iypyb9iwdcofe693k9 X-Rspam-User: X-HE-Tag: 1707451654-620383 X-HE-Meta: U2FsdGVkX18q5/4IBMlX8zA0vgjrn/dlwgU9jo4gcYjoM2fAFvXL0Ag/T9p3UqatMv3KtjNkqCWfjZYvP/3+RFIKYObUkdO2r/QlslAMUPJ7YjoIKZzl2m0P7WrPaKin8/S3AORoQLJdPBHqISBrlWLmrIUyl9g0UE0/+EYct59dU1Koiu9dCSH5BlOXORWY7S0aeEDRv8MGru8md6YC7Oh+ei+TNzfcOisalpcBWZRwZu4nVlWRo7PUJWpxC/5qXpRTdU12KeoubacAK1csSgKSA09VsiUFB3ctcIsn4UHiwTda2PA5vEjxO1y/fGgVlDt2Ij+YWEAE6LldK2Wz9qGsCaTcwCrPzYfa1m088f00hHZnSfgBytuSaD+ziwGJO8AFJkUbEGY3Zha0Hcun5e7rnXZhBKcU5ToTOBPx70D71RRIdyiX5DwEM+TeEL/k3rvuReQx2YI4pRHaCd7YsWHbEfl2cf2BCsTKuFu4D0cfc8VabhUbVe608NSzTZcEYCSrEbfpJf/e0FKigT49lC+G+wX6FmdK7cNdgkLbF1JrdHIbDnFIrNSF78sCPfhOC8TwK6/mefiSXXzdSYKG6BtuB9yzYjFwgxOiqG35EG4uL6SLUM3GlP3wxVS9Ie6Qz7DEncR4A8fsOhit8eE5toW+WSCY7bF6MDv+SR/CeT4IXU/0qntAwrPGbrJNA1cO1a7EDjK7Ko3Z+38k8IiRP1Q+sTU3dcPFsedN167Lvbxqo253gqQCRVYFUJelGbbisj8TGBCCbGVmdfzmxmuI66EF5aENhrouBWVnv2s2jKsj+nSN9KPVCh6Q7GVhVxrYrx+4qwudg5TxwXhfHpYcTDm26K0EaJ6mqGqOabyIcsY7VurKNI1TMqVDEOIIf84yc4lZld2d57CbxL/O9NnLVCcTNNTCU8frMcBQdpzfrQEJcFDnGjFTP1htxvdXtD+09tgz88IBhgnWyiAMuUM S3CHs/R2 WcV81O5ZzAy31ddrSvYGoAMGd9/oJ9/GZnyOpvnG0RosNGEQ3etXPI8+0oG19dObTx99e8CyttDrv0nYYohDL6RbPHc+41z4UIwc8/Dg8uugDMAIqWUU4oUEDMdfu2vTLReCPokKN4f6vKw2blyR4qklaBeasxsSCkvy6Yp7idjA8HgyCPctcFFkCuuefd2zQyeXirHI3daCa69zSQrB1DXI3T4IlS4OG2EzyO0V5/cSr9CULBm1wt6ZjEcbFT3KvA8OCdducIRIUZHMZTNg3HhE+umW8EFIOj3eZCEKQQZK6mI8ntGpe+FQJwt3jJULiMUomLWdmxFZKjF+NAJ0VIM75hSS6M2fgXTo2UwXrycXDapoPYsvi8Fp9lqysu6pcuSJhnceJzH2vokhTMlqAUiEl9BcZ4X7Oo96yrwRhwuwMhw1na3CbTWmaKnctLG/L13DuoF6mch7wVEzP7lwlGuAl0siUofeQjC2DM6afqRdhJ1Fj98st/vwF/xdREqkupQ8wGrNyD4s8DVqJ6Hn0PbPXJENjFhnVj6zFpPrD+yC3Z5uBjpJRS56uxYaZ7Sce4f5u0qEEx/6pknRV9MHWYK9Y6YC5wOnBkD37 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov bpf_arena_htab.h - hash table implemented as bpf program Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/bpf_arena_htab.h | 100 ++++++++++++++++++ .../selftests/bpf/prog_tests/arena_htab.c | 88 +++++++++++++++ .../testing/selftests/bpf/progs/arena_htab.c | 46 ++++++++ .../selftests/bpf/progs/arena_htab_asm.c | 5 + 4 files changed, 239 insertions(+) create mode 100644 tools/testing/selftests/bpf/bpf_arena_htab.h create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_htab.c create mode 100644 tools/testing/selftests/bpf/progs/arena_htab.c create mode 100644 tools/testing/selftests/bpf/progs/arena_htab_asm.c diff --git a/tools/testing/selftests/bpf/bpf_arena_htab.h b/tools/testing/selftests/bpf/bpf_arena_htab.h new file mode 100644 index 000000000000..acc01a876668 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_htab.h @@ -0,0 +1,100 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once +#include +#include "bpf_arena_alloc.h" +#include "bpf_arena_list.h" + +struct htab_bucket { + struct arena_list_head head; +}; +typedef struct htab_bucket __arena htab_bucket_t; + +struct htab { + htab_bucket_t *buckets; + int n_buckets; +}; +typedef struct htab __arena htab_t; + +static inline htab_bucket_t *__select_bucket(htab_t *htab, __u32 hash) +{ + htab_bucket_t *b = htab->buckets; + + cast_kern(b); + return &b[hash & (htab->n_buckets - 1)]; +} + +static inline arena_list_head_t *select_bucket(htab_t *htab, __u32 hash) +{ + return &__select_bucket(htab, hash)->head; +} + +struct hashtab_elem { + int hash; + int key; + int value; + struct arena_list_node hash_node; +}; +typedef struct hashtab_elem __arena hashtab_elem_t; + +static hashtab_elem_t *lookup_elem_raw(arena_list_head_t *head, __u32 hash, int key) +{ + hashtab_elem_t *l; + + list_for_each_entry(l, head, hash_node) + if (l->hash == hash && l->key == key) + return l; + + return NULL; +} + +static int htab_hash(int key) +{ + return key; +} + +__weak int htab_lookup_elem(htab_t *htab __arg_arena, int key) +{ + hashtab_elem_t *l_old; + arena_list_head_t *head; + + cast_kern(htab); + head = select_bucket(htab, key); + l_old = lookup_elem_raw(head, htab_hash(key), key); + if (l_old) + return l_old->value; + return 0; +} + +__weak int htab_update_elem(htab_t *htab __arg_arena, int key, int value) +{ + hashtab_elem_t *l_new = NULL, *l_old; + arena_list_head_t *head; + + cast_kern(htab); + head = select_bucket(htab, key); + l_old = lookup_elem_raw(head, htab_hash(key), key); + + l_new = bpf_alloc(sizeof(*l_new)); + if (!l_new) + return -ENOMEM; + l_new->key = key; + l_new->hash = htab_hash(key); + l_new->value = value; + + list_add_head(&l_new->hash_node, head); + if (l_old) { + list_del(&l_old->hash_node); + bpf_free(l_old); + } + return 0; +} + +void htab_init(htab_t *htab) +{ + void __arena *buckets = bpf_arena_alloc_pages(&arena, NULL, 2, NUMA_NO_NODE, 0); + + cast_user(buckets); + htab->buckets = buckets; + htab->n_buckets = 2 * PAGE_SIZE / sizeof(struct htab_bucket); +} diff --git a/tools/testing/selftests/bpf/prog_tests/arena_htab.c b/tools/testing/selftests/bpf/prog_tests/arena_htab.c new file mode 100644 index 000000000000..0766702de846 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/arena_htab.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include + +#include "arena_htab_asm.skel.h" +#include "arena_htab.skel.h" + +#define PAGE_SIZE 4096 + +#include "bpf_arena_htab.h" + +static void test_arena_htab_common(struct htab *htab) +{ + int i; + + printf("htab %p buckets %p n_buckets %d\n", htab, htab->buckets, htab->n_buckets); + ASSERT_OK_PTR(htab->buckets, "htab->buckets shouldn't be NULL"); + for (i = 0; htab->buckets && i < 16; i += 4) { + /* + * Walk htab buckets and link lists since all pointers are correct, + * though they were written by bpf program. + */ + int val = htab_lookup_elem(htab, i); + + ASSERT_EQ(i, val, "key == value"); + } +} + +static void test_arena_htab_llvm(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct arena_htab *skel; + struct htab *htab; + size_t arena_sz; + void *area; + int ret; + + skel = arena_htab__open_and_load(); + if (!ASSERT_OK_PTR(skel, "arena_htab__open_and_load")) + return; + + area = bpf_map__initial_value(skel->maps.arena, &arena_sz); + /* fault-in a page with pgoff == 0 as sanity check */ + *(volatile int *)area = 0x55aa; + + /* bpf prog will allocate more pages */ + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_htab_llvm), &opts); + ASSERT_OK(ret, "ret"); + ASSERT_OK(opts.retval, "retval"); + if (skel->bss->skip) { + printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__); + test__skip(); + goto out; + } + htab = skel->bss->htab_for_user; + test_arena_htab_common(htab); +out: + arena_htab__destroy(skel); +} + +static void test_arena_htab_asm(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct arena_htab_asm *skel; + struct htab *htab; + int ret; + + skel = arena_htab_asm__open_and_load(); + if (!ASSERT_OK_PTR(skel, "arena_htab_asm__open_and_load")) + return; + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_htab_asm), &opts); + ASSERT_OK(ret, "ret"); + ASSERT_OK(opts.retval, "retval"); + htab = skel->bss->htab_for_user; + test_arena_htab_common(htab); + arena_htab_asm__destroy(skel); +} + +void test_arena_htab(void) +{ + if (test__start_subtest("arena_htab_llvm")) + test_arena_htab_llvm(); + if (test__start_subtest("arena_htab_asm")) + test_arena_htab_asm(); +} diff --git a/tools/testing/selftests/bpf/progs/arena_htab.c b/tools/testing/selftests/bpf/progs/arena_htab.c new file mode 100644 index 000000000000..441fc502312f --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_htab.c @@ -0,0 +1,46 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include "bpf_experimental.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 100); /* number of pages */ +} arena SEC(".maps"); + +#include "bpf_arena_htab.h" + +void __arena *htab_for_user; +bool skip = false; + +SEC("syscall") +int arena_htab_llvm(void *ctx) +{ +#if defined(__BPF_FEATURE_ARENA_CAST) || defined(BPF_ARENA_FORCE_ASM) + struct htab __arena *htab; + __u64 i; + + htab = bpf_alloc(sizeof(*htab)); + cast_kern(htab); + htab_init(htab); + + /* first run. No old elems in the table */ + bpf_for(i, 0, 1000) + htab_update_elem(htab, i, i); + + /* should replace all elems with new ones */ + bpf_for(i, 0, 1000) + htab_update_elem(htab, i, i); + cast_user(htab); + htab_for_user = htab; +#else + skip = true; +#endif + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/arena_htab_asm.c b/tools/testing/selftests/bpf/progs/arena_htab_asm.c new file mode 100644 index 000000000000..6cd70ea12f0d --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_htab_asm.c @@ -0,0 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#define BPF_ARENA_FORCE_ASM +#define arena_htab_llvm arena_htab_asm +#include "arena_htab.c" From patchwork Fri Feb 9 04:06:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13550861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 826A2C48297 for ; Fri, 9 Feb 2024 04:07:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6FA78D0011; Thu, 8 Feb 2024 23:07:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1F728D0001; Thu, 8 Feb 2024 23:07:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC0518D0011; Thu, 8 Feb 2024 23:07:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B6E458D0001 for ; Thu, 8 Feb 2024 23:07:40 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 943484019E for ; Fri, 9 Feb 2024 04:07:40 +0000 (UTC) X-FDA: 81770931480.02.CC9B5C1 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf09.hostedemail.com (Postfix) with ESMTP id BF2DB140032 for ; Fri, 9 Feb 2024 04:07:38 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xux0hNhz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707451658; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6ll3cFrEr9hbCQcocHDOoHYlUKQul/zqkLdXGSoHE9U=; b=XR+Ex508yDnLk8ZIWngzWeNS18a3PPFZQcvNUU6kamxdaP6GF5UwXxUAfn7l6CgOemh72Q OFn+wTXdFwm+j5EQzXhSs14YnRWAU6hdV1GX+DA9qp7H79ELMup0gbEahf4pGn0ZkhN/Pk KB6+29Ho/tfD4VYFx8Nck1JqM/i+hQI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Xux0hNhz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf09.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707451658; a=rsa-sha256; cv=none; b=u5Z8Qya06u0VgQZZ7luPXCeWYB2WaW4NmQIxMRqAbgH5Q44bFIK8MLAuijdk43xMLV+bj6 XT3sR9WFL0mto+0wVjzF9ldYF9+Q0v7htumlQyw46ExxQ5tSNuFzIlYEpmS1I+GInJ9DgK KosSSNKbAZZYpLKBW/dPiumCwqdIUSE= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1d780a392fdso3994745ad.3 for ; Thu, 08 Feb 2024 20:07:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707451657; x=1708056457; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6ll3cFrEr9hbCQcocHDOoHYlUKQul/zqkLdXGSoHE9U=; b=Xux0hNhz8ST1kaggi9hr/yJcdfFWyvpsWP0Q2A0dqtHGxznY9srlcBXCdqcAiBryEj In4Evw9wOeun9vL7fir2v4VxCqAFZiEKDdSwmxRlzqMwp7CVDk78NXmhjIXzMuWjyU8E k6dodMDdvQ2uE27ySKvVTJBbDbgWaHi/3vU9B3buviDV2++AeTMTEFISxgIL+bcMqm3M C2sS9QDMoFkKITd+4SuJX+SR74n14AQlX50KwL1NqYXox0/q9Th5SZ1Ma+cKAea80YUM 78pkmsQ0rKEmpAKZcZhmXQ7qdB9T4JRi6Jqw1mjddQ9aFTZpdfiY27pycNtKkQv3nQVa qbjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707451657; x=1708056457; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6ll3cFrEr9hbCQcocHDOoHYlUKQul/zqkLdXGSoHE9U=; b=AlIDGGCWBlL3vZMGPJ0YHwwR4NdOcbJMeuqd3CkRuUeciqZp9NlNAuiipYKURIvej2 SEyMCsP548BdqhDUWjQiL4EDxyDaOjWy/FqINriPzbRS7hts3ymLrkeyFVkyhbpR020I k9uM+oxuGzftxkH0x9B0hNfmzWwLjVpK+OCHqOb1FGKxGH/CqSZFaEzusm+hSdHji0sh MrohSXUkfqi8+CXkLrXHRyQt92CEPidL/8MMReHVU6EodpPzvYw68anVjctXr/8UJIk6 Bylnl389yUHDW0xB2SvfHdQLX9SOLAaR8YQBabrcjNGDj3WTVFTHG1jqk9odBYDEb57y bskA== X-Gm-Message-State: AOJu0Yyb8q897kwFNJoBdzHBZ1GH970eRgnqCASBqXgprMbEqRZGqWsO dZCa/A8xMPTqCn0xm+cH9GCwEk8S7zIyprXSyVLGug4adr8bb5DK X-Google-Smtp-Source: AGHT+IGkOiYtxyH9h4Hxmx0HmJv/rugkioUJ3yihq6NNOSDK+ZXse18c8Q3zRElKlWdCYTq+E6Xxsw== X-Received: by 2002:a17:903:2a90:b0:1d6:f185:f13b with SMTP id lv16-20020a1709032a9000b001d6f185f13bmr618327plb.17.1707451657685; Thu, 08 Feb 2024 20:07:37 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXDiS092+UkDyF4aEZPoblkWtOoQh1U8TGTdcCSduROIhOu+T04/eDIqGY6NzkwsOOzzZi0ztZBRPjETZoAFTsBxT2B3gdn7EYY0CrHwnxfASTjykvyk4yUPd99WN1zKGqA6dczojNPrJ0u4cqZeHkVclQfcV0afn+ij1NLT26mTYwX8tDsBlFcO3vI1tfpczHFjzQ5zEfAj3sIJaM7Ybtr2QS+kHiCwRn1K+R9vmundeWwOcBO4F+N9D7ITsVvn4Z59MKMI8lC9bdSbwy4VqUhhg30uZJOtPtrVLHLhwPbV64heLkaKzgW5Tk+DG2lHuTnYn9hyuJbcl0Hv6xLbiFvHOEwp1ebFpnSWQc32i3o5DeLAiU5HQ== Received: from macbook-pro-49.dhcp.thefacebook.com ([2620:10d:c090:400::4:a894]) by smtp.gmail.com with ESMTPSA id l4-20020a170902d04400b001d9fcd344afsm541162pll.222.2024.02.08.20.07.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 08 Feb 2024 20:07:37 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, lstoakes@gmail.com, akpm@linux-foundation.org, urezki@gmail.com, hch@infradead.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH v2 bpf-next 20/20] selftests/bpf: Convert simple page_frag allocator to per-cpu. Date: Thu, 8 Feb 2024 20:06:08 -0800 Message-Id: <20240209040608.98927-21-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240209040608.98927-1-alexei.starovoitov@gmail.com> References: <20240209040608.98927-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BF2DB140032 X-Stat-Signature: enu55b9cdgokpugbmqjjofgno8ay11dj X-Rspam-User: X-HE-Tag: 1707451658-72902 X-HE-Meta: U2FsdGVkX19nGH3Vk4ulw44LYEjx+FH3GK1+Cgj6TrvG0p36IoSuE7YrEsJyQ9i2NbbawmMJFBQwO+vj6U6X3HbcKxjle9bQK0bfPLLHaNjCjHJ6dOgB46WMHWNg+gSYUzO2DD/id8LoCtlXtwB7x1E2dZX1afgbNJiZdKaAcwY8BmcqvkftYe29IOnuM8WBDjCHcth3ijttzzAwPCUTw3I5dEBiX58xo76vpD00NzN8jE6fTkYf8CZgGjLT94D8SgT3qrUmtQFmO1n0LBBPcOXxpZV5syGdWbEWOer9d3Ye/AEkzUururxzF4ISQeOkJtKF/Uu1fQ/bdUTP6+ZDHdO2VZH/4aO8EOMwFlL3XP3vq2HnWgnX8qtHNTJvDo/n7QM0h5vl4nH8O5YjJTOGrrLB1v7CemYKDNA/dsdNgd5CQIEke6LvU7XsFAEyEl64jCuYEf/7tj+hVz7x9z2q1MS7OPFtZZ7UgapPurizFDemcng5zJ5s1emhmV7aGh7C1sooN6VdLy0Tn66k0fMC9/ZZXY/eeXMI5Csrk9eePGijMRXHYcxFxna+NV8nnAQIsHjdI0U9ofpTHglJa0T3SsxHbXCNuoh0T1wucespfofgWHcGydkEMwaG0iSqWWxVngBin+GF44Qt6DxtH2rnMXL4uUaH2pJzQadtRIL/5AhH/a4+gZktUAcNF2GYecEwU5M0UBSKAQ/zr9u7PqYVXSopIwxkRChwplcK2J7ofPDnwxzjoeK0JXzy2/Wmizu/vjj7L+ifRPrwb8xC5DV10JTvPtL8kSvfnhxjIZbxCKxP5dVkJ8N1+yBAw4Gck6ZLvENZes6MvmI7ENim27mmwWHCXvG7dQZGINNfUKXeMcYqEWuEhO2pQbl+HTW7KuJB7IzW5h8l+tWiSseZ5kQ816nVkkZNq0XsFZHz9EtSrSLF4XEaAtO4tsIf3j8MV8KOeD6EvfdQxetNelMh7cP 7ODUMkB6 7seJ72tRLg0mAqKSveT7ptxW2vAGLb7UEQvMSzybOC/UtQ/Daj5u3iLiccIr/5PhPWRK429NxeRVR+vY/TJPk9vFP22C6nfsY0bhXQqjrRdXJOkOnsnBE2dLARcI0aV1YcKVbs+HoO2iJJ7hDJMzaKr7F3LkhFORkHiWrO/UI2MP05xYK+wvrNytA8zBLOtwt45kYhmcPbMVab3CbrG20hGV04fKYyETjM5wCIOS8AlMOYXT3/+lMYVRb3m+cKCCi3S20MfUUcTkSbBHDrt2Q0P18CsknJ3TiA2z5Oc4vdIdQKqg5VEyLSFK7bs2yj1e2M4VOwUX4dUlww0/4bHAwTeUF3jQOdKaJyj71cubQJi1DNldYRLQK3e6KhimR91+KsPCPxHcb01thGGy+ASOw6LsNL8HSz4kdpLv9KcjUQw+wFFtA2hn1VhtpiasARYaw+uDpLwnSyaWzvrhf/dscs5iAM5pK1xtJXv4yf1hrIIVgDJqe5IjqFSO5VrhH+mGb7Inl1Mh2hCQM+zIqY5jT10VlMsJNvzbHxEO/mU+JFylQYbJWwN7k3Lcijlboy7iL0RTPmUkDoOuQvQMmP+0L6OZFVbn5yuFGE8CK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Convert simple page_frag allocator to per-cpu page_frag to further stress test a combination of __arena global and static variables and alloc/free from arena. Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/bpf_arena_alloc.h | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_arena_alloc.h b/tools/testing/selftests/bpf/bpf_arena_alloc.h index 0f4cb399b4c7..c27678299e0c 100644 --- a/tools/testing/selftests/bpf/bpf_arena_alloc.h +++ b/tools/testing/selftests/bpf/bpf_arena_alloc.h @@ -10,14 +10,19 @@ #define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1) #endif -void __arena *cur_page; -int cur_offset; +#ifdef __BPF__ +#define NR_CPUS (sizeof(struct cpumask) * 8) + +static void __arena * __arena page_frag_cur_page[NR_CPUS]; +static int __arena page_frag_cur_offset[NR_CPUS]; /* Simple page_frag allocator */ static inline void __arena* bpf_alloc(unsigned int size) { __u64 __arena *obj_cnt; - void __arena *page = cur_page; + __u32 cpu = bpf_get_smp_processor_id(); + void __arena *page = page_frag_cur_page[cpu]; + int __arena *cur_offset = &page_frag_cur_offset[cpu]; int offset; size = round_up(size, 8); @@ -29,8 +34,8 @@ static inline void __arena* bpf_alloc(unsigned int size) if (!page) return NULL; cast_kern(page); - cur_page = page; - cur_offset = PAGE_SIZE - 8; + page_frag_cur_page[cpu] = page; + *cur_offset = PAGE_SIZE - 8; obj_cnt = page + PAGE_SIZE - 8; *obj_cnt = 0; } else { @@ -38,12 +43,12 @@ static inline void __arena* bpf_alloc(unsigned int size) obj_cnt = page + PAGE_SIZE - 8; } - offset = cur_offset - size; + offset = *cur_offset - size; if (offset < 0) goto refill; (*obj_cnt)++; - cur_offset = offset; + *cur_offset = offset; return page + offset; } @@ -56,3 +61,7 @@ static inline void bpf_free(void __arena *addr) if (--(*obj_cnt) == 0) bpf_arena_free_pages(&arena, addr, 1); } +#else +static inline void __arena* bpf_alloc(unsigned int size) { return NULL; } +static inline void bpf_free(void __arena *addr) {} +#endif