From patchwork Tue Feb 6 22:04:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69D9AC48297 for ; Tue, 6 Feb 2024 22:04:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6C8B6B0081; Tue, 6 Feb 2024 17:04:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DF5B76B0082; Tue, 6 Feb 2024 17:04:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C98876B0083; Tue, 6 Feb 2024 17:04:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B4D3C6B0081 for ; Tue, 6 Feb 2024 17:04:52 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7B65D1406FA for ; Tue, 6 Feb 2024 22:04:52 +0000 (UTC) X-FDA: 81762759624.09.77DF762 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf19.hostedemail.com (Postfix) with ESMTP id C7FE21A0015 for ; Tue, 6 Feb 2024 22:04:50 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="lK/B2/00"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vTdQC2aUXQJJJCr+aCpdI86HWrK6Y2hTgBNTwS7bp1E=; b=K+fEv0DPC5gDt4xWzGYPJaDst7VcBk5gcPUdgZSArkXPRNv2sfFkP5E7x0zP4rWnkPw7nR nrpjP2i99x0NIgqfu6LnHOvkLHzk+qWkUUZ+1vesiWyUzK4+jpGiCmalRpjaPwX1H+A7N0 q8wUvj5Yi887Nj2GMBmD4moxZjsOyK0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="lK/B2/00"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257090; a=rsa-sha256; cv=none; b=xWcpljE4ibxQhGVibRpQ7owBN/YSe9UDrW3Iz4Qp7BRTTHqISszMQIei5RrBFccGlwg4EA MOTvKis6sPzlN7GScPBGCBEpU5sImM3nAiCVaY54brfOw3WZZviOlCAzKI9PL46rK5O6m7 aiKVLyXTg4F95MlKlV2PeKac04ifjrI= Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-6da4a923b1bso5959b3a.2 for ; Tue, 06 Feb 2024 14:04:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257089; x=1707861889; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vTdQC2aUXQJJJCr+aCpdI86HWrK6Y2hTgBNTwS7bp1E=; b=lK/B2/00R925DPJt9xHF6XX5Ko9f2UeChXoyF4tXon1K5kea8OxT1sY0WZghiTXols woptO8cxpTHl1lx4RRDDZrAce24yF6hSyw26zp9lQoqfCLvwpI0dyzBi1dUNpZJmxdst Cd0PQvypJLWE58na2iRfuckgbW489Vaj6hGiCHcyGEXfKhvjzc68IUcwUhZGn4bUZvEx 2rgXbzjKQhsTee7bJFx4o+cIXGJP1fWUJm238pAu1ip1bx+XpZvAEuXxzZB3hMySCVUs 7YAMwHfLwlMKZqwq7/6t5aa74AWAxqsaohedoWp0CfFt3SyuV4tO7iw1BBwCwMfslAhC 7cZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257089; x=1707861889; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vTdQC2aUXQJJJCr+aCpdI86HWrK6Y2hTgBNTwS7bp1E=; b=jsfDw2JGX9e3T1zNnMsrZWqOCGv6IbuTma7SRBUJ2W1vZlwitkxIrd/24XDhTtuX7N hD8yFSqLovEBqqrIrMfNIVgIrfJMrdyC0183OE3adcelTjylKphXF1p/Fq/ZlzjxdnCP 5RWmefWb6OWuLtrZsufaP1Vn65aLShoh2Srrszg+bz9ONiOFCMxc5tKvKv4pX1pMgU3Y zUQYQ2ZnBlJzoO8/o0hyBSzok/hAXaUW/HckQf/jroe1MgiCsWfX2nrb9GAxfJW3iNPD 5LR7uni7rK79Zo7wFbW4S/gd0TgNLm9axEHi6RmVuay3i4pUUybPFvtwU8ts7vpk3xdI Ulsg== X-Gm-Message-State: AOJu0YzUAjrlg/pu6D7WEh31cGJIwr65E5otl0kza7BL345qGBY7IIZI TkgY1gYaa5Q7TPPPswjM6tAEDHwla9ilZ4gzKvIUzBktFJXZxkRR X-Google-Smtp-Source: AGHT+IEOBWDdHBSn6MYZcDsMAjg06YN4MswMauxkUU1AohdAC2pdYdWD/fKtCiGD2UT6AxSc8zBdng== X-Received: by 2002:a05:6a20:9687:b0:19b:56f0:c880 with SMTP id hp7-20020a056a20968700b0019b56f0c880mr3151794pzc.39.1707257089523; Tue, 06 Feb 2024 14:04:49 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWKAdt7d7meDWiqsPZBETrzTMdLFa3A8xeHxysTX4BnJCCUBehTYcyFmzK/mu5UhvwvCiGftMYGrDP47TFLDjE660wuY1T5GY0wrbXx4X1S03zWbWSXrOQ+tdSlnAu5YJXkDcgBFW2A3CDYdLKDKBnFqYTMtQA2EY6Myv9mLdg8OuDd+rL3sbhg2fdft6PrbNQoJEP1c/MrDfcTHtSmtmEH1NCKjDvOzx4UtVeD0UtYuEuc0aiQfVw/iVuyTncavF/YiNn57WBDn7q1VeXkmEMUcE3rFGnER8Ry Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id y192-20020a62cec9000000b006e02ce964b7sm2560051pfg.184.2024.02.06.14.04.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:04:49 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 01/16] bpf: Allow kfuncs return 'void *' Date: Tue, 6 Feb 2024 14:04:26 -0800 Message-Id: <20240206220441.38311-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C7FE21A0015 X-Stat-Signature: rmtgq8wxc1a5oadarqyganawn8gsmhdh X-Rspam-User: X-HE-Tag: 1707257090-670767 X-HE-Meta: U2FsdGVkX19ichblTy0A5SL/2oxMpuoHR4Un2538FOSKKvbDn989osdcW51ZAp22TFTyO1Q8w8HFjlPb5dmav7w5tao/okJDZyuRd+VUlSjTvYBtNUZTb2Wb7R9SIrb/XD5b/PqnYJB07xg0+21cJbq3P0zLx573V+mosTuH3Tsp5Gt9sgXz/VC0CO1l+p92L1H3AoUn4lxErtQ2qIHMuYnzz5VTIqSvF1iHkLl5mOG4IG6cKV55jT6rOf9wT+rUWeAOHpMp8rwLHSTK8Yk44nn3NwaIkgo9Q3xW0Q8BPXrl9vsTepvANYnHtKYMxFYh68c6CoYmJVieWqSYYLAtNnBG230ixFeYsxJIYgMOn2XQ9bjlVnQ/wD36wZStYl/GVGAT9SHzXxt5F+cGm1cYes8zIMiVS7TuNl2G3K76X9zoYeDWPcK5PXkmz73nBrxVHf5WisFg4yRgrbF6VYgAlPXsw5A1ZalG/thvND5Ullv9uTo2VqUgc1dE/FKep40PYeiNLe338iPynCyZtepHQO04VlMPYNPEOLlfLVlrWMYY4ymOG+CNbJ8WmExC9j6d2srdCQeL98ZOCHWOdltptXr7boQcjOC05dTeuSgiG83JgjbjARtl9yxaqfzn2rgjBcFK9qLj07PbeNdRd6DlXFIq4K4MZOaYSz7IPiMStJuGJ5/8uiPEbE3DEav0ZJAGHDeq4hKXPYI1kTs0TDNJGA+qrq2w0NiWqJnqERFIdEH47yZJiRVq38WzMrmRRyWLr0c70abYYXwOQzufd1H3QnJtCr5OsutZBnO1ohggCJpctVlRaNpn2VTNnEKK/q1mbGqW+OCNdFJYjaWFqE9ixWutVbw7SN5CbePW+x0krvqZr2yk2L0ne7G9y6gxE8zAB2zMf2mqita8L4DjFWuon6h3iOFTf67Wz2ocquHbMgoV80cZ1mkIsnIJkm0vreEoQJKiNrBcAK7eu9wIdX0 V/LVv3MP KVVymMQu+dvLDSGcfwATu3yAqtlT1pwtcL4a8YGINZCRUFCbOniatSNK/sM/5O16gfNkF/cwZlw8HTFATLushGMrkcF/sWZRFrAIp8lQiKaTMRPb8ZoucnHDSrWgtq6G6EAuBvxaVLLl+ShaVtvEIj4TxrQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000089, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Recognize return of 'void *' from kfunc as returning unknown scalar. Signed-off-by: Alexei Starovoitov Acked-by: Andrii Nakryiko Acked-by: David Vernet --- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ddaf09db1175..d9c2dbb3939f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12353,6 +12353,9 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, meta.func_name); return -EFAULT; } + } else if (btf_type_is_void(ptr_type)) { + /* kfunc returning 'void *' is equivalent to returning scalar */ + mark_reg_unknown(env, regs, BPF_REG_0); } else if (!__btf_type_is_struct(ptr_type)) { if (!meta.r0_size) { __u32 sz; From patchwork Tue Feb 6 22:04:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BFDAC48297 for ; Tue, 6 Feb 2024 22:04:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08B0A6B0083; Tue, 6 Feb 2024 17:04:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 03A706B0085; Tue, 6 Feb 2024 17:04:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA8FC6B0087; Tue, 6 Feb 2024 17:04:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BE2066B0083 for ; Tue, 6 Feb 2024 17:04:56 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 880921408B6 for ; Tue, 6 Feb 2024 22:04:56 +0000 (UTC) X-FDA: 81762759792.02.E89691C Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf13.hostedemail.com (Postfix) with ESMTP id 9E57320014 for ; Tue, 6 Feb 2024 22:04:54 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ik2rmXIE; spf=pass (imf13.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RYMIDAZLgBUgMpP99zrxeTSMtNJZxQwPNyQYFWZYzso=; b=LK9+W+4b3xe9fbC75kRHXxIUFpMtWIXIyU/q+EZWewm8tq3r3V/kflnVGKbHTQ8umMZ8if 9+zXX9Mj/mdGTXsW+n/VBt3JbZ7FAnsG7BLsgQjcGxqvvxjEVpUypewai2foj5fEcm9b5k ozMpF4np6wCUbPBJWiOlbIy1xhuCqis= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257094; a=rsa-sha256; cv=none; b=HID+eWJsfS8IlxYhEB6SApbS9Pq5MoY7JlmmbfLFO+ePcY9mpFwGzMmVdqVKcXx3MKSYbu F5EZbW48oLcVIsLhW6PRongaDDXiZFMAZRJmIiVQiYar1dduHWX+2cSwbn5N+yBiZmMUWg Dqrs/i5Wc5zwuSizGvpCRUGiRRPKkGA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ik2rmXIE; spf=pass (imf13.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6e055baec89so6021b3a.1 for ; Tue, 06 Feb 2024 14:04:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257093; x=1707861893; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RYMIDAZLgBUgMpP99zrxeTSMtNJZxQwPNyQYFWZYzso=; b=Ik2rmXIEXhxqT3/BUy6tkg+7XSHUYfM2MTYdZHRU/MgO1Kvnd2hUN90kc6aLS38Y89 8tIIdHDQ9FZj59chcVtFsqjVaeMt4yPKC0fpfDrctQaLom8lYnHPjnIUlekEV+/vqs53 /gg4Zl8oVtkzmvEmL4mvjlFnCOUHnLHXB+AjlX6a1111bJxzcK/cVMcfDjc1UuZUvcn2 1oMQD4ILmDoeFQHhDAtWbSoJJd4foK9YpVgHV/IohYBAmTAiCbeRKvLSlD+mMHsXdbLv Vhc6HcXskzCxgHne/POjYMGg7IbOByJY0gr1mjyaj306zlp8fHt4OjXAOw2hPE9Z+db7 e5vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257093; x=1707861893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RYMIDAZLgBUgMpP99zrxeTSMtNJZxQwPNyQYFWZYzso=; b=mc+QuFJIW5mi3M3aeX2bCmuaGnRS2veX8INWo00A5J4rOTFOBorQeUcmcCiaih7c8/ k9uya9SVxYVrVS3SsKbI5T88wR4hGRCS1D9lO9gGN+M6ii5QKkoRKWiKNzKAZsV74BGq s1EsqvZM8uoq1fSLNkwqw9oCK+sPO6pwPGy0ySktd7bdFFDASdeUOwC1EWCu5dJT6C6e m6OJvAxi1Jy4xEoR4GyKXYxtLHtdbls17UanmgVF0NmCJmdJ3R/qKfaxs+Qd14G4qeD2 lcY9h5N4bh+bau8EsnMSFx1AZuH0/fcPuSqRH72oZnXg0kmDLDwaqXaPHzUV+RAuB8DW c06A== X-Forwarded-Encrypted: i=1; AJvYcCX0NZ2tVvxWFjp3x6NH0fhAG56tIK1e6rkJUHdnw+2h8SEkzINPWVBMcRJW9UiIhY2xoDQI/8UyiTpLN3+LH2LS6ic= X-Gm-Message-State: AOJu0YwatRN0199h1h6kk5RrZxT0E7lWrOvvR56yvI8nScX7BOdIhyVq /lxl2JhnwbNL9mPKO/cjvzqDLAkH2VO32YFVf1UXa3FZUgCisX1q X-Google-Smtp-Source: AGHT+IGOIOdH45J5dyDgzanegb8JLdqXenulcK+qVHNvlZfYkUWkLxixF+6eKSWhF7qb3dfASt+aRw== X-Received: by 2002:a05:6a00:b93:b0:6e0:4f30:bcfc with SMTP id g19-20020a056a000b9300b006e04f30bcfcmr1405410pfj.9.1707257093567; Tue, 06 Feb 2024 14:04:53 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWUQoVkIthQQOsy6U4eSoQ5WdjVc0nJCVsDbkA2v0JGF6Yabw2lW6vfJzu68baFahsluKlv1NXHpG3TR+PKOZ23wdIUvI68+73W6afFYOGp3111MbSF540b1PZ6MPixqg3IhHR0uNOvN8cle0tWOioXnGsAI2TDADOi5Fif9LN8YHbGgy0QINttwPB4URjcnMqY8BB11Ij1d6XI/7mcn2HIpchSHafTgbAynmFH5kUNenEDFjaaPYgtauQDqNmqat3SfznSOSAwQYvn61xLPdSXcFtvHj6P1zvY Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id y20-20020aa78554000000b006e0322f072asm2488200pfn.35.2024.02.06.14.04.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:04:53 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 02/16] bpf: Recognize '__map' suffix in kfunc arguments Date: Tue, 6 Feb 2024 14:04:27 -0800 Message-Id: <20240206220441.38311-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9E57320014 X-Rspam-User: X-Stat-Signature: tuy4ht5uey6inxe9oupd5figgdbebnet X-Rspamd-Server: rspam03 X-HE-Tag: 1707257094-642241 X-HE-Meta: U2FsdGVkX185E1McaxdEg+ma/Hbzl2TNS/TCwzNhykouZmarElTYLBqcehCUNW5Ui7AGD/bXFG5dnM98vfFVEpA5MqlsqhntTPScJTVrOh4uLv/Q5n6jlh2/k+XtV6yPNXhwqTJywz4E05q5t73O+Ashnsul6dmILa9rtCIHcFcUg/hPv/jnRM+eZEnloggI46WuchNGtpy5ym5Ocddky/nRD7LTY4i24kDK21W0n6fD2fNGFDdv4Z6mxSj+DQ0bZLg4pC9wc+fvLg1gVY2LzZ0muQXjse5UBly7A3u7j6u0KbWj6S+jzwGafZ7VLU4az7jORneTaYAkz0hVAjD14YTDzBRzggZ83k9OIFMmVC+zNZ45+BmnHythI1pTTQNiC9gp86eJs++dcuPHQCay5jLk9kbjRo6ndjpGtM6Us3/+t7rDtFt4NgsU5r69geYF9vpWQ8++gXIcl0OJ3RBLy0wRgfyUKF59Bhc6lbQr19ptMoO3MpRppJI20HoFAEyXK9i0lczac8qzo5MQn0bIgrbTE0uRibCKsLNBzAFttV1zPAOzyGO8xwB5fNYZModWQslzfVSOEr4hsc7KgrmpRl3RxSVM65JIbvCz0OOBCO43tE5xTRagiVbQfM0E34NBEhnSG4nzdU+GrFTMRHXVzuTEDtgVwB5I2pvUwtks8eNvAGmFL0hAs9xO5JM47zI0fPYDsrydYUNa+aml0D3QyQdl/HpathIO/wJwlMEM5k3JRmmKmzZ6UpOHkut6z6+brYkzioaFfgnvqmlkTwY8JNO6JZVKF59rTJOkih3bSUNN/dZ0gpIoL6Xlg2Jh3GXLigh2eInqjibnXPFxjouAn+YnpQGao+UZnvPMFQw6QZPEP5QZ3G2BwekMV2xZvALlgKtMxdZ8jh/CNaCh70CQPa3Qh0Hjdc6IaUwIeCMFJU6PmsTkGLIoiUjFFOwKGuV/uhp3/4vxWO8QPp8KZ6N qiIS5Zus KNux7XIb7rLFbHvMe7SiC0xGjyTdfMpB/oxDjb1Qshcen8DweJnG9M5bvJ2xnamO3d+IdbGUVInrxY1mDV0j04vwAyzC/884zKcIF2a+Tn9R1z3Z1PyriSb8KKuxnCcJ0FXubsztvrj8vBuK0l4Lsp+fHYdz6qj7wElYaOShY4yUSRFy0mY3EIlELEblYzcAgwJQ1l8W1kW8I65pWgw3+/PfbuH++OIQJLPjTuv1P805kFD8toniD+It51Qjxhw9Yh4pbnQFIfBEYM54NhqDACsMyZQly3/LIWAmBn1QPYRucT4Cq0xck1Jq77guOWT+xkzLrB3avvk7q9hI+YWduySgBND3HZ39cNgRtnkGFOCF72G3l44iprrgegqyXowAPvxV1V/1BtVDIO0/+H/ydDAjQPySnCsmgCbF6ym7l9+9xp+RDLGSuX0Cs83BKfdz2B1fgbtvv8TvVCA2XEzXJB8tpwuVkaTDogMWee1wO7pT2TH88cLGiAJpAYwHtA3gk2+Il/x0/Zz8OphXpFA02Zy3Jkn2uv36E0pLBF3kNU+X6kL/wSKBbxmRJmHU9tgJO/lT9nSNOVcuarTxuTriiofchRKd1CoDRJ3VBRwRD/WFDA1lL5aakP1MaIYnZyYkSnPEvCzwrpSARTGs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.004168, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Recognize 'void *p__map' kfunc argument as 'struct bpf_map *p__map'. It allows kfunc to have 'void *' argument for maps, since bpf progs will call them as: struct { __uint(type, BPF_MAP_TYPE_ARENA); ... } arena SEC(".maps"); bpf_kfunc_with_map(... &arena ...); Underneath libbpf will load CONST_PTR_TO_MAP into the register via ld_imm64 insn. If kfunc was defined with 'struct bpf_map *' it would pass the verifier, but bpf prog would need to use '(void *)&arena'. Which is not clean. Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d9c2dbb3939f..db569ce89fb1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10741,6 +10741,11 @@ static bool is_kfunc_arg_ignore(const struct btf *btf, const struct btf_param *a return __kfunc_param_match_suffix(btf, arg, "__ign"); } +static bool is_kfunc_arg_map(const struct btf *btf, const struct btf_param *arg) +{ + return __kfunc_param_match_suffix(btf, arg, "__map"); +} + static bool is_kfunc_arg_alloc_obj(const struct btf *btf, const struct btf_param *arg) { return __kfunc_param_match_suffix(btf, arg, "__alloc"); @@ -11064,7 +11069,7 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, return KF_ARG_PTR_TO_CONST_STR; if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { - if (!btf_type_is_struct(ref_t)) { + if (!btf_type_is_struct(ref_t) && !btf_type_is_void(ref_t)) { verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", meta->func_name, argno, btf_type_str(ref_t), ref_tname); return -EINVAL; @@ -11660,6 +11665,13 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (kf_arg_type < 0) return kf_arg_type; + if (is_kfunc_arg_map(btf, &args[i])) { + /* If argument has '__map' suffix expect 'struct bpf_map *' */ + ref_id = *reg2btf_ids[CONST_PTR_TO_MAP]; + ref_t = btf_type_by_id(btf_vmlinux, ref_id); + ref_tname = btf_name_by_offset(btf, ref_t->name_off); + } + switch (kf_arg_type) { case KF_ARG_PTR_TO_NULL: continue; From patchwork Tue Feb 6 22:04:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DF43C48297 for ; Tue, 6 Feb 2024 22:05:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F09EF6B0087; Tue, 6 Feb 2024 17:05:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EB8E56B0088; Tue, 6 Feb 2024 17:05:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D59746B0089; Tue, 6 Feb 2024 17:05:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BDCD56B0087 for ; Tue, 6 Feb 2024 17:05:00 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7CD871C162D for ; Tue, 6 Feb 2024 22:05:00 +0000 (UTC) X-FDA: 81762759960.21.B226E4E Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf28.hostedemail.com (Postfix) with ESMTP id 90D11C0007 for ; Tue, 6 Feb 2024 22:04:58 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SJ9DwrPc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257098; a=rsa-sha256; cv=none; b=59Zw2F09kBUUOHtrp0ptsiMcJV8g9Qch8LL/5SQpbAtecJ+Xeuwgt9hwGr+ZNvsmMwdRLt HGJy0Kbmr++mrnf+y9E8Dc1GbxM+Xvhe4W7MJj5wKZbP2UIRArywXn5U15NHKjZBx2t6OV LtDFiunfJNPiriygepXbcpfOLHAuPSU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=SJ9DwrPc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Tj/XQbWXLMz6O6W1qhMmj7yBJ5q6YLDIAk/QCdta60=; b=e9vF8LXbiiLINITHV2uBcWESvcs2oqq07tmTJzuDPs3ygMQjjwvoZsdIWBDEceYYwq+heo pt7SXWNI9fCDC69zImlir1eoKYlGQmj+IkXcj5qRaZ4sigroyUdU1bfjqPJg2PWmjQNXAq 00qNrMHTT6lzK2jTPqscV1t3vJyH0VE= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6de3141f041so24815b3a.0 for ; Tue, 06 Feb 2024 14:04:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257097; x=1707861897; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Tj/XQbWXLMz6O6W1qhMmj7yBJ5q6YLDIAk/QCdta60=; b=SJ9DwrPcQylHcjUud8M7Z946seypusXtfaFoiq672Fdrik0E/HtJs7m7jEOf2CKASz H8XSP6LOV2nPujZVaLVS0T/eEwsbVeg61Fnqw6sqPZHY0V/0SQcAUGDw6MDRYDvkAgZL 2mOKFJUoa9yKKQsXX2PPBeHa4r8U2IsuIemaAgNHgDxUa2N0UeDYCp3sOHHEXc+NkWxK sloyrAuai31aRYt+Ut3a/daqfowS+sPdGkJvPaTW3cqahLk76dbXteIbLdloSXsNd5O2 k3wNrfz2iz+nf6afoFBuR4PDM7B352hBqptGcxXuDR+udT6LTXIJW21P0Ym88QDSj6HO N8xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257097; x=1707861897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Tj/XQbWXLMz6O6W1qhMmj7yBJ5q6YLDIAk/QCdta60=; b=R7q+Vr8GCkiQiXQKxJbNZVPv3IEquea1d7l6l6XBfiFUgPuE2PSd1GXKGWfiy47G+/ q4RbpvjuMwupy1XQJxWyQoTo/209fU9O+DQzSBNXOtwDHx4LLDdCc4iAG0QKYH4gdCEj XHcTEoGsoIeQhG+3pc3vr+KUaPT9Z4xwTqhn6My9HTOL1kQgU4vomX6sfLb0C9r56oLg IgZsIA6EY8dOaReCJkY8GjjYVUdzTNC/feRsLJp+V2v3X9umAcZIjF0SaN6xE+vxDLck y8zEJiO4l6X30hdnV2yxCDEKG0VrbvbldsvQSDZmDm/74zrHhMu+qNwBeTBDIrR3LVOX 9Q+A== X-Forwarded-Encrypted: i=1; AJvYcCUuBQAWSwdGhtvb3Kb3DG+kO831GZFq6VP7j0v+pwugFfmost2SAEK1KSbdfg3P1oghLD4AlTOz4rIxSLsGNXCMdVs= X-Gm-Message-State: AOJu0YwgYH/LUH8yWezvU0JVf5VBVVAgFsHnfYWBioD1XaL8A5YFCZsE ECn03TnyGg1KOi0qb96AhoBSIIAOv/Kbgh2JxEeD73ZM0KjaHE7Z X-Google-Smtp-Source: AGHT+IEjcAf6uXoS2ZepvZd4WPEhsb74cH/NcraOd61wF2bJGbGsztqlhDT3saZuKsz+GWEev5IaMA== X-Received: by 2002:a05:6a20:3598:b0:19c:2a8d:8b75 with SMTP id j24-20020a056a20359800b0019c2a8d8b75mr2472114pze.28.1707257097406; Tue, 06 Feb 2024 14:04:57 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUog5B9mBd/4ypN4iPfsLncV0Q6zXF9MI3t+m9OSEmXJlkyaYwbJU3Z5CWn6TyAb2DiJtsdoKuHJLryjn1E+YF6B9ICtSLE8PV+NwaK7s/bobASa1TzhWkioQw/pNKVdOB/uuhkbiOyjjZJRLYOlpKWVmq5s3/lmII40TDt44WDD0EUbz7vv76WbdbJJBqhRa1G7/kQyL5mMT0bXWt/GXEdG4CTcUpNAwn3jDPEp8+fy2NVxuh9M65ywbwVm9XrCApUkiY64O7kR5q1Bkwzb8au/2GMwMcXdy3L Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id c1-20020aa78c01000000b006e02e816f13sm2491180pfd.111.2024.02.06.14.04.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:04:57 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 03/16] mm: Expose vmap_pages_range() to the rest of the kernel. Date: Tue, 6 Feb 2024 14:04:28 -0800 Message-Id: <20240206220441.38311-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 90D11C0007 X-Stat-Signature: qwbing99164w786adhcipgmgxg65346e X-HE-Tag: 1707257098-203330 X-HE-Meta: U2FsdGVkX19eQZ1HpVHFhIpuDMgjmO2ShY1ZQoN6txUUJKkypIFuPWNUJ3FFRCpn/Hv47ESEwA168zK8eH3fAyuvfINrLXARhsx87bS38wqserw/U/KEjy5hc318v5Qi9MHA7yVbc7RtpT+ecN/Mqfh7/byzYJdlZbzaMQcsMmq1gjYq8/YJ2PQThIfwg1WR2LlN4b8A6Ly/hNVmuhdf37sv4oU0WX2mjJiNwRYqYZaXRJVsEJSuMgM9VH0yB2+1Wo4jzwkZ3JmIO7luGEnTlr8HbC9Bu/vfvX6y9QhGVEMCpOaXul2y2faQtsIqoz/cWNV9R7zUiL+CT0ST1kA6jC83TPi8wqkA6Uuq1OJE5OijStI5f1bkjewrU6a9lo6vKQQtL1QmAjTuIIbCrfgaYR7Zwc7KBJcUca0dvKxp3hXRfSXbLknjRvP2AwXVBEXJSa/36Zf/dAQu2I6Hitojsy54NyMbjDLIcAo3hd+6r2084Wm1yjSkUhVzHBvl+KKwEZgSzf6G7XozTBzogFidQCA1RhRDsmY66IzdX9hcHG4ajsE19xGq2g9H3EXOn6gAGauM7DmVzZD5GiRro9jUTa0Xpk6a4QAJfMpKLfbzJD446d0AiKVzvFZN0A8gkX9ivhVoSEuuSnVEOYJytNDiLJicHL/nUw/SR3NDlXKVGWXAN1cnj4jlFkHB02PERmDR//0pAros5+JwADdKiuAoLP+AnHbsQ5m4BOs1WUOz5vDWj+E6PsWlhaRE2OJL05Sq5k3UBKPGuhTfALFivZf9PJFQ+bf0NP5i1ChRBiievAxP6ky0N18ny4NNNY9ztASu0BVzOZrDxcwR58H+PjxD3fqnaTud38djChinFNhDD+9GAlRJJKeSFs2gdZB0Hms87iFLX3Pd4fw9Fv+0ZW7JNPpEAclxDdhpKctHK4Kohzqnexk5ZjsABjruUfCw5ccqi2pDz432nnMogm4Sqqv 05Jlcysb 54EVtpRtOyqzNb4CVfIhA+MposMwoP1TWpS/L1EoSYiYblgIdd84DFNqEPdEzKuQqOso/G/Nw42B6sao17LspqDqGk3PStxoZH6PU39SlZc0EsSkrgmABgrxFbT7s4aA217OnEzlh3fjwnqJdoDniTaGWLIRdVVhDddMWLP15Wudq76LyiHKkIYV+hXysm5wlPhOICCXOzg6QLgTKAruA7B5HJalvAEg7mYQ7cI092IFtRvc/br3amBTTWALq5Zoc+fRhS9QBhMDdgLyStzh2cE1W/ToktXO5zyHYdufdMDRgJRPi/FtGvm+pBPtOk+qbhCw7P41udNstVlw5ierdeA/UZfXzu7Qg6br587pYZi6s7mR8QhKoCXilk4Z9oCWW4anHxAJK6wyUeri+sYp+6X7wq/HhgmPRSMN/m3EHFt0fUYUKMlTvyCIzihbzhj2xx5RjOv1muCToeWaf8M+CQbDixWA1+sQbJwjt+C1wZLMgSfT1jkNu/n1MHHhsUDsdXF1hhPV9So0tuLCI7qw5hxG2F9zDTNyEa8SPkDRt+5qByTC96J8qolcpYlrVJgSjS0jKySbZQgQvs4k92dzwraTK/909TQEaTux7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000090, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov The next commit will introduce bpf_arena which is a sparsely populated shared memory region between bpf program and user space process. It will function similar to vmalloc()/vm_map_ram(): - get_vm_area() - alloc_pages() - vmap_pages_range() Signed-off-by: Alexei Starovoitov --- include/linux/vmalloc.h | 2 ++ mm/vmalloc.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..bafb87c69e3d 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -233,6 +233,8 @@ static inline bool is_vm_area_hugepages(const void *addr) #ifdef CONFIG_MMU void vunmap_range(unsigned long addr, unsigned long end); +int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift); static inline void set_vm_flush_reset_perms(void *addr) { struct vm_struct *vm = find_vm_area(addr); diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..eae93d575d1b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -625,8 +625,8 @@ int vmap_pages_range_noflush(unsigned long addr, unsigned long end, * RETURNS: * 0 on success, -errno on failure. */ -static int vmap_pages_range(unsigned long addr, unsigned long end, - pgprot_t prot, struct page **pages, unsigned int page_shift) +int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) { int err; From patchwork Tue Feb 6 22:04:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7106C4828D for ; Tue, 6 Feb 2024 22:05:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5952D6B008C; Tue, 6 Feb 2024 17:05:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 51D4B6B008A; Tue, 6 Feb 2024 17:05:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 349A26B008C; Tue, 6 Feb 2024 17:05:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 17D916B0089 for ; Tue, 6 Feb 2024 17:05:05 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BB2B580259 for ; Tue, 6 Feb 2024 22:05:04 +0000 (UTC) X-FDA: 81762760128.15.8525FEF Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf04.hostedemail.com (Postfix) with ESMTP id A36C44001A for ; Tue, 6 Feb 2024 22:05:02 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=d9e18RXk; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257102; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zRPDF2WyJ90sxBcsR07zzIbiv8DSmynkp0VU3oUCItw=; b=0OUb+fLx+9Qcx5ATeMEF0jXsXfmLH+Mh0HQvqwxW+OAPkMhM3EkIfkkNpefFN6C4h30VeN fFVdhzlMXNXEr13wOyp49KHLUwTUGUtvndhb5Qw+99a9aS+EAfIk0mao94Vpa6BCMSxVO7 TEjQgNbLjZJBZxsJlMaG7HvUrg1sXbk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=d9e18RXk; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf04.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257102; a=rsa-sha256; cv=none; b=njew75aqn4MCwfVxi6L00MN3Z8XZb77bmh8U5QL/Gi2bsjl6ZXQcdFbEr4F701YssJA+if ZHX5mKVIED7f0skUWxmed/Q+3UUi3AfEiEaeOk2HlW3CF4VHyR6WLkpib5gB49dGDWkf7Y F8orHnMl9LA/476wz/QdDH6V4o2uEi4= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d751bc0c15so49345ad.2 for ; Tue, 06 Feb 2024 14:05:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257101; x=1707861901; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zRPDF2WyJ90sxBcsR07zzIbiv8DSmynkp0VU3oUCItw=; b=d9e18RXkfs4ZsazGiN65bh45XEj8KEfbMxDcfCGXSvaaVDpylwcODVojxB6YZ5mDkF pUdEGxssy1b58w72HNAoKyYcrUBx8wzGbbF5tiHr8cMf1utE6sHLr4lci8bFjdgqyO3U o8lmBy+YEqSQurzvrT8LyEwFoOrmliZ5Q+LygFNkSiBdo4OAEZFfgBXf5ETve6GdnB6O 5eXwJWtjQxpGRMh4XHwcninUQHDnkS+/2hBIsFVxpq8PLjYFb5kjyR51/p7tMUbTT7Ke 0PG8x3UM4iMBjtDprJEZ8gsPtTUqvoLYAlvaUn9dOUqZtwOW9azrTsvQ/Ksln78BnYlN Aarg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257101; x=1707861901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zRPDF2WyJ90sxBcsR07zzIbiv8DSmynkp0VU3oUCItw=; b=dglXMU8AljN93elSRbkpYqw4EBOaJn9i7oGljx4+85u5FWa8OcVL1EhyMRbj0we1KC BZeFuBJoQzRRnAtG/O3356+7H15s2sY+hZ8GyWB7kl+oTDKRtzZmCELmhG19f1IYLPrS jL9WGjNdtG7KGcr//Pv6eZxtYuOmR5WKUUcCBYeVDBRbISTWWCVXJ5WF12IKwz9/q02v DS3MPIYA8rO1pLbYFO7ZDManUgaPGN56PJqwxL75O9cclf6kC/k5roHyAz8ZiSqCYyq4 hPuoDKj4i0JCEzscoXsqj1LkOevSR9jDGUEUQLdxcXQ1TDhvS7WE2XyQTBZ20/K0RU6G iPvQ== X-Gm-Message-State: AOJu0YwkaAc06SNsCS9ROUUt/uC0DjdMXn9Vm5HlJyShJ+xsVVcN0I99 ocdWQb8M21RO42nhE9y73AmmtyV8hbfTOUCiHUux7c2jqxHZv/nU X-Google-Smtp-Source: AGHT+IEtepAdRLBSLWFUq9tirKmmT0gbpTXjex2ZKR4a8TyPd8k5+A2+/4mAAuVQAMyu0aOaVpxbow== X-Received: by 2002:a17:902:650c:b0:1d9:e181:51a1 with SMTP id b12-20020a170902650c00b001d9e18151a1mr2088356plk.63.1707257101247; Tue, 06 Feb 2024 14:05:01 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCV8WWbVPJWh6zaUDnen2OVpdX6umSdhMLuzYuwxIuWOROsCkheZJbKYxODBqj9D4gKW4NLXhbrD3Nc/89x3ajgOY6atbAYsNzkE7gtanRzF5khpPDjCK/uW3EbENwLeTOqVffa8uTuLlwIHyaq/TdhauoinYbr1d1Se8qKo5puES63Zry+0vBAx0X3h82RHVdkNgSjT0JgN32v9KE3qV2778vVMWozCEdYB/WmuDMO4MPssiv+WACo28FfE3ZzL612oL6ZaYyW7gXoCSIkLgN5TJlvucX8Z326K Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id iy15-20020a170903130f00b001d9edac54b2sm2834plb.205.2024.02.06.14.04.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:00 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 04/16] bpf: Introduce bpf_arena. Date: Tue, 6 Feb 2024 14:04:29 -0800 Message-Id: <20240206220441.38311-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A36C44001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: n1xatd9ekdh1awu41e6i59te4tsx4tqo X-HE-Tag: 1707257102-919127 X-HE-Meta: U2FsdGVkX1/0oZEFj5tZDa5eExQVIn4C/m8dneufnXcBKeNq6/o1CayfgNRnIMeHk8b4jNfotG0cJ7EU/uyYE6ZKKtXmfwymc6AL0RgKHDoKpX8oxBaRUE5czx1k0JB8IkkbXSHjeEelo6wxB4eNrWyS1MFGMGo/SgOVI4MkXUNczkii6EfgLeL+0fC3untOe0RKvKM+rmoG+293KQnNRg+IXi0kFnFhh3zvknlSq3FqkbZaO6YRBOPmOYRebR33hfznDEa/2wGqePLmc+3Kt20/+MPqRIuc5BkbKr8rT/sPLAsjD2u1NS+ATStYWQK0w9XhKHmKxUIhfU7PGk/x180Bpv2DHr7c7vJFoyEK06+t02tbxZrRiDs2eFQwi+6/75iDgYrDjE6AxzW/UGQcUXoTsCNQn9srBZ4Rtw2tpW/XVyqrTTVjIxa/cXyNQ5fL78/Od1kPj7vmog6Ai7DWojHhT1tTvHRgBRJNv6ZA0Y2K1ruAW24ACQ+eA8Sje84+FD9MKQLOaIPQ17MC3rHI+HkJm+4IHO3C+jR2TQ068BWShgdhv6T05B+KzRfzYgA7lUSphcdMDsYFU3WEjOMsgsVxRk56r049FxPYTho+94BVT00fx0d31/U1SDXcDm9OHTvk1w8IAyAickwQ5Wwgvq+KSXZEyw2Ky6U7uMzEVnmwZWhyFntCVcQmaBYc3nqtmSwzDB7ouTuSNToL3bI/oBzmahhLJTSy8S4s2uKCoiBcUMftznN73mA+Z3xJBjqDw1mO3ZMEPXDTIpdUTR1VtgB+pRf4N7SbsrSNTkB/doI3AXUxhplX7yizhrXJTCiFj7nMyVzX6nl0Qv+1LurpZw4gkXx0GtG9pZLVqrXgyo0gWMpw+WsbWHimRumb0qhHJQU54/vqE/vLfbWyheVlKRj/G1vCPfujZ02Aoq1IAta00U6erwQEEnz5u3oMWC+zYIbxU+H/mL/lfYhHIjq vl4Hcj9m 2k9kTt0SuKs8oFws9qHAnRAFLQ2Nm+P9/cGPeK2lQ9PaYIWM67X0eH6am/V91y7K3Zzufug5ibIzwAO2OrHzfITT1UfDK4hmCtyWV0JrQT6QU9xK2Yi3sxpeDOGabrCfwd1P4ZOKii5mXRA65PBzwRr3h0F2150T+OreYVpxWQ5n8feESKYNLpQonyfd8nZBtKeWdoiouJnKqco0fstx9OIXV+9faCQDaEmnk/qXxNIj8Yu/xL5cBbU/vGfR/Kykyd2LVwy0V7kOyC1m4nbOvohQy3WDrGluKINWep3FDPnHlZK0GZM8o/AKq3aLyDj2f0vNvmrzavN1xfRiV5NxjxZA09mvKTY1JM1O1nB+EhAr2npUs/EU8QzKGFo9vZQDhcf2TzWz5GHVjY7r9AGYJWXe/4jcFcjZgxuVzfQcdyiwzCVeq2VtDw8bgy/cwIZIe1RkpeUI6hH1F08/eMaNMrPZ4rjXaeSaT5UAGJqCE/PNJsLfPKp8QB8FO4DwgukwW/144Rh6WzRb8FJvifdPWqGwvH7cwsAx0wDpfKOgeNtWV7GQi3+CmUVkqHcchTSdIFlBMbh6klJSnNXcVmsNy2SZ2Pj0Wq3wEaC6RGywe0qazvGdgqklMVBXyHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce bpf_arena, which is a sparse shared memory region between the bpf program and user space. Use cases: 1. User space mmap-s bpf_arena and uses it as a traditional mmap-ed anonymous region, like memcached or any key/value storage. The bpf program implements an in-kernel accelerator. XDP prog can search for a key in bpf_arena and return a value without going to user space. 2. The bpf program builds arbitrary data structures in bpf_arena (hash tables, rb-trees, sparse arrays), while user space occasionally consumes it. 3. bpf_arena is a "heap" of memory from the bpf program's point of view. It is not shared with user space. Initially, the kernel vm_area and user vma are not populated. User space can fault in pages within the range. While servicing a page fault, bpf_arena logic will insert a new page into the kernel and user vmas. The bpf program can allocate pages from that region via bpf_arena_alloc_pages(). This kernel function will insert pages into the kernel vm_area. The subsequent fault-in from user space will populate that page into the user vma. The BPF_F_SEGV_ON_FAULT flag at arena creation time can be used to prevent fault-in from user space. In such a case, if a page is not allocated by the bpf program and not present in the kernel vm_area, the user process will segfault. This is useful for use cases 2 and 3 above. bpf_arena_alloc_pages() is similar to user space mmap(). It allocates pages either at a specific address within the arena or allocates a range with the maple tree. bpf_arena_free_pages() is analogous to munmap(), which frees pages and removes the range from the kernel vm_area and from user process vmas. bpf_arena can be used as a bpf program "heap" of up to 4GB. The memory is not shared with user space. This is use case 3. In such a case, the BPF_F_NO_USER_CONV flag is recommended. It will tell the verifier to treat the rX = bpf_arena_cast_user(rY) instruction as a 32-bit move wX = wY, which will improve bpf prog performance. Otherwise, bpf_arena_cast_user is translated by JIT to conditionally add the upper 32 bits of user vm_start (if the pointer is not NULL) to arena pointers before they are stored into memory. This way, user space sees them as valid 64-bit pointers. Diff https://github.com/llvm/llvm-project/pull/79902 taught LLVM BPF backend to generate the bpf_cast_kern() instruction before dereference of the arena pointer and the bpf_cast_user() instruction when the arena pointer is formed. In a typical bpf program there will be very few bpf_cast_user(). From LLVM's point of view, arena pointers are tagged as __attribute__((address_space(1))). Hence, clang provides helpful diagnostics when pointers cross address space. Libbpf and the kernel support only address_space == 1. All other address space identifiers are reserved. rX = bpf_cast_kern(rY, addr_space) tells the verifier that rX->type = PTR_TO_ARENA. Any further operations on PTR_TO_ARENA register have to be in the 32-bit domain. The verifier will mark load/store through PTR_TO_ARENA with PROBE_MEM32. JIT will generate them as kern_vm_start + 32bit_addr memory accesses. The behavior is similar to copy_from_kernel_nofault() except that no address checks are necessary. The address is guaranteed to be in the 4GB range. If the page is not present, the destination register is zeroed on read, and the operation is ignored on write. rX = bpf_cast_user(rY, addr_space) tells the verifier that rX->type = unknown scalar. If arena->map_flags has BPF_F_NO_USER_CONV set, then the verifier converts cast_user to mov32. Otherwise, JIT will emit native code equivalent to: rX = (u32)rY; if (rX) rX |= arena->user_vm_start & ~(u64)~0U; After such conversion, the pointer becomes a valid user pointer within bpf_arena range. The user process can access data structures created in bpf_arena without any additional computations. For example, a linked list built by a bpf program can be walked natively by user space. Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 5 +- include/linux/bpf_types.h | 1 + include/uapi/linux/bpf.h | 7 + kernel/bpf/Makefile | 3 + kernel/bpf/arena.c | 518 +++++++++++++++++++++++++++++++++ kernel/bpf/core.c | 11 + kernel/bpf/syscall.c | 3 + kernel/bpf/verifier.c | 1 + tools/include/uapi/linux/bpf.h | 7 + 9 files changed, 554 insertions(+), 2 deletions(-) create mode 100644 kernel/bpf/arena.c diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1ebbee1d648e..42f22bc881f0 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -37,6 +37,7 @@ struct perf_event; struct bpf_prog; struct bpf_prog_aux; struct bpf_map; +struct bpf_arena; struct sock; struct seq_file; struct btf; @@ -531,8 +532,8 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock); void bpf_rb_root_free(const struct btf_field *field, void *rb_root, struct bpf_spin_lock *spin_lock); - - +u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena); +u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena); int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size); struct bpf_offload_dev; diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h index 94baced5a1ad..9f2a6b83b49e 100644 --- a/include/linux/bpf_types.h +++ b/include/linux/bpf_types.h @@ -132,6 +132,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_STRUCT_OPS, bpf_struct_ops_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) +BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops) BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint) BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index d96708380e52..f6648851eae6 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -983,6 +983,7 @@ enum bpf_map_type { BPF_MAP_TYPE_BLOOM_FILTER, BPF_MAP_TYPE_USER_RINGBUF, BPF_MAP_TYPE_CGRP_STORAGE, + BPF_MAP_TYPE_ARENA, __MAX_BPF_MAP_TYPE }; @@ -1370,6 +1371,12 @@ enum { /* BPF token FD is passed in a corresponding command's token_fd field */ BPF_F_TOKEN_FD = (1U << 16), + +/* When user space page faults in bpf_arena send SIGSEGV instead of inserting new page */ + BPF_F_SEGV_ON_FAULT = (1U << 17), + +/* Do not translate kernel bpf_arena pointers to user pointers */ + BPF_F_NO_USER_CONV = (1U << 18), }; /* Flags for BPF_PROG_QUERY. */ diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile index 4ce95acfcaa7..368c5d86b5b7 100644 --- a/kernel/bpf/Makefile +++ b/kernel/bpf/Makefile @@ -15,6 +15,9 @@ obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o obj-$(CONFIG_BPF_JIT) += trampoline.o obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o +ifeq ($(CONFIG_MMU)$(CONFIG_64BIT),yy) +obj-$(CONFIG_BPF_SYSCALL) += arena.o +endif obj-$(CONFIG_BPF_JIT) += dispatcher.o ifeq ($(CONFIG_NET),y) obj-$(CONFIG_BPF_SYSCALL) += devmap.o diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c new file mode 100644 index 000000000000..9db720321700 --- /dev/null +++ b/kernel/bpf/arena.c @@ -0,0 +1,518 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include +#include + +/* + * bpf_arena is a sparsely populated shared memory region between bpf program and + * user space process. + * + * For example on x86-64 the values could be: + * user_vm_start 7f7d26200000 // picked by mmap() + * kern_vm_start ffffc90001e69000 // picked by get_vm_area() + * For user space all pointers within the arena are normal 8-byte addresses. + * In this example 7f7d26200000 is the address of the first page (pgoff=0). + * The bpf program will access it as: kern_vm_start + lower_32bit_of_user_ptr + * (u32)7f7d26200000 -> 26200000 + * hence + * ffffc90001e69000 + 26200000 == ffffc90028069000 is "pgoff=0" within 4Gb + * kernel memory region. + * + * BPF JITs generate the following code to access arena: + * mov eax, eax // eax has lower 32-bit of user pointer + * mov word ptr [rax + r12 + off], bx + * where r12 == kern_vm_start and off is s16. + * Hence allocate 4Gb + GUARD_SZ/2 on each side. + * + * Initially kernel vm_area and user vma are not populated. + * User space can fault-in any address which will insert the page + * into kernel and user vma. + * bpf program can allocate a page via bpf_arena_alloc_pages() kfunc + * which will insert it into kernel vm_area. + * The later fault-in from user space will populate that page into user vma. + */ + +/* number of bytes addressable by LDX/STX insn with 16-bit 'off' field */ +#define GUARD_SZ (1ull << sizeof(((struct bpf_insn *)0)->off) * 8) +#define KERN_VM_SZ ((1ull << 32) + GUARD_SZ) + +struct bpf_arena { + struct bpf_map map; + u64 user_vm_start; + u64 user_vm_end; + struct vm_struct *kern_vm; + struct maple_tree mt; + struct list_head vma_list; + struct mutex lock; +}; + +u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) +{ + return arena ? (u64) (long) arena->kern_vm->addr + GUARD_SZ / 2 : 0; +} + +u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) +{ + return arena ? arena->user_vm_start : 0; +} + +static long arena_map_peek_elem(struct bpf_map *map, void *value) +{ + return -EOPNOTSUPP; +} + +static long arena_map_push_elem(struct bpf_map *map, void *value, u64 flags) +{ + return -EOPNOTSUPP; +} + +static long arena_map_pop_elem(struct bpf_map *map, void *value) +{ + return -EOPNOTSUPP; +} + +static long arena_map_delete_elem(struct bpf_map *map, void *value) +{ + return -EOPNOTSUPP; +} + +static int arena_map_get_next_key(struct bpf_map *map, void *key, void *next_key) +{ + return -EOPNOTSUPP; +} + +static long compute_pgoff(struct bpf_arena *arena, long uaddr) +{ + return (u32)(uaddr - (u32)arena->user_vm_start) >> PAGE_SHIFT; +} + +#define MT_ENTRY ((void *)&arena_map_ops) /* unused. has to be valid pointer */ + +/* + * Reserve a "zero page", so that bpf prog and user space never see + * a pointer to arena with lower 32 bits being zero. + * bpf_cast_user() promotes it to full 64-bit NULL. + */ +static int reserve_zero_page(struct bpf_arena *arena) +{ + long pgoff = compute_pgoff(arena, 0); + + return mtree_insert(&arena->mt, pgoff, MT_ENTRY, GFP_KERNEL); +} + +static struct bpf_map *arena_map_alloc(union bpf_attr *attr) +{ + struct vm_struct *kern_vm; + int numa_node = bpf_map_attr_numa_node(attr); + struct bpf_arena *arena; + int err = -ENOMEM; + + if (attr->key_size != 8 || attr->value_size != 8 || + /* BPF_F_MMAPABLE must be set */ + !(attr->map_flags & BPF_F_MMAPABLE) || + /* No unsupported flags present */ + (attr->map_flags & ~(BPF_F_SEGV_ON_FAULT | BPF_F_MMAPABLE | BPF_F_NO_USER_CONV))) + return ERR_PTR(-EINVAL); + + if (attr->map_extra & ~PAGE_MASK) + /* If non-zero the map_extra is an expected user VMA start address */ + return ERR_PTR(-EINVAL); + + kern_vm = get_vm_area(KERN_VM_SZ, VM_MAP | VM_USERMAP); + if (!kern_vm) + return ERR_PTR(-ENOMEM); + + arena = bpf_map_area_alloc(sizeof(*arena), numa_node); + if (!arena) + goto err; + + INIT_LIST_HEAD(&arena->vma_list); + arena->kern_vm = kern_vm; + arena->user_vm_start = attr->map_extra; + bpf_map_init_from_attr(&arena->map, attr); + mt_init_flags(&arena->mt, MT_FLAGS_ALLOC_RANGE); + mutex_init(&arena->lock); + if (arena->user_vm_start) { + err = reserve_zero_page(arena); + if (err) { + bpf_map_area_free(arena); + goto err; + } + } + + return &arena->map; +err: + free_vm_area(kern_vm); + return ERR_PTR(err); +} + +static int for_each_pte(pte_t *ptep, unsigned long addr, void *data) +{ + struct page *page; + pte_t pte; + + pte = ptep_get(ptep); + if (!pte_present(pte)) + return 0; + page = pte_page(pte); + /* + * We do not update pte here: + * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug + * 2. TLB flushing is batched or deferred. Even if we clear pte, + * the TLB entries can stick around and continue to permit access to + * the freed page. So it all relies on 1. + */ + __free_page(page); + return 0; +} + +static void arena_map_free(struct bpf_map *map) +{ + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + /* + * Check that user vma-s are not around when bpf map is freed. + * mmap() holds vm_file which holds bpf_map refcnt. + * munmap() must have happened on vma followed by arena_vm_close() + * which would clear arena->vma_list. + */ + if (WARN_ON_ONCE(!list_empty(&arena->vma_list))) + return; + + /* + * free_vm_area() calls remove_vm_area() that calls free_unmap_vmap_area(). + * It unmaps everything from vmalloc area and clears pgtables. + * Call apply_to_existing_page_range() first to find populated ptes and + * free those pages. + */ + apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), + KERN_VM_SZ - GUARD_SZ / 2, for_each_pte, NULL); + free_vm_area(arena->kern_vm); + mtree_destroy(&arena->mt); + bpf_map_area_free(arena); +} + +static void *arena_map_lookup_elem(struct bpf_map *map, void *key) +{ + return ERR_PTR(-EINVAL); +} + +static long arena_map_update_elem(struct bpf_map *map, void *key, + void *value, u64 flags) +{ + return -EOPNOTSUPP; +} + +static int arena_map_check_btf(const struct bpf_map *map, const struct btf *btf, + const struct btf_type *key_type, const struct btf_type *value_type) +{ + return 0; +} + +static u64 arena_map_mem_usage(const struct bpf_map *map) +{ + return 0; +} + +struct vma_list { + struct vm_area_struct *vma; + struct list_head head; +}; + +static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) +{ + struct vma_list *vml; + + vml = kmalloc(sizeof(*vml), GFP_KERNEL); + if (!vml) + return -ENOMEM; + vma->vm_private_data = vml; + vml->vma = vma; + list_add(&vml->head, &arena->vma_list); + return 0; +} + +static void arena_vm_close(struct vm_area_struct *vma) +{ + struct vma_list *vml; + + vml = vma->vm_private_data; + list_del(&vml->head); + vma->vm_private_data = NULL; + kfree(vml); +} + +static vm_fault_t arena_vm_fault(struct vm_fault *vmf) +{ + struct bpf_map *map = vmf->vma->vm_file->private_data; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + struct page *page; + long kbase, kaddr; + int ret; + + kbase = bpf_arena_get_kern_vm_start(arena); + kaddr = kbase + (u32)(vmf->address & PAGE_MASK); + + guard(mutex)(&arena->lock); + page = vmalloc_to_page((void *)kaddr); + if (page) + /* already have a page vmap-ed */ + goto out; + + if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT) + /* User space requested to segfault when page is not allocated by bpf prog */ + return VM_FAULT_SIGSEGV; + + ret = mtree_insert(&arena->mt, vmf->pgoff, MT_ENTRY, GFP_KERNEL); + if (ret == -EEXIST) + return VM_FAULT_RETRY; + if (ret) + return VM_FAULT_SIGSEGV; + + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) { + mtree_erase(&arena->mt, vmf->pgoff); + return VM_FAULT_SIGSEGV; + } + + ret = vmap_pages_range(kaddr, kaddr + PAGE_SIZE, PAGE_KERNEL, &page, PAGE_SHIFT); + if (ret) { + mtree_erase(&arena->mt, vmf->pgoff); + __free_page(page); + return VM_FAULT_SIGSEGV; + } +out: + page_ref_add(page, 1); + vmf->page = page; + return 0; +} + +static const struct vm_operations_struct arena_vm_ops = { + .close = arena_vm_close, + .fault = arena_vm_fault, +}; + +static int arena_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) +{ + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + int err; + + if (arena->user_vm_start && arena->user_vm_start != vma->vm_start) + /* + * 1st user process can do mmap(NULL, ...) to pick user_vm_start + * 2nd user process must pass the same addr to mmap(addr, MAP_FIXED..); + * or + * specify addr in map_extra at map creation time and + * use the same addr later with mmap(addr, MAP_FIXED..); + */ + return -EBUSY; + + if (arena->user_vm_end && arena->user_vm_end != vma->vm_end) + /* all user processes must have the same size of mmap-ed region */ + return -EBUSY; + + if (vma->vm_end - vma->vm_start > 1ull << 32) + /* Must not be bigger than 4Gb */ + return -E2BIG; + + if (remember_vma(arena, vma)) + return -ENOMEM; + + if (!arena->user_vm_start) { + arena->user_vm_start = vma->vm_start; + err = reserve_zero_page(arena); + if (err) + return err; + } + arena->user_vm_end = vma->vm_end; + /* + * bpf_map_mmap() checks that it's being mmaped as VM_SHARED and + * clears VM_MAYEXEC. Set VM_DONTEXPAND as well to avoid + * potential change of user_vm_start. + */ + vm_flags_set(vma, VM_DONTEXPAND); + vma->vm_ops = &arena_vm_ops; + return 0; +} + +BTF_ID_LIST_SINGLE(bpf_arena_map_btf_ids, struct, bpf_arena) +const struct bpf_map_ops arena_map_ops = { + .map_meta_equal = bpf_map_meta_equal, + .map_alloc = arena_map_alloc, + .map_free = arena_map_free, + .map_mmap = arena_map_mmap, + .map_get_next_key = arena_map_get_next_key, + .map_push_elem = arena_map_push_elem, + .map_peek_elem = arena_map_peek_elem, + .map_pop_elem = arena_map_pop_elem, + .map_lookup_elem = arena_map_lookup_elem, + .map_update_elem = arena_map_update_elem, + .map_delete_elem = arena_map_delete_elem, + .map_check_btf = arena_map_check_btf, + .map_mem_usage = arena_map_mem_usage, + .map_btf_id = &bpf_arena_map_btf_ids[0], +}; + +static u64 clear_lo32(u64 val) +{ + return val & ~(u64)~0U; +} + +/* + * Allocate pages and vmap them into kernel vmalloc area. + * Later the pages will be mmaped into user space vma. + */ +static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt, int node_id) +{ + long page_cnt_max = (arena->user_vm_end - arena->user_vm_start) >> PAGE_SHIFT; + u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena); + long pgoff = 0, kaddr, nr_pages = 0; + struct page **pages; + int ret, i; + + if (page_cnt >= page_cnt_max) + return 0; + + if (uaddr) { + if (uaddr & ~PAGE_MASK) + return 0; + pgoff = compute_pgoff(arena, uaddr); + if (pgoff + page_cnt > page_cnt_max) + /* requested address will be outside of user VMA */ + return 0; + } + + /* zeroing is needed, since alloc_pages_bulk_array() only fills in non-zero entries */ + pages = kvcalloc(page_cnt, sizeof(struct page *), GFP_KERNEL); + if (!pages) + return 0; + + guard(mutex)(&arena->lock); + + if (uaddr) + ret = mtree_insert_range(&arena->mt, pgoff, pgoff + page_cnt, + MT_ENTRY, GFP_KERNEL); + else + ret = mtree_alloc_range(&arena->mt, &pgoff, MT_ENTRY, + page_cnt, 0, page_cnt_max, GFP_KERNEL); + if (ret) + goto out_free_pages; + + nr_pages = alloc_pages_bulk_array_node(GFP_KERNEL | __GFP_ZERO, node_id, page_cnt, pages); + if (nr_pages != page_cnt) + goto out; + + kaddr = kern_vm_start + (u32)(arena->user_vm_start + pgoff * PAGE_SIZE); + ret = vmap_pages_range(kaddr, kaddr + PAGE_SIZE * page_cnt, PAGE_KERNEL, + pages, PAGE_SHIFT); + if (ret) + goto out; + kvfree(pages); + return clear_lo32(arena->user_vm_start) + (u32)(kaddr - kern_vm_start); +out: + mtree_erase(&arena->mt, pgoff); +out_free_pages: + if (pages) + for (i = 0; i < nr_pages; i++) + __free_page(pages[i]); + kvfree(pages); + return 0; +} + +/* + * If page is present in vmalloc area, unmap it from vmalloc area, + * unmap it from all user space vma-s, + * and free it. + */ +static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) +{ + struct vma_list *vml; + + list_for_each_entry(vml, &arena->vma_list, head) + zap_page_range_single(vml->vma, uaddr, + PAGE_SIZE * page_cnt, NULL); +} + +static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt) +{ + u64 full_uaddr, uaddr_end; + long kaddr, pgoff, i; + struct page *page; + + /* only aligned lower 32-bit are relevant */ + uaddr = (u32)uaddr; + uaddr &= PAGE_MASK; + full_uaddr = clear_lo32(arena->user_vm_start) + uaddr; + uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT)); + if (full_uaddr >= uaddr_end) + return; + + page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT; + + kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr; + + guard(mutex)(&arena->lock); + + pgoff = compute_pgoff(arena, uaddr); + /* clear range */ + mtree_store_range(&arena->mt, pgoff, pgoff + page_cnt, NULL, GFP_KERNEL); + + if (page_cnt > 1) + /* bulk zap if multiple pages being freed */ + zap_pages(arena, full_uaddr, page_cnt); + + for (i = 0; i < page_cnt; i++, kaddr += PAGE_SIZE, full_uaddr += PAGE_SIZE) { + page = vmalloc_to_page((void *)kaddr); + if (!page) + continue; + if (page_cnt == 1 && page_mapped(page)) /* mapped by some user process */ + zap_pages(arena, full_uaddr, 1); + vunmap_range(kaddr, kaddr + PAGE_SIZE); + __free_page(page); + } +} + +__bpf_kfunc_start_defs(); + +__bpf_kfunc void *bpf_arena_alloc_pages(void *p__map, void *addr__ign, u32 page_cnt, + int node_id, u64 flags) +{ + struct bpf_map *map = p__map; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + if (map->map_type != BPF_MAP_TYPE_ARENA || !arena->user_vm_start || flags) + return NULL; + + return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id); +} + +__bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) +{ + struct bpf_map *map = p__map; + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); + + if (map->map_type != BPF_MAP_TYPE_ARENA || !arena->user_vm_start) + return; + arena_free_pages(arena, (long)ptr__ign, page_cnt); +} +__bpf_kfunc_end_defs(); + +BTF_KFUNCS_START(arena_kfuncs) +BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_KFUNCS_END(arena_kfuncs) + +static const struct btf_kfunc_id_set common_kfunc_set = { + .owner = THIS_MODULE, + .set = &arena_kfuncs, +}; + +static int __init kfunc_init(void) +{ + return register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &common_kfunc_set); +} +late_initcall(kfunc_init); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 71c459a51d9e..2539d9bfe369 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2970,6 +2970,17 @@ void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, { } +/* for configs without MMU or 32-bit */ +__weak const struct bpf_map_ops arena_map_ops; +__weak u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) +{ + return 0; +} +__weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) +{ + return 0; +} + #ifdef CONFIG_BPF_SYSCALL static int __init bpf_global_ma_init(void) { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index b2750b79ac80..ac0e4a8bb852 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -164,6 +164,7 @@ static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, if (bpf_map_is_offloaded(map)) { return bpf_map_offload_update_elem(map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_CPUMAP || + map->map_type == BPF_MAP_TYPE_ARENA || map->map_type == BPF_MAP_TYPE_STRUCT_OPS) { return map->ops->map_update_elem(map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH || @@ -1160,6 +1161,7 @@ static int map_create(union bpf_attr *attr) } if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER && + attr->map_type != BPF_MAP_TYPE_ARENA && attr->map_extra != 0) return -EINVAL; @@ -1249,6 +1251,7 @@ static int map_create(union bpf_attr *attr) case BPF_MAP_TYPE_LRU_PERCPU_HASH: case BPF_MAP_TYPE_STRUCT_OPS: case BPF_MAP_TYPE_CPUMAP: + case BPF_MAP_TYPE_ARENA: if (!bpf_token_capable(token, CAP_BPF)) goto put_token; break; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index db569ce89fb1..3c77a3ab1192 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18047,6 +18047,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, case BPF_MAP_TYPE_SK_STORAGE: case BPF_MAP_TYPE_TASK_STORAGE: case BPF_MAP_TYPE_CGRP_STORAGE: + case BPF_MAP_TYPE_ARENA: break; default: verbose(env, diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index d96708380e52..f6648851eae6 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -983,6 +983,7 @@ enum bpf_map_type { BPF_MAP_TYPE_BLOOM_FILTER, BPF_MAP_TYPE_USER_RINGBUF, BPF_MAP_TYPE_CGRP_STORAGE, + BPF_MAP_TYPE_ARENA, __MAX_BPF_MAP_TYPE }; @@ -1370,6 +1371,12 @@ enum { /* BPF token FD is passed in a corresponding command's token_fd field */ BPF_F_TOKEN_FD = (1U << 16), + +/* When user space page faults in bpf_arena send SIGSEGV instead of inserting new page */ + BPF_F_SEGV_ON_FAULT = (1U << 17), + +/* Do not translate kernel bpf_arena pointers to user pointers */ + BPF_F_NO_USER_CONV = (1U << 18), }; /* Flags for BPF_PROG_QUERY. */ From patchwork Tue Feb 6 22:04:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41974C48297 for ; Tue, 6 Feb 2024 22:05:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B865B6B0089; Tue, 6 Feb 2024 17:05:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE8D36B008A; Tue, 6 Feb 2024 17:05:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 962776B0092; Tue, 6 Feb 2024 17:05:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 821736B0089 for ; Tue, 6 Feb 2024 17:05:08 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5D8FE1408B6 for ; Tue, 6 Feb 2024 22:05:08 +0000 (UTC) X-FDA: 81762760296.05.21E58E0 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf12.hostedemail.com (Postfix) with ESMTP id 5A1804001D for ; Tue, 6 Feb 2024 22:05:06 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ig1Vh2+m; spf=pass (imf12.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257106; a=rsa-sha256; cv=none; b=jaBeH4TH3z2Vql0PG1d6vDL3m7iItDymVzrvoxhDTMvC6xi3+ne6pjf1NqVzSqpQB7CzLT IluYvB7fGxdbGHLyyRlDBh8E43R8fK41hQO+V7SyDSrBpdLxibpE5HjP8xNZYiA5qGOglv SHCEqX1de4Zygt6hE6+JVZBpZjeilRA= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ig1Vh2+m; spf=pass (imf12.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257106; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dj+O15ERlc2oNccgk7u96NynUdtw7IljkhhXG+cbct8=; b=8MScsbwT1vbeNZ1bMHv4ZpqU6jS347PsNSKHFWi/xLuMhONNKBePQMRGIwL6KCSH0n0CDA 6oraOFFzJXwGuZKxLsC9HNAT0euVlOhnjJCm1zL8r1RulTbyFCyOnFJySU4+kawOkC3EDL QSwp9c6ASBx0ZaVqzno7wfaS2X8/RAE= Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e04f29c7bcso20654b3a.0 for ; Tue, 06 Feb 2024 14:05:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257105; x=1707861905; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dj+O15ERlc2oNccgk7u96NynUdtw7IljkhhXG+cbct8=; b=ig1Vh2+mbeCOarpR9SDVfWtsX6NxvDCGlgaFaGjpmMxxtiyugJ2/z/6nqWeokBAvrW Acr2WyMn4X5Zk45TrFsrvzyoIqdiD1zwHKLImXiiO4+17IxJAiru/mIEkxUmz95xmTnv qmYSxUbmFFIDfF6NVaWVAzXZzKvg/B8hmsj/6xFyDBEBBWIOMj446smqDVRW4ITiXhqG Ph9wp4Hsc46x04bupNktnhhnVJhbhb61lzDw9XYJGuRny6bxAIVXk2xKQr3Qzpq+z/tp K+ZPOx4kvS7Hbwb9GItsy5ZPwbdze9tjNsSE04CPfmaJuyGP0qEizu+L7/pthX3JzPXQ kgCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257105; x=1707861905; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dj+O15ERlc2oNccgk7u96NynUdtw7IljkhhXG+cbct8=; b=CAAjREAdZ7PahcBOexlBYxkQxk7FQGf6fdoTPSf5hm4Z0ZrH0qadCrLfgRQTr5nMBg nZRRFzUThEtK2KkRz8R6Hb7aANYo9TuW2WvkgHqAq+z5zeY+mb2pLwqfjoG5PC3LRLRi HNl1vYFcZ6VgdV0tE0YBrbGj+2ausyI6ZtUSC9FQD+xqKdqIy8B1o+mCBy4jFqhbcwZK fHTV054Mbfl6JySA3NgcOYGfHbIKbpGFRy9TT5CidjScse8d4R2RRO37WBdC7Jf5L9WA neTORFC+fWBfeXDWnQqzMTSR0OakThU+vifT+jf0ltd17EHgnfUolGhmKYY9lq7h9OpN JhKw== X-Forwarded-Encrypted: i=1; AJvYcCUGZy5iol3mad8OkNxJXHEMAPi/15Ynw3VL6SBVr5hHlz64vENEJv1cI6fMY1532GFHNZ3tkLKZNT/52RE+MFHjD60= X-Gm-Message-State: AOJu0Yxqyl1OgJ1NR6gkb935Bw8educBbey8gtRDM8jZ6cYBEFt1+pTD mp3JJzeKKvFyLikTlgr3OeKrCcTVtbyTwWA0GxSMAKlBdo4r4YvT X-Google-Smtp-Source: AGHT+IF7ufxUlX2aXoymen9Yws0uT15WBLonSQYDvE1rmVUQQKdyNHEjxdlBQPSGpbNEV8D1c9ZwsQ== X-Received: by 2002:a05:6a21:3a97:b0:19e:a19f:f4de with SMTP id zv23-20020a056a213a9700b0019ea19ff4demr1218529pzb.41.1707257105150; Tue, 06 Feb 2024 14:05:05 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXj2AKcZu7kXri6G+OwPqAfvSb7LujQdy2VsFPv9eQ5TTr8zwqyW+OhhQE42K7slkCGEV7sQJxPbSPGLxYy8wHDClhLz9RnrYV7FZ+YKL6xYzjNjxvM+EanhnVJ7dKX24XsaCGCfAiX9N3zPMVgAVfAp5dRizJWf8lYj3JR7V1VJkb1D7jhXy5H1h4GflKBzqGRglGhP2MSRAtPNqQSMMAPaPzrVj0mH2kZaapd1a8eYTmbX1n+M3iNC1d1P9HzYC0d5elXhE2qkBHHLnqX9578jlxiWafpgkAE Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id e26-20020aa798da000000b006dd810cdd91sm2519731pfm.88.2024.02.06.14.05.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:04 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 05/16] bpf: Disasm support for cast_kern/user instructions. Date: Tue, 6 Feb 2024 14:04:30 -0800 Message-Id: <20240206220441.38311-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5A1804001D X-Stat-Signature: zs3hf4qtaaxrkka6z548i7zmcbzthy6w X-Rspam-User: X-HE-Tag: 1707257106-532361 X-HE-Meta: U2FsdGVkX18rb9omLPSg0nmjIs+ZegPi8PXTqqoaeECvLTpUoA4QEP1EojGyelRoYeBxY9EqsJREUMaQY66XwSrJsozRXUJTfXGaI16qlFW2GdgAeB+wQytQ/5DHH3Y1zP/xtzzgyKExTM+PsECy7KohBpzu6iLbx3UO1tIeRi/JHKI5Rg2H4CEWC4w37qaalP21hLlykRq00537KOUAgMotf6kXXZw43K+IJzn+2JuMR0alG5xPOAELUjnNslp+2CSk8NbM3yncf8LV+MmbnGGaf6K9Ewky12qbf6SMlddbaY738+og1CnpfiyqUBSUpamZrtce7e7fLqZfOO7nlOzHoH+HSGDSvBtVcGHE1VtpV2gJmhoZZfLi1QOCaAu8aPbkzzuJk+Am2TFPHIHSzklxfraImQK/uyZg6YedDFY6xmSgzwwDyhglLruX/iiH/l7aPN2MxjY3ADTrZrXefGJUzIl8VcLCK44XWkAjbfrxEl8xg709T7MSCL7fLh+IB7dGehAqzBQ69SsCOMLC3bF5zr+HuJ7v9n/RW68qWj/7s8Fju6d8jkZ08FK97XrV+P7LgTU83nUZ9AFya/CtjxE2CDTIgj8dtVIStucpv2mIVxVVxOBYBhw8FGNQLcE3mQ0jgcnvnWpPiuhuHtBQ8wAjrmLNxXk+EnMoj+0Yp1/ZfWvArJhRktJYpWa8RTFYnnsZ4VJHQjVK3rlXmCDZ3aacAQweERDlrB4KFDGgWbpGJbYiDrkCaVtnVaQbRKk0vwDpu97liQiSY1IGF6fRU7342LvOn6UCJ2xZcesP1yNEaUWg7yyTdEmKm9tMX0Tt7XIvwGvF/4WQ/pA3BFOovwjCCJwPmf0bM2Xu4p61bc3nmbbp+0Lg8J2ExfnsSYMPfWs001C256flejgcASIZFJHG2+l+Yoi7F4Rk/4fV6aAvLj57qdA/2eFaONIuU2t4irk6My/jNIHLIto76k6 iNPploOO 6t90KmYKCx8RZ6Mj/w9nmqiy7Y2AxXEIv3gxM/dSWDwEP8lifNOXOk79qJFBi9HLhnaGKnI3s9TzNslLYAdnZB6tbXUuWaMKfNJfa2zYkHrTIHAps4oRMdeNBgGpx2VCpicUsFe+C0YmXf2zIvg+7jkX+dOrLNtaqkFyz2kG0RR2Lyw9nyL+AC40DAMtedt/Rb43gBQI65eAHmqtKG06uyR3BqpNIfS9txELy4A4BR74vmuIJYbCdAzkUvbdA3Ia0FLfw2qDapTYSqUGJMQXA/lKE4Eq6UCXJxW4+aW+znAvdsks6KRMq8H4MBHU0AFydElsXdarD7jCvm441jKiQhgABJO0nNZd38w/U8I1kcilov3yMJQAuCwhl/+E0A/++08W5UUK+Fur5X5MevrVd9x9yhU/S733qLY+niApSoBDIjFqeIzM6QPXiMNVf1TIMsZOYpi2kGXfJr5rn5ksXGOdt6G2uYCl+1sJHaUfMjF8ndUVEtEVTpP3Q2ZTyXS2GkJk8vLzmmFbEnzQp2kvS03BJAjfpyz2YqugzWoR2DZwTljmzY50uKWoiyHodbCol6FJnP5LVEzKXVVC0wWYJjsGejG9fnmdhO21i X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov LLVM generates rX = bpf_cast_kern/_user(rY, address_space) instructions when pointers in non-zero address space are used by the bpf program. Signed-off-by: Alexei Starovoitov --- include/uapi/linux/bpf.h | 5 +++++ kernel/bpf/disasm.c | 11 +++++++++++ tools/include/uapi/linux/bpf.h | 5 +++++ 3 files changed, 21 insertions(+) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index f6648851eae6..3de1581379d4 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1313,6 +1313,11 @@ enum { */ #define BPF_PSEUDO_KFUNC_CALL 2 +enum bpf_arena_cast_kinds { + BPF_ARENA_CAST_KERN = 1, + BPF_ARENA_CAST_USER = 2, +}; + /* flags for BPF_MAP_UPDATE_ELEM command */ enum { BPF_ANY = 0, /* create new element or update existing */ diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 49940c26a227..37d9b37b34f7 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -166,6 +166,12 @@ static bool is_movsx(const struct bpf_insn *insn) (insn->off == 8 || insn->off == 16 || insn->off == 32); } +static bool is_arena_cast(const struct bpf_insn *insn) +{ + return insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && + (insn->off == BPF_ARENA_CAST_KERN || insn->off == BPF_ARENA_CAST_USER); +} + void print_bpf_insn(const struct bpf_insn_cbs *cbs, const struct bpf_insn *insn, bool allow_ptr_leaks) @@ -184,6 +190,11 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->code, class == BPF_ALU ? 'w' : 'r', insn->dst_reg, class == BPF_ALU ? 'w' : 'r', insn->dst_reg); + } else if (is_arena_cast(insn)) { + verbose(cbs->private_data, "(%02x) r%d = cast_%s(r%d, %d)\n", + insn->code, insn->dst_reg, + insn->off == BPF_ARENA_CAST_KERN ? "kern" : "user", + insn->src_reg, insn->imm); } else if (BPF_SRC(insn->code) == BPF_X) { verbose(cbs->private_data, "(%02x) %c%d %s %s%c%d\n", insn->code, class == BPF_ALU ? 'w' : 'r', diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index f6648851eae6..3de1581379d4 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1313,6 +1313,11 @@ enum { */ #define BPF_PSEUDO_KFUNC_CALL 2 +enum bpf_arena_cast_kinds { + BPF_ARENA_CAST_KERN = 1, + BPF_ARENA_CAST_USER = 2, +}; + /* flags for BPF_MAP_UPDATE_ELEM command */ enum { BPF_ANY = 0, /* create new element or update existing */ From patchwork Tue Feb 6 22:04:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77A1DC48297 for ; Tue, 6 Feb 2024 22:05:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 100396B0074; Tue, 6 Feb 2024 17:05:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 089E26B0075; Tue, 6 Feb 2024 17:05:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1F2F6B0093; Tue, 6 Feb 2024 17:05:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB09C6B0074 for ; Tue, 6 Feb 2024 17:05:12 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 70F39160184 for ; Tue, 6 Feb 2024 22:05:12 +0000 (UTC) X-FDA: 81762760464.22.943A6A8 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf02.hostedemail.com (Postfix) with ESMTP id 9859E80026 for ; Tue, 6 Feb 2024 22:05:10 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Bhr0Cn2z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257110; a=rsa-sha256; cv=none; b=22XdtpHb7zO+Le04CmujEBa0lLpcuxYbud0nQEu+zsyChSf3hUhpb2vaDA1WcHOlvQzgxF DB9IInp1vsa8LYgPZVhQLsQ1P6iU3ovVMMfxyX6noHbmMkM/oP9IELxQaHFyHJccFwhfaY yXcmmCnVXgvgIJlqu8Es+x65ECxmqZw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Bhr0Cn2z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257110; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6enFVSH9RvXWbj69sDWzp9NqEgt7n2TNQMLPChkoghg=; b=ql6Bj070WB5AYIa78q8EaaFbjUIRoXuS20YZIpExv7jIzCY0uS/kn/fhm52MqjhwT70/kE tL6iMwC7pjgbYzkvxyZYuC6fS3IcvYdYHSl8dovSM+Ipi4OaZsC2d7IPBTAdlCILvOaYYc LO4cApIvYLCp2w+W24EbJwnVnvf/2jY= Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-5dbcfa0eb5dso1357036a12.3 for ; Tue, 06 Feb 2024 14:05:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257109; x=1707861909; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6enFVSH9RvXWbj69sDWzp9NqEgt7n2TNQMLPChkoghg=; b=Bhr0Cn2zFMeKHXmEWC38wqoGFW6YXvLdWWDtFTx9XS5OxRNfihdkL5ZwLOeDVMXZmy hfwwY4n97g8Tb/UEMDkeq40RdYs9JHwkQbUzN24Cnn6Yjb3X5AUuiUaX/y8D57/FMycP zcZsJ9V+c64hu36ywLDSGzx5EhcHFRzVXcvpXMZe6SwGyufBGP70dx4JTsRaQTL1h3qc FAakiYP7Kup12dK3awhqB7lABFByT5xCwo3VRDzbEJ4JapPVGj0ZkkYnMWz9UzmTh29D xvMFsuC1DnBkx7yFaBcrOtfHwipZ7SZhoDRM5wwXl1pf+1mFB7kJJb4/7Q8bcNsEGF3t pBdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257109; x=1707861909; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6enFVSH9RvXWbj69sDWzp9NqEgt7n2TNQMLPChkoghg=; b=renYl4L92LAjIcZabmyTXzVbnoAQTW579+nvmL2xemfNJqVIspiQl1F5fy3rUC+Vwt o+yDcgrwOquwJP2C6Mm+OFE+89fpGwex0JJvIxwV4t9lnKTLllgb3Pa3YbSkhC3TDL6c PYNq1ct9H+FQDzPFWVmja0ax5LdqJRaialRoah36pW3LqwIfPHpU4/qEP8PrtcsW2Txe sYtQRY57eaiKOtpOi7zw6JaN9R3DdSufF8uiwG8qnn3MepOskQVmMiOZFNQMv9RWRJmy p+GkMb77lUMlDjV2pktxAmV/FFgSJ1YS7ERrhUganRsCWRSoDE2OlzPVy92onGJBf7s4 8hzg== X-Gm-Message-State: AOJu0YwFGoKovNKQR5ndq3aUoAMjR9o5NnecPlBQq5M/7zluzhGPORI2 nq9M2i+bClrhd8YB2Mj4Jl7+GGCnApkbLbAYCyhoSRdsSSbg7gFp X-Google-Smtp-Source: AGHT+IFJTMJzd37fFoOyeiC8Vlz2bibOaLXp0NCvLZHx/AUz1x1BcqEdisFhlYf4KrANb3jkuOV5FQ== X-Received: by 2002:a17:90a:fc82:b0:296:a70f:e96b with SMTP id ci2-20020a17090afc8200b00296a70fe96bmr810904pjb.46.1707257109520; Tue, 06 Feb 2024 14:05:09 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWTlE2SqXoxgbHIhq0G2pAApG/U2NZcA5v2musiC9iAqVjKj1tmWN0x2jXiNzl+bSHzBLchN9EcCVA65jMPupfxr+IPyW3uB1BbgGJucFcciAKDPSma9ckJXdxkGwr5dryD0TmNxrXOI2eGFoxgjmbe1V9Glj5vQ9bUH+pRBiCH+njQ0ko5+lmzLDx4II82tzdKWQNT8vvskJIIsUrU5j8iu7kXOy+/2GbiZmkj3OH58FJns8wJnBYFa+KhP1PyQYlDzNsXXw9Uq3cD256jC+iitZhaf55/mvjP Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id pd3-20020a17090b1dc300b00290f9e8b4f9sm2275950pjb.46.2024.02.06.14.05.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:09 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 06/16] bpf: Add x86-64 JIT support for PROBE_MEM32 pseudo instructions. Date: Tue, 6 Feb 2024 14:04:31 -0800 Message-Id: <20240206220441.38311-7-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9859E80026 X-Stat-Signature: 6pnoqzcqy4itpzgxmbqwuw34udua78ag X-HE-Tag: 1707257110-663337 X-HE-Meta: U2FsdGVkX1+7xF1PYANcNEhbSPA8NEEOS8ZCDGbjGXFN2bWe+VDRjZ9kBj8sQ3vmLVFbK5H3Pny7UMi4Pe1T6gtWubkl0gS/q5x9YYG6dYV1lrKUpb9dD2kXdL1RZvAqK86T4plVo7xCFYpADe3pXLPEUMV2koXajmuAK3c0PJhZME3U/p1fWCdDPt1YBz9tGlDUE9ta4Acbto3mAHPZ3qlKDTOl+0tJdGMdQBZp97fdGLkGg2Bi89rnMlXfcBZKjccMsuhqyfaW3c7n2Tkh2FyYkSI3UrBxJG/CpwsjFIs+7RdH9rGvOYwFhoYKVCLhVP9PkshOKfMqteD3QvxK8FpezP7Yz59v4opY1LZzJ1fjUNgXjM7/7uxe5F24zZXS2Tw3IpUVIcPUQhb+OYq7127q6B50XaKJxmUAeGdDsLWkX/rxdEj32a3ZaihRaCAE75WWE490/xwTk6YKU0tdDp5odxVLYh0dORpb0Xqsg1Rq97Kv7zurX4OHlyeNrJ/RfqaRONGXsMZPsBAwTBkEgYwUq59la20ewoqbD2DSAbCfxKodeuIjXOl+ZmZC/0YAgnGuPpjlx53KbCZyIsjRQxVozICsk5WOSdjgEhnP91utv87izTMo0V3pJ7LVHABLZv5VVKFdQAg3zSbIB5Ds+IlyiJCWM4CJjewjnoEWQtX1N1kjJ/1t5naYWulYtpOjNk+XI6FZXyRN1FlFtisdHNr0ivVwmltwc6Iq1v6644t0dh2FX4ZlvuDcsc/ss1fvEwPIldhyIB+m2s6ZOYhsiS1khWcLkM/JV93IBY3O9Bc8BMJMZ1NbQRrVXUt31Rw4kgSY18jFVDMEAWIkSASMRZhfRXA/vUL7uQzYdihaWYHm95yjsueq4yr//pq5BPy46VjP9KtiNImMy8zPztO5qcJ0YG0rCdvaA8aMElR4OYLaX9NPUgSvU6aGAC3V7lAHrLoToPtEsKHH4D2w1cr giv0C9Vu tNTCvN0+ocHm0NrNJvXy6Pc3OlOx+3a2EoSTFSUmNnRhnJ4U3U5k1uoLxca4AtIAlaveBYm8x7tHC7XVQAvmboijn6YBTMWCBCASxE2S8sU74iVn8KSrS0Zb2GIlh6JW4jznZUtpk2KW8cjQHLlFAQlMvkF2kXwS1knKyXF34IPg0RNoTQjpjwB2Wg74UUpHLC7b1z1YPtQ7jq2AVtDd85riDm/yE44GcWefFIFrI383vRABU3NmBY5FrD2zyaS8A6cYFRWFl9wMp71xKm3m2+qdGPJIMBcEeXk1/QvfxRxNBp9mAv+bVyusYuXnvHRA/etvBWsGyWImNRZqwvwj4ispYbWRBChoawhQOt3rS9yFZv8VAV8IxFSGHipQH5lxOCwnJq59m1zuRMbNEja+1FPqi2iQb6NPwAqvVfTMxaTpPrEqeNxGfxhqrV+tGRFXQ8E2QbApr0AMSZOs+X6c6vPblPPXH+ZLJNfL5dorEEhw2uHalt20gidVswgORP2shDlkFLj5ut4AjK5WfgxFCM/VycRcOXHzOa7JGByrAMWcWQ+Qycjq7OZ/jJ/CF5sVJRI86wwb+9DWSmegHjRxSp7thlLOSJws5Sbqg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences: - PROBE_MEM has to check that the address is in the kernel range with src_reg + insn->off >= TASK_SIZE_MAX + PAGE_SIZE check - PROBE_MEM doesn't support store - PROBE_MEM32 relies on the verifier to clear upper 32-bit in the register - PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in %r12 in the prologue) Due to bpf_arena constructions such %r12 + %reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time. - PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed. Signed-off-by: Alexei Starovoitov --- arch/x86/net/bpf_jit_comp.c | 183 +++++++++++++++++++++++++++++++++++- include/linux/bpf.h | 1 + include/linux/filter.h | 3 + 3 files changed, 186 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e1390d1e331b..883b7f604b9a 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -113,6 +113,7 @@ static int bpf_size_to_x86_bytes(int bpf_size) /* Pick a register outside of BPF range for JIT internal work */ #define AUX_REG (MAX_BPF_JIT_REG + 1) #define X86_REG_R9 (MAX_BPF_JIT_REG + 2) +#define X86_REG_R12 (MAX_BPF_JIT_REG + 3) /* * The following table maps BPF registers to x86-64 registers. @@ -139,6 +140,7 @@ static const int reg2hex[] = { [BPF_REG_AX] = 2, /* R10 temp register */ [AUX_REG] = 3, /* R11 temp register */ [X86_REG_R9] = 1, /* R9 register, 6th function argument */ + [X86_REG_R12] = 4, /* R12 callee saved */ }; static const int reg2pt_regs[] = { @@ -167,6 +169,7 @@ static bool is_ereg(u32 reg) BIT(BPF_REG_8) | BIT(BPF_REG_9) | BIT(X86_REG_R9) | + BIT(X86_REG_R12) | BIT(BPF_REG_AX)); } @@ -205,6 +208,17 @@ static u8 add_2mod(u8 byte, u32 r1, u32 r2) return byte; } +static u8 add_3mod(u8 byte, u32 r1, u32 r2, u32 index) +{ + if (is_ereg(r1)) + byte |= 1; + if (is_ereg(index)) + byte |= 2; + if (is_ereg(r2)) + byte |= 4; + return byte; +} + /* Encode 'dst_reg' register into x86-64 opcode 'byte' */ static u8 add_1reg(u8 byte, u32 dst_reg) { @@ -887,6 +901,18 @@ static void emit_insn_suffix(u8 **pprog, u32 ptr_reg, u32 val_reg, int off) *pprog = prog; } +static void emit_insn_suffix_SIB(u8 **pprog, u32 ptr_reg, u32 val_reg, u32 index_reg, int off) +{ + u8 *prog = *pprog; + + if (is_imm8(off)) { + EMIT3(add_2reg(0x44, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off); + } else { + EMIT2_off32(add_2reg(0x84, BPF_REG_0, val_reg), add_2reg(0, ptr_reg, index_reg) /* SIB */, off); + } + *pprog = prog; +} + /* * Emit a REX byte if it will be necessary to address these registers */ @@ -968,6 +994,37 @@ static void emit_ldsx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +static void emit_ldx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +{ + u8 *prog = *pprog; + + switch (size) { + case BPF_B: + /* movzx rax, byte ptr [rax + r12 + off] */ + EMIT3(add_3mod(0x40, src_reg, dst_reg, index_reg), 0x0F, 0xB6); + break; + case BPF_H: + /* movzx rax, word ptr [rax + r12 + off] */ + EMIT3(add_3mod(0x40, src_reg, dst_reg, index_reg), 0x0F, 0xB7); + break; + case BPF_W: + /* mov eax, dword ptr [rax + r12 + off] */ + EMIT2(add_3mod(0x40, src_reg, dst_reg, index_reg), 0x8B); + break; + case BPF_DW: + /* mov rax, qword ptr [rax + r12 + off] */ + EMIT2(add_3mod(0x48, src_reg, dst_reg, index_reg), 0x8B); + break; + } + emit_insn_suffix_SIB(&prog, src_reg, dst_reg, index_reg, off); + *pprog = prog; +} + +static void emit_ldx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +{ + emit_ldx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); +} + /* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) { @@ -1002,6 +1059,71 @@ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* STX: *(u8*)(dst_reg + index_reg + off) = src_reg */ +static void emit_stx_index(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, u32 index_reg, int off) +{ + u8 *prog = *pprog; + + switch (size) { + case BPF_B: + /* mov byte ptr [rax + r12 + off], al */ + EMIT2(add_3mod(0x40, dst_reg, src_reg, index_reg), 0x88); + break; + case BPF_H: + /* mov word ptr [rax + r12 + off], ax */ + EMIT3(0x66, add_3mod(0x40, dst_reg, src_reg, index_reg), 0x89); + break; + case BPF_W: + /* mov dword ptr [rax + r12 + 1], eax */ + EMIT2(add_3mod(0x40, dst_reg, src_reg, index_reg), 0x89); + break; + case BPF_DW: + /* mov qword ptr [rax + r12 + 1], rax */ + EMIT2(add_3mod(0x48, dst_reg, src_reg, index_reg), 0x89); + break; + } + emit_insn_suffix_SIB(&prog, dst_reg, src_reg, index_reg, off); + *pprog = prog; +} + +static void emit_stx_r12(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) +{ + emit_stx_index(pprog, size, dst_reg, src_reg, X86_REG_R12, off); +} + +/* ST: *(u8*)(dst_reg + index_reg + off) = imm32 */ +static void emit_st_index(u8 **pprog, u32 size, u32 dst_reg, u32 index_reg, int off, int imm) +{ + u8 *prog = *pprog; + + switch (size) { + case BPF_B: + /* mov byte ptr [rax + r12 + off], imm8 */ + EMIT2(add_3mod(0x40, dst_reg, 0, index_reg), 0xC6); + break; + case BPF_H: + /* mov word ptr [rax + r12 + off], imm16 */ + EMIT3(0x66, add_3mod(0x40, dst_reg, 0, index_reg), 0xC7); + break; + case BPF_W: + /* mov dword ptr [rax + r12 + 1], imm32 */ + EMIT2(add_3mod(0x40, dst_reg, 0, index_reg), 0xC7); + break; + case BPF_DW: + /* mov qword ptr [rax + r12 + 1], imm32 */ + EMIT2(add_3mod(0x48, dst_reg, 0, index_reg), 0xC7); + break; + } + emit_insn_suffix_SIB(&prog, dst_reg, 0, index_reg, off); + EMIT(imm, bpf_size_to_x86_bytes(size)); + *pprog = prog; +} + +static void emit_st_r12(u8 **pprog, u32 size, u32 dst_reg, int off, int imm) +{ + emit_st_index(pprog, size, dst_reg, X86_REG_R12, off, imm); +} + static int emit_atomic(u8 **pprog, u8 atomic_op, u32 dst_reg, u32 src_reg, s16 off, u8 bpf_size) { @@ -1043,12 +1165,15 @@ static int emit_atomic(u8 **pprog, u8 atomic_op, return 0; } +#define DONT_CLEAR 1 + bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs) { u32 reg = x->fixup >> 8; /* jump over faulting load and clear dest register */ - *(unsigned long *)((void *)regs + reg) = 0; + if (reg != DONT_CLEAR) + *(unsigned long *)((void *)regs + reg) = 0; regs->ip += x->fixup & 0xff; return true; } @@ -1147,11 +1272,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image bool tail_call_seen = false; bool seen_exit = false; u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; + u64 arena_vm_start; int i, excnt = 0; int ilen, proglen = 0; u8 *prog = temp; int err; + arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); + detect_reg_usage(insn, insn_cnt, callee_regs_used, &tail_call_seen); @@ -1172,8 +1300,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image push_r12(&prog); push_callee_regs(&prog, all_callee_regs_used); } else { + if (arena_vm_start) + push_r12(&prog); push_callee_regs(&prog, callee_regs_used); } + if (arena_vm_start) + emit_mov_imm64(&prog, X86_REG_R12, + arena_vm_start >> 32, (u32) arena_vm_start); ilen = prog - temp; if (rw_image) @@ -1564,6 +1697,52 @@ st: if (is_imm8(insn->off)) emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); break; + case BPF_ST | BPF_PROBE_MEM32 | BPF_B: + case BPF_ST | BPF_PROBE_MEM32 | BPF_H: + case BPF_ST | BPF_PROBE_MEM32 | BPF_W: + case BPF_ST | BPF_PROBE_MEM32 | BPF_DW: + start_of_ldx = prog; + emit_st_r12(&prog, BPF_SIZE(insn->code), dst_reg, insn->off, insn->imm); + goto populate_extable; + + /* LDX: dst_reg = *(u8*)(src_reg + r12 + off) */ + case BPF_LDX | BPF_PROBE_MEM32 | BPF_B: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_H: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_W: + case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW: + case BPF_STX | BPF_PROBE_MEM32 | BPF_B: + case BPF_STX | BPF_PROBE_MEM32 | BPF_H: + case BPF_STX | BPF_PROBE_MEM32 | BPF_W: + case BPF_STX | BPF_PROBE_MEM32 | BPF_DW: + start_of_ldx = prog; + if (BPF_CLASS(insn->code) == BPF_LDX) + emit_ldx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); + else + emit_stx_r12(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); +populate_extable: + { + struct exception_table_entry *ex; + u8 *_insn = image + proglen + (start_of_ldx - temp); + s64 delta; + + if (!bpf_prog->aux->extable) + break; + + ex = &bpf_prog->aux->extable[excnt++]; + + delta = _insn - (u8 *)&ex->insn; + /* switch ex to rw buffer for writes */ + ex = (void *)rw_image + ((void *)ex - (void *)image); + + ex->insn = delta; + + ex->data = EX_TYPE_BPF; + + ex->fixup = (prog - start_of_ldx) | + ((BPF_CLASS(insn->code) == BPF_LDX ? reg2pt_regs[dst_reg] : DONT_CLEAR) << 8); + } + break; + /* LDX: dst_reg = *(u8*)(src_reg + off) */ case BPF_LDX | BPF_MEM | BPF_B: case BPF_LDX | BPF_PROBE_MEM | BPF_B: @@ -2036,6 +2215,8 @@ st: if (is_imm8(insn->off)) pop_r12(&prog); } else { pop_callee_regs(&prog, callee_regs_used); + if (arena_vm_start) + pop_r12(&prog); } EMIT1(0xC9); /* leave */ emit_return(&prog, image + addrs[i - 1] + (prog - temp)); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 42f22bc881f0..a0d737bb86d1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1460,6 +1460,7 @@ struct bpf_prog_aux { bool xdp_has_frags; bool exception_cb; bool exception_boundary; + struct bpf_arena *arena; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/include/linux/filter.h b/include/linux/filter.h index fee070b9826e..cd76d43412d0 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -72,6 +72,9 @@ struct ctl_table_header; /* unused opcode to mark special ldsx instruction. Same as BPF_IND */ #define BPF_PROBE_MEMSX 0x40 +/* unused opcode to mark special load instruction. Same as BPF_MSH */ +#define BPF_PROBE_MEM32 0xa0 + /* unused opcode to mark call to interpreter with arguments */ #define BPF_CALL_ARGS 0xe0 From patchwork Tue Feb 6 22:04:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AA5AC48297 for ; Tue, 6 Feb 2024 22:05:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEE276B0075; Tue, 6 Feb 2024 17:05:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A9BDB6B0093; Tue, 6 Feb 2024 17:05:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EE4B6B0095; Tue, 6 Feb 2024 17:05:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 769DD6B0075 for ; Tue, 6 Feb 2024 17:05:16 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 451E1A152C for ; Tue, 6 Feb 2024 22:05:16 +0000 (UTC) X-FDA: 81762760632.13.8C2FA4A Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf05.hostedemail.com (Postfix) with ESMTP id 6E845100003 for ; Tue, 6 Feb 2024 22:05:14 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FSVIF6Xd; spf=pass (imf05.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257114; a=rsa-sha256; cv=none; b=eklufzEy92+zVAoOfRCAOsXFCp9TOXTItD8b0qds9R1SE0v05RZwdd22cpt2cNf2zoG8+t iXp09GUc65UgxuZjvWjec9nhTcIzIOKDCw5gGpBwvc5TfQAPgp6+emWZOj7yV2J+SLAdT6 /PjcoU0DJ0nGBmIaI95AADSMyht5UTA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FSVIF6Xd; spf=pass (imf05.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257114; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GxmcZQXB6hILUuERStuFRZ0MPT4uFMATBLnu6rXBqLE=; b=FidUVpO/AedMg6KAjiXK4sWxORQ/hlXabCPWGhiCxmWgP7m08FN45tEcNuxWg00YiWEyg0 F6GzfTCrrOyFprLhaKSojWgdPo4taJqJ+dausB7SAMLmaPqmlr+5s+gysrvKzh0mWY6hQw YSVbt8U5awr5WkZIXJzlQnrMoMKaXxo= Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-296dcd75be7so272515a91.2 for ; Tue, 06 Feb 2024 14:05:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257113; x=1707861913; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GxmcZQXB6hILUuERStuFRZ0MPT4uFMATBLnu6rXBqLE=; b=FSVIF6XdUmP5EhTnYRwG7lEwIDgRdLf0sHK+7c0PH4w5gnPlDP/XklF/fFz3Xy0VFl 5OFuwzvZr9sm33I4WLA87yRpx0GhTrbFENsDnIZrvR5xmlePvJ75UWR2VkNQVRPyc/nZ J1oHpGuu4jgRBRS/BpFZfMG+74dXWqLR69d3fF1BWB4B9F1dAmP9dPCSy5tqPSKqQsrL YUddlJOPObK1Sn4JqWY0dWCY2u6vKt28S/UaKEV2FOGUlZljV2hxHZriDDfwQL57k2Ve HcXD9a2rziWLeAV5CRcX8T5lUmQQISJtpV4ZVpV/aCf6zHO7jnauUCMQieNRGcVSrLm/ c3SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257113; x=1707861913; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GxmcZQXB6hILUuERStuFRZ0MPT4uFMATBLnu6rXBqLE=; b=ZSP/M3XT/XE/kE0PrHBtdwJIAxKJPMAxQ7NIxaU/L+uyIkLL/yhKnzIUvnD4oIrtG2 M1vDT7boipqDxKhOBZkeY4vOaSTjjQ2fn0YeFo9lDm08yVea8nP+uYDLqQg4Y95ViLTL TU4zCTSQO1yPBO7HNMmWNqqTe5hN3siZHgkMy8u3WRRIOcojBryPe2UQ4TDpcQix2Kg/ hF38A26J/2ujFWkNtPYWDRj5WVL1mY5TyY2vqqVdYJBfhqGI7Ku13wDppKm/36icsvW6 W3U7b9VMtr9BjqKFlrs3FXaIUbMCncds0bHCan04ioOoPbFserzw1tDV3aWfHIjxZ6oz W6sA== X-Gm-Message-State: AOJu0YzL0el2UOb1Z3wmYkh/IQ6lDOJAjTe74K9drJeBrtDLA5KHcaTH +Y8EMeMVL0C/2/yGHb36aOYrDuts8VS5l9GqFQZqpKDeGtU6igad X-Google-Smtp-Source: AGHT+IG8oAHzR3nsr++sOa64aFqWnOY2L1+MMxXsvlNE4aY5ZiS0oyor0jQVKD0uoMrje1RiSvSdsA== X-Received: by 2002:a17:90b:350c:b0:296:c1a:f651 with SMTP id ls12-20020a17090b350c00b002960c1af651mr884749pjb.36.1707257113308; Tue, 06 Feb 2024 14:05:13 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXfwYB7FRrQvWfTk6tEr+uUBGbmaCq7+V+6esU7E5tJSAj1KK9tCNvR2xsEZBB081Xgy46YpR2sEOIbwIiK/1CsFdSNyQsZddCFwiLLxjRVFRaNhwuCFDq191OQ8uonJA+2UerpQK+JRy9zZAlFHa1wanOWMMIS+Hq7SpgXQH9IqaaVBQYUqYpa2d39hQfaUwO1B6BULxhIfZ+C2s4IrW++qdX5AoMMGyC9BAWf+AY6fWAWyCtugRHgQWZ74iaZkRKe4wMe8lkrZkmmvR1EPCVE9kBbPDe3DnDo Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id li4-20020a17090b48c400b00296b2f99946sm7168pjb.30.2024.02.06.14.05.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:12 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 07/16] bpf: Add x86-64 JIT support for bpf_cast_user instruction. Date: Tue, 6 Feb 2024 14:04:32 -0800 Message-Id: <20240206220441.38311-8-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6E845100003 X-Stat-Signature: 3m49ue3uwijn9ubkf5xq3g9dufbj5ibr X-Rspam-User: X-HE-Tag: 1707257114-53281 X-HE-Meta: U2FsdGVkX19n9dtASjHFPKf+6TY1T4KbSsZCg3LQbCuf5RXQzowuraEu0tBuE1L1Cvlfe1bl+gI2ZdgojGck1QC6A2wBrHg21GK46Q4PIh98B6N4K7jCgJfFAAElOt/5alMcC0BXxTVAEwqd1I2CzXDCFD67KuQAEAlHt9mlIon86O8rbttnakYrcJCnA1+I0HPhiv9+leeurmn415hC+2vd1hM928BMVAEm119mnLrjbIzTKZiUHsU4GsWXuMt7uCshUNsSPpTJzvkq7+GAottNC30zt7BM4DzrCnJC1uz/TWYL0Etb4TrebnCjDZegtxZHyCGd45raveg2Qn6PigJjkD+YX1Hca5tXUOpc4feUhZybRcH3QB8DRPNZGvsQR7nwyjgOb+A/JBk7EciC6IBAbU3xK2wJIZ8GlHu267WZNOqAYcgEZKU7pZev3DDZLFOLFAQ6jacX56SDbHqteNqIR6DdwJEGLsMStYL9upB2ARpbgcpjqUzoiFApuUKjkTWdyE5OZcAUg2C+zQ3N8aulstWe+7hRsId/zYRI+VYJpp8odpY3whwnLvekaCZVSbA2zajzz/8gSIELCCoNt5oCbheJuHaqeaDE2GHySSvlE0y+eOeIg/lfZGBCopQ3lmNz3BMzysJzCutLWm1/DJ+GN+blRJL9r5uufu94bNFm6r7fGFOGYQOTf8lHcIsNHy+r6KPyoQqBOcfdI+/zTma8dlrpWXY4iYDcXYCwb8OQlraPviFLcnq7ADI3UFGHtSZGGToM5iRBFWt/35MQdWfsg/jGmlEYIfcj6PXhG920EyNI374y8u/Lz26WCDIHIU2Bcua+tQFQItMN5Ph3tlR4hpwPDxaG7lnz0klX8N3Cw804Ue3+TYtgByopZPhxCk56K15H6Q46+3x4xvZswPOBNdvXu6tUwDCTeo0J9FY/IcvV5twRrkncbUECI1tj+94x2NgmV7Xpg4q1zq2 yUVDX9Vo pdJ17xN3fHJXD6Jgs0XbTKTu6J9muwmTHy9jp2Hrs8wXmd9fQ2vrOA14SwTa6XYUFO3NNHTnyeFiEAfhn1CMfc0hsq03wU1ufQycITSi+5i92R8dnd6Ra2sYzFOmOvI6+6jvIE509Q0C4Ftny+C1vKolWoPtRYK4I452l8o2jlz9dxgPm7ZGMO4hTP4r4RDuDhH1BYvabeYUC2qSTrHYvCzUnxUdifhdfDeLieaWcMJV25xUVNqOsP9XyzP1m8nFVfsyaC/s2NvDmFfQ6jf1NtV0xRHoM/NYu+ScVnDk+oLsaCRj6HZ+M+ja3bYD5gp64vghOuhxDcPbQAx97m4eX0DRKqqqKp1+FKPs7ML4/X03+DAyRK9Qq3WKwQvavVcTCxgbHYwi9XakYYN00HSHYlaxBfAvhHSGA4h+x8m2G5OjgwLCe8fcv6p+hr/+y7/jcXqI2WiA6PmjrAnYqsfb2tmRNlC92LAAVCdqMyE9XgoFQSbdAvH8nOx4TRNEn3BEnSSqkzEETHErEFZSndsNj4gS87awn3ofk1HQmn+wyg6tkJTvfGlGrdcJi1L0jTkeo8jxe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov LLVM generates bpf_cast_kern and bpf_cast_user instructions while translating pointers with __attribute__((address_space(1))). rX = cast_kern(rY) is processed by the verifier and converted to normal 32-bit move: wX = wY bpf_cast_user has to be converted by JIT. rX = cast_user(rY) is aux_reg = upper_32_bits of arena->user_vm_start aux_reg <<= 32 wX = wY // clear upper 32 bits of dst register if (wX) // if not zero add upper bits of user_vm_start wX |= aux_reg JIT can do it more efficiently: mov dst_reg32, src_reg32 // 32-bit move shl dst_reg, 32 or dst_reg, user_vm_start rol dst_reg, 32 xor r11, r11 test dst_reg32, dst_reg32 // check if lower 32-bit are zero cmove r11, dst_reg // if so, set dst_reg to zero // Intel swapped src/dst register encoding in CMOVcc Signed-off-by: Alexei Starovoitov --- arch/x86/net/bpf_jit_comp.c | 41 ++++++++++++++++++++++++++++++++++++- include/linux/filter.h | 1 + kernel/bpf/core.c | 5 +++++ 3 files changed, 46 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 883b7f604b9a..a042ed57af7b 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1272,13 +1272,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image bool tail_call_seen = false; bool seen_exit = false; u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; - u64 arena_vm_start; + u64 arena_vm_start, user_vm_start; int i, excnt = 0; int ilen, proglen = 0; u8 *prog = temp; int err; arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); + user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena); detect_reg_usage(insn, insn_cnt, callee_regs_used, &tail_call_seen); @@ -1346,6 +1347,39 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image break; case BPF_ALU64 | BPF_MOV | BPF_X: + if (insn->off == BPF_ARENA_CAST_USER) { + if (dst_reg != src_reg) + /* 32-bit mov */ + emit_mov_reg(&prog, false, dst_reg, src_reg); + /* shl dst_reg, 32 */ + maybe_emit_1mod(&prog, dst_reg, true); + EMIT3(0xC1, add_1reg(0xE0, dst_reg), 32); + + /* or dst_reg, user_vm_start */ + maybe_emit_1mod(&prog, dst_reg, true); + if (is_axreg(dst_reg)) + EMIT1_off32(0x0D, user_vm_start >> 32); + else + EMIT2_off32(0x81, add_1reg(0xC8, dst_reg), user_vm_start >> 32); + + /* rol dst_reg, 32 */ + maybe_emit_1mod(&prog, dst_reg, true); + EMIT3(0xC1, add_1reg(0xC0, dst_reg), 32); + + /* xor r11, r11 */ + EMIT3(0x4D, 0x31, 0xDB); + + /* test dst_reg32, dst_reg32; check if lower 32-bit are zero */ + maybe_emit_mod(&prog, dst_reg, dst_reg, false); + EMIT2(0x85, add_2reg(0xC0, dst_reg, dst_reg)); + + /* cmove r11, dst_reg; if so, set dst_reg to zero */ + /* WARNING: Intel swapped src/dst register encoding in CMOVcc !!! */ + maybe_emit_mod(&prog, AUX_REG, dst_reg, true); + EMIT3(0x0F, 0x44, add_2reg(0xC0, AUX_REG, dst_reg)); + break; + } + fallthrough; case BPF_ALU | BPF_MOV | BPF_X: if (insn->off == 0) emit_mov_reg(&prog, @@ -3424,6 +3458,11 @@ void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke, } } +bool bpf_jit_supports_arena(void) +{ + return true; +} + bool bpf_jit_supports_ptr_xchg(void) { return true; diff --git a/include/linux/filter.h b/include/linux/filter.h index cd76d43412d0..78ea63002531 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -959,6 +959,7 @@ bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_far_kfunc_call(void); bool bpf_jit_supports_exceptions(void); bool bpf_jit_supports_ptr_xchg(void); +bool bpf_jit_supports_arena(void); void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie); bool bpf_helper_changes_pkt_data(void *func); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2539d9bfe369..2829077f0461 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2926,6 +2926,11 @@ bool __weak bpf_jit_supports_far_kfunc_call(void) return false; } +bool __weak bpf_jit_supports_arena(void) +{ + return false; +} + /* Return TRUE if the JIT backend satisfies the following two conditions: * 1) JIT backend supports atomic_xchg() on pointer-sized words. * 2) Under the specific arch, the implementation of xchg() is the same From patchwork Tue Feb 6 22:04:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E67A4C48297 for ; Tue, 6 Feb 2024 22:05:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 768826B0095; Tue, 6 Feb 2024 17:05:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 718516B0096; Tue, 6 Feb 2024 17:05:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B9566B0098; Tue, 6 Feb 2024 17:05:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4573D6B0095 for ; Tue, 6 Feb 2024 17:05:20 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 165AC120879 for ; Tue, 6 Feb 2024 22:05:20 +0000 (UTC) X-FDA: 81762760800.09.0F9659F Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf25.hostedemail.com (Postfix) with ESMTP id 3BFFFA0010 for ; Tue, 6 Feb 2024 22:05:18 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ECdynRiR; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257118; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x70vLSLuXpXSToMN/uastZH05CPVL6I6J8RYkA3WTV8=; b=B6ERo7F/0DZZxWSbQ6Vui9AHWOObDcKEm0K6ffaiwEadfVyFWZoYkKxiEp1Xy5vzq50fci +4mIcMxvuafguAF0hdn7B89EVECSFpFrtizmTjmvjF8hEKaItmEjQAlWnIyFZTtwGqkMs7 weGtJuUPmWJq7RfUmx3ZNLYISWXX17k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257118; a=rsa-sha256; cv=none; b=6l0xUfiQkaEWypyzPRzY2A6mTi7AZJn5PFkYTumQzQ7VJVQuQS//Tr2KfS026iZ5Hj3HVG LNdLwxbor8xfAsVlLL60LikLYlUH3+ACDGzelImccfeqODnh7RwQD/k7L3X3SzXnD5IdZK 42ZDbfXwpf6Cr59FPrCjWRje6HAwTJc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ECdynRiR; spf=pass (imf25.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6e05d958b61so7915b3a.2 for ; Tue, 06 Feb 2024 14:05:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257117; x=1707861917; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x70vLSLuXpXSToMN/uastZH05CPVL6I6J8RYkA3WTV8=; b=ECdynRiRFRobrMk11pGcxtG0QtpzTXdZqJ0HzqOBLgx2fcuv/e1blcnv9F0uNFXP9/ dqqIZxJEQVzNtklvToZJKgSlumCSmdmjaBgnH3hepEWlSbbqPXuBcxdl9DbzbEkrHhkD 8/UUg2Oy5GbJMZl7H+yb/WS2KIFFMmiBG6BOfOdYCyeVyMi3Lxd08NhywzmULMv+GBnP lgN9IWRtvQ9yWDrUW68nzO4KwOIaFvcorJtzJDSHn9d8OJqxDctnasS2+6laxCBGH8Fk ZvMvS21eR7r9kZZAv4nkRInNCKNKx+j/PHnQlN6kaEtr1tXSwtrtxkDtvQjuVi3hgIsN Vhfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257117; x=1707861917; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x70vLSLuXpXSToMN/uastZH05CPVL6I6J8RYkA3WTV8=; b=FK8Ye9Ge/vN3M3dy5zkarOxidBBumBXwlnTBC72NAQdM7F0MnJXpupaE592pkcffk7 zv8v6wA2Elf/MeaDDLFxhEVorbJ/7a4rnpeZb/nubYxwceuxMFQATyX2BwrUC6k4CC5q IoNexn08nIlL8PXlncJKsR41/2PkYDDSjl70bpVAVRK4JtX7Hi9DhDfEm9fynpoYEDBn itpWph1tCHoHyNfzL1ZO4PazQXA5xAG7K3RhMdhx0WXFjVrry6+hn2WPjggD1iA4rAEP 3z3sg5xCx4j7Lr5d1VsNPeoEyqCXDzCZOKV4pFfKVr6lM9gLbSjY2EDFCcJgOUcnNyBG Q9uQ== X-Gm-Message-State: AOJu0YzaACxSfPU8eOojo17mKJkZHdCKO17hNuQ819E2uMhD2pIRfMV3 1oCKgil6WNjon6jjy2lAYoM5vZ9aZZA80terg99so+wDIKB2ldJn X-Google-Smtp-Source: AGHT+IEzPOfG6Yk1/DlYA0HQAsLNnAeAbJiYKnYVhkz+oFjGy38tTVOSA4WEfIoEPimN4Ul29wfrSg== X-Received: by 2002:a62:e807:0:b0:6de:40e:65a3 with SMTP id c7-20020a62e807000000b006de040e65a3mr834468pfi.16.1707257117133; Tue, 06 Feb 2024 14:05:17 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCU1D19B7Hqb61Kyd2+Fvj5kDQspG3/YW/fJlehTD32oKpMmLdiL77sc0UopdB4RBsoXI9kowCg9Cnov8RwV43zLduFGVuvX6yZl8S6HcHqledsiBtXT3bpKw/GTMsaNbtV4bdRY6yq6CzwY5YsdlEO6vvoKEVhMcejghIFGRFTM7Ul0iQBc/fBoCI39YuCHEA9V5wa7G9k0+0lIS2JEyLnad8uR9ktURW/yhgXJJc4/CTw0cK+b8wA47kg3DaAFHICVILo2nlpQyF162HLFXhp3nNGsQp4v3BJ0 Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id w20-20020aa78594000000b006e046c04b81sm2555282pfn.147.2024.02.06.14.05.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:16 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 08/16] bpf: Recognize cast_kern/user instructions in the verifier. Date: Tue, 6 Feb 2024 14:04:33 -0800 Message-Id: <20240206220441.38311-9-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 48hq6ugbq7617srq3jr5mrhu9ufqoptx X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3BFFFA0010 X-Rspam-User: X-HE-Tag: 1707257118-276115 X-HE-Meta: U2FsdGVkX1/Zc9VbjofuNOpPmi1w1Q+W4Vsw7os0G6FopiHiFz0je/K7UkEe3Tzu9VMnLNHUrORdsasyCKkX3jkZJqcHNrUlHvjtQt/iQXjG2FSJ8rSRZQYa63JnLaxntLo1fXBmHRxiEuNDVeKyR3hfP0/JYpLDEOP7lf7+IK+LaXgFdgsWs9y+JukxwS7bqAEqeNqzdhw7Ej5ZFb4xVRVt9QoIOmIjsUOQUcSjXspbASDm2+MRg+007guzS5hnl6r4Y0myQ3Yl8YZvEBOBUxChyLxJSL8aXnkOEDnSJVXIyRJYHhjWaLGu1f97A8jcOQ5zyNY8RjO0ihVCCgg/p4scOgz5dTCAiob9zpvvncFTFuEgAtia2eINLdMIFTMju03v1cMViBgjBL7IYXJT2bN6ZiKHOpIE8tl1pzwHetgHHrheiBvOGQnDXj6XF3y/fim3j41qL0p0kkuHlNYlal+oJKhTDkRE19wqmHavk3rKv493dbYdOsFjd6wByOWeeCH/rUk3QiJSSfvoOWca3NMX5HM05BFrWSEsVGWmomoNobp/8iZP2mnMhTpWv/s1pe8zhuHw6XcMnbKflaaY+odphv8FQvn7UT5BaeJ2wVOBeyHec8M62E4qmUZVi3i1qRyVxLgX6UntLRwmg7i03fkTafZNKh5d482i2lT0QhvNTtLjmic3JTSFI4XXn63+0uVUeAVQ/MpGcroY2C56rkaTPkye5KwDN7dr0diC8lRO7LjVYlKegI1JPeh4dC5NCLAMypXITnzkHLya2edLY+yy6OyPIwUxxLPsjmVRhXwPZw/PE0rLQUr4QlW26Bykb6PePcaEYUxBqhIm/qYblR3vjaqtlahiy3qUa9hSqQjYAMvy10BFnW9YwC4GOkrIPm39mbDi9np7G4ay3mRUA+IYn/MVI6QoaDdsEkBc1lMbfVYkH95uNK9biYkC9q2DZl8Bj2qIWx7vxN/9EZY xZgCHjDD rgqbRnwq20M4mhNEv7WW97DY3ZdF+pWeqlM4uPAPzjHFlMkqvb/IaizlM94lAHLs4lO1zcMA5oJ6HOf8cPGkIkjYmDfw+GbW/q7t6ayh/5ftT5hiUaMxC1N9ipVrm0uRxyGaAJ7QuDkTQVtvBAsIw0+iL2pXNtoT/zf/LMsHhLYw5wZQfmb0xDtnhER9yDG0YxZrrfgxlQRij+aCVYCawywnrhdTQW4Npx7lkFro+brX87sxjIxfjAq5vSp4PoalI2ah2klXAemARkSwCnQQ0eHiNgswU/A/b+18qFAxx2Lp4H+aYRz3JXjr8K+8WBPUs1fBUIOk5Zz2OHY2UEMl1idKam8cekTNSFIdJuLbxv2HP3XGNmTAt3TJ0N044AiC42T3f+KGEenP88M9t/mddLpLyP/MvGkXi0crawjHRcRBAaCG15VlVBSUOIhQmGc8bDpKMJL9KJzsSOkNTbumcjS4RqmqClCLfkSMf6OsBX80CeNnMBQOMJE05FJTlSk2ci0sYz0/yU3dt+GM9yKnlIynNBMcE+ID7ZtYEGU9OLsgGsW3KczgppJECY3P4GdAnrfu5E2S0QCzEgW5k2lxsViPINrkVXxO6BIOh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov rX = bpf_cast_kern(rY, addr_space) tells the verifier that rX->type = PTR_TO_ARENA. Any further operations on PTR_TO_ARENA register have to be in 32-bit domain. The verifier will mark load/store through PTR_TO_ARENA with PROBE_MEM32. JIT will generate them as kern_vm_start + 32bit_addr memory accesses. rX = bpf_cast_user(rY, addr_space) tells the verifier that rX->type = unknown scalar. If arena->map_flags has BPF_F_NO_USER_CONV set then convert cast_user to mov32 as well. Otherwise JIT will convert it to: rX = (u32)rY; if (rX) rX |= arena->user_vm_start & ~(u64)~0U; Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 1 + kernel/bpf/log.c | 3 ++ kernel/bpf/verifier.c | 94 +++++++++++++++++++++++++++++++++--- 4 files changed, 92 insertions(+), 7 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a0d737bb86d1..82f7727e434a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -886,6 +886,7 @@ enum bpf_reg_type { * an explicit null check is required for this struct. */ PTR_TO_MEM, /* reg points to valid memory region */ + PTR_TO_ARENA, PTR_TO_BUF, /* reg points to a read/write buffer */ PTR_TO_FUNC, /* reg points to a bpf program function */ CONST_PTR_TO_DYNPTR, /* reg points to a const struct bpf_dynptr */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 84365e6dd85d..43c95e3e2a3c 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -547,6 +547,7 @@ struct bpf_insn_aux_data { u32 seen; /* this insn was processed by the verifier at env->pass_cnt */ bool sanitize_stack_spill; /* subject to Spectre v4 sanitation */ bool zext_dst; /* this insn zero extends dst reg */ + bool needs_zext; /* alu op needs to clear upper bits */ bool storage_get_func_atomic; /* bpf_*_storage_get() with atomic memory alloc */ bool is_iter_next; /* bpf_iter__next() kfunc call */ bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */ diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c index 594a234f122b..677076c760ff 100644 --- a/kernel/bpf/log.c +++ b/kernel/bpf/log.c @@ -416,6 +416,7 @@ const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type) [PTR_TO_XDP_SOCK] = "xdp_sock", [PTR_TO_BTF_ID] = "ptr_", [PTR_TO_MEM] = "mem", + [PTR_TO_ARENA] = "arena", [PTR_TO_BUF] = "buf", [PTR_TO_FUNC] = "func", [PTR_TO_MAP_KEY] = "map_key", @@ -651,6 +652,8 @@ static void print_reg_state(struct bpf_verifier_env *env, } verbose(env, "%s", reg_type_str(env, t)); + if (t == PTR_TO_ARENA) + return; if (t == PTR_TO_STACK) { if (state->frameno != reg->frameno) verbose(env, "[%d]", reg->frameno); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3c77a3ab1192..6bd5a0f30f72 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4370,6 +4370,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) case PTR_TO_MEM: case PTR_TO_FUNC: case PTR_TO_MAP_KEY: + case PTR_TO_ARENA: return true; default: return false; @@ -5805,6 +5806,8 @@ static int check_ptr_alignment(struct bpf_verifier_env *env, case PTR_TO_XDP_SOCK: pointer_desc = "xdp_sock "; break; + case PTR_TO_ARENA: + return 0; default: break; } @@ -6906,6 +6909,9 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ)) mark_reg_unknown(env, regs, value_regno); + } else if (reg->type == PTR_TO_ARENA) { + if (t == BPF_READ && value_regno >= 0) + mark_reg_unknown(env, regs, value_regno); } else { verbose(env, "R%d invalid mem access '%s'\n", regno, reg_type_str(env, reg->type)); @@ -8377,6 +8383,7 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, case PTR_TO_MEM | MEM_RINGBUF: case PTR_TO_BUF: case PTR_TO_BUF | MEM_RDONLY: + case PTR_TO_ARENA: case SCALAR_VALUE: return 0; /* All the rest must be rejected, except PTR_TO_BTF_ID which allows @@ -13837,6 +13844,21 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, dst_reg = ®s[insn->dst_reg]; src_reg = NULL; + + if (dst_reg->type == PTR_TO_ARENA) { + struct bpf_insn_aux_data *aux = cur_aux(env); + + if (BPF_CLASS(insn->code) == BPF_ALU64) + /* + * 32-bit operations zero upper bits automatically. + * 64-bit operations need to be converted to 32. + */ + aux->needs_zext = true; + + /* Any arithmetic operations are allowed on arena pointers */ + return 0; + } + if (dst_reg->type != SCALAR_VALUE) ptr_reg = dst_reg; else @@ -13954,16 +13976,17 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) } else if (opcode == BPF_MOV) { if (BPF_SRC(insn->code) == BPF_X) { - if (insn->imm != 0) { - verbose(env, "BPF_MOV uses reserved fields\n"); - return -EINVAL; - } - if (BPF_CLASS(insn->code) == BPF_ALU) { - if (insn->off != 0 && insn->off != 8 && insn->off != 16) { + if ((insn->off != 0 && insn->off != 8 && insn->off != 16) || + insn->imm) { verbose(env, "BPF_MOV uses reserved fields\n"); return -EINVAL; } + } else if (insn->off == BPF_ARENA_CAST_KERN || insn->off == BPF_ARENA_CAST_USER) { + if (!insn->imm) { + verbose(env, "cast_kern/user insn must have non zero imm32\n"); + return -EINVAL; + } } else { if (insn->off != 0 && insn->off != 8 && insn->off != 16 && insn->off != 32) { @@ -13993,7 +14016,12 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) struct bpf_reg_state *dst_reg = regs + insn->dst_reg; if (BPF_CLASS(insn->code) == BPF_ALU64) { - if (insn->off == 0) { + if (insn->imm) { + /* off == BPF_ARENA_CAST_KERN || off == BPF_ARENA_CAST_USER */ + mark_reg_unknown(env, regs, insn->dst_reg); + if (insn->off == BPF_ARENA_CAST_KERN) + dst_reg->type = PTR_TO_ARENA; + } else if (insn->off == 0) { /* case: R1 = R2 * copy register state to dest reg */ @@ -14059,6 +14087,9 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) dst_reg->subreg_def = env->insn_idx + 1; coerce_subreg_to_size_sx(dst_reg, insn->off >> 3); } + } else if (src_reg->type == PTR_TO_ARENA) { + mark_reg_unknown(env, regs, insn->dst_reg); + dst_reg->type = PTR_TO_ARENA; } else { mark_reg_unknown(env, regs, insn->dst_reg); @@ -16519,6 +16550,8 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, * the same stack frame, since fp-8 in foo != fp-8 in bar */ return regs_exact(rold, rcur, idmap) && rold->frameno == rcur->frameno; + case PTR_TO_ARENA: + return true; default: return regs_exact(rold, rcur, idmap); } @@ -18235,6 +18268,27 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env) fdput(f); return -EBUSY; } + if (map->map_type == BPF_MAP_TYPE_ARENA) { + if (env->prog->aux->arena) { + verbose(env, "Only one arena per program\n"); + fdput(f); + return -EBUSY; + } + if (!env->allow_ptr_leaks || !env->bpf_capable) { + verbose(env, "CAP_BPF and CAP_PERFMON are required to use arena\n"); + fdput(f); + return -EPERM; + } + if (!env->prog->jit_requested) { + verbose(env, "JIT is required to use arena\n"); + return -EOPNOTSUPP; + } + if (!bpf_jit_supports_arena()) { + verbose(env, "JIT doesn't support arena\n"); + return -EOPNOTSUPP; + } + env->prog->aux->arena = (void *)map; + } fdput(f); next_insn: @@ -18799,6 +18853,18 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code == (BPF_ST | BPF_MEM | BPF_W) || insn->code == (BPF_ST | BPF_MEM | BPF_DW)) { type = BPF_WRITE; + } else if (insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) && insn->imm) { + if (insn->off == BPF_ARENA_CAST_KERN || + (((struct bpf_map *)env->prog->aux->arena)->map_flags & BPF_F_NO_USER_CONV)) { + /* convert to 32-bit mov that clears upper 32-bit */ + insn->code = BPF_ALU | BPF_MOV | BPF_X; + /* clear off, so it's a normal 'wX = wY' from JIT pov */ + insn->off = 0; + } /* else insn->off == BPF_ARENA_CAST_USER should be handled by JIT */ + continue; + } else if (env->insn_aux_data[i + delta].needs_zext) { + /* Convert BPF_CLASS(insn->code) == BPF_ALU64 to 32-bit ALU */ + insn->code = BPF_ALU | BPF_OP(insn->code) | BPF_SRC(insn->code); } else { continue; } @@ -18856,6 +18922,14 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) env->prog->aux->num_exentries++; } continue; + case PTR_TO_ARENA: + if (BPF_MODE(insn->code) == BPF_MEMSX) { + verbose(env, "sign extending loads from arena are not supported yet\n"); + return -EOPNOTSUPP; + } + insn->code = BPF_CLASS(insn->code) | BPF_PROBE_MEM32 | BPF_SIZE(insn->code); + env->prog->aux->num_exentries++; + continue; default: continue; } @@ -19041,13 +19115,19 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->nr_linfo = prog->aux->nr_linfo; func[i]->aux->jited_linfo = prog->aux->jited_linfo; func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx; + func[i]->aux->arena = prog->aux->arena; num_exentries = 0; insn = func[i]->insnsi; for (j = 0; j < func[i]->len; j++, insn++) { if (BPF_CLASS(insn->code) == BPF_LDX && (BPF_MODE(insn->code) == BPF_PROBE_MEM || + BPF_MODE(insn->code) == BPF_PROBE_MEM32 || BPF_MODE(insn->code) == BPF_PROBE_MEMSX)) num_exentries++; + if ((BPF_CLASS(insn->code) == BPF_STX || + BPF_CLASS(insn->code) == BPF_ST) && + BPF_MODE(insn->code) == BPF_PROBE_MEM32) + num_exentries++; } func[i]->aux->num_exentries = num_exentries; func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable; From patchwork Tue Feb 6 22:04:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11057C4828D for ; Tue, 6 Feb 2024 22:05:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 918A06B0096; Tue, 6 Feb 2024 17:05:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A1AD6B009A; Tue, 6 Feb 2024 17:05:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F37A6B009B; Tue, 6 Feb 2024 17:05:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 54D816B0096 for ; Tue, 6 Feb 2024 17:05:24 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3038EA199F for ; Tue, 6 Feb 2024 22:05:24 +0000 (UTC) X-FDA: 81762760968.02.B05232F Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 343FF40024 for ; Tue, 6 Feb 2024 22:05:21 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ifsl9gRy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257122; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=waJIZfpNliIHPa/9880riUne4F1M98q2cd6wla9GZjk=; b=3sfz4C9645TyHMKn9CyTxGUiiKeQXSkVI6Yxhov9PbUtluF0iGkmurk1To/Vz96+JxDLf8 cI2y/HVK/yiajn5pC9bQFCUfAoBoJCjd41AEwuxokKV3NhEaJXCFnwtuKB9za/B0nX8at4 rt6ktCoAsey6zsyCxEWpY0THfBtaPDI= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ifsl9gRy; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257122; a=rsa-sha256; cv=none; b=MlVgYL6yRf65KWqPE17bP06cbhFV5fu8OO+SugCM3lCrkMrtEEz2SO+jJzu8li0nmQ5jwT tJrxW92pwnSaCQkiE7ly2Bw820QmFoRPshI6QgdeFjZy/D6Kn9uvRKXHzYHAahoOvWoLdk hUrqaj67IEVB6FljhVkgHmBPZbcdT9E= Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6da9c834646so1648b3a.3 for ; Tue, 06 Feb 2024 14:05:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257121; x=1707861921; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=waJIZfpNliIHPa/9880riUne4F1M98q2cd6wla9GZjk=; b=ifsl9gRye8t+TC6Jv5vtYFU1MxACZcjbMfxCcYDq9MH86qHx8UEmDytJWs7wzJYwyE A3I9AQtrP81iyxvmTYthuNb9htCwsDrxo87csSdiKPJa2QPXkrgow62HN+oeWtK8d1L5 h+dOqqKYVLupx/5RCkLOT/cndxIpSI2bFNq5Qw0gpu2456ZqDuFQgmctVGpXbr0Wupma GUFQXqHOfYt0upxsv03pXSET2KK7WsgWxpaBK1rA04deUZw2mivN4PGerEOwjKSxHHPL hFov5jdUW1/UA1RtDe/lSXX3m+x0vGCxWR/bUY7Op2js+jhW8745LeJpfNtJKtrciTTk v5Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257121; x=1707861921; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=waJIZfpNliIHPa/9880riUne4F1M98q2cd6wla9GZjk=; b=P+N4lSzP9h0hj5VR3ttTjoJn3alXKA1V3ctyfHfavZ3XMvU1Mq3fTGo4nL4VwuTf5B leS/h43lqHu87KhlhC6tQmE96n6UynzJxqDJNtAFYJ4FG0iOvr+d9VwUi3CdDmD6UQYe GodFvclk9xgoGojPw+MH9wbaDQkUfpyDKzSk3V0OIRdCZh4sp9/0KLS1E17+mJYZ9Jvr UzWkHAAcsTX11oe8dnp+o7rpxPYDkBX1DOOHA5IWIJB9Lv/gaZf6KnIR4Q4CcLe1VNqq uo8wS+363MmGWhIgHaUD9qGilXtejgKo5cqPRljPyTM3BglefjRj3OHENo3zc1tF1FKA NEUA== X-Gm-Message-State: AOJu0Ywr1ih3kUmDSd7tEKdRP9yb87Ff0jhKWKGso1VSdkxYUGpXn//y Omn5hO1IaH/lplvwKQWP9gqlsOLMlLj7+sbpGxMZeIXbvqyoKg0F X-Google-Smtp-Source: AGHT+IGEm41W1mIMfg18EMXRi5EZlNoYfBurQNFDegQZNFeL2j9w6VizaJwjHHTqyxrGN0HSh8f/bg== X-Received: by 2002:a05:6a20:8154:b0:19e:a23f:fa73 with SMTP id u20-20020a056a20815400b0019ea23ffa73mr606797pza.5.1707257121125; Tue, 06 Feb 2024 14:05:21 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXxN8orindXUKGG1h/zyY3mQyICEUtCIntzBop61kdfolwDs7jXajQYTwYORXc3hFm7NsG4OasP13LHB6c852skW89A5Q3SEJ9IeXDoIlf8HFxDkSEQwtp0/hR6/4zyqyoEUUOyZl7qxTHsrtAYLfY96aJG11wKBcbQfUK+jAghxuHvzl5GfAONh4LbS5QeBT2B9USv2mg4iHKbjiyPWsCkS1ydEmzx+CC1iJngFwB5VSmgSP3SEUQKbB3N9kyl4hqzgAWt3xXJbEfM3pfWaP0AZ0sXpYwjQtuN Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id x17-20020a056a00271100b006e0542f9689sm2474751pfv.103.2024.02.06.14.05.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:20 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 09/16] bpf: Recognize btf_decl_tag("arg:arena") as PTR_TO_ARENA. Date: Tue, 6 Feb 2024 14:04:34 -0800 Message-Id: <20240206220441.38311-10-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 343FF40024 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: oacifoyrgbw3zif8tipsu3yyfa35odft X-HE-Tag: 1707257121-477960 X-HE-Meta: U2FsdGVkX1+KGn2PvIQJzbrwLTGuzhLXGDpdUeBd0eIuptyqHv874vzTnuTGn1j1NiXNc5B+JBrIalPEWB4qgD2JL9S/5V5lHLhucONPHp8TzmxDSPghC+OLWeeuychhlQDlOaqpYeloNd1dl5tct9i/f4xU+tdeKxHrsMA2VkGqCGf/ACVyOPR9IttsfqxLyRntVMnxL2KqP7P1U6HE0IQc0sUC7TtWPAwm3rsRkOcnb9WihooIgS6KRixURBdOnEZ2YF8sGA4PVVvK/p3xxYsmQ4Vr9QFJ2bNQ3GAgtnKQNmM0mqTGCqeDBVw3p6j37xGAfAfoUfhjOlQu/xNFAtrxaktuc4uCw2X7u9v7wSUuaU0pAf2W8Gm0RWWXUV8rO5m8D6GcynVH79D/y8S+30ir3Qc1S+gFKZIqcOX65xWIlbKbdq/KGJVbuogmRkJG/5zUcUOEMclZ4DjDFE0JwunGW83Nq3CwHOyL/X2smDX6hg/iDYvamjgBDwdM9YE3KzK8ZQWTwJDQao+klW0ZOJz9Y2+KQ7brRw5XeZdDC6OyV0r/mCPdv9rN9AEWkmpAn4HklJDOO/fAGwQzEgj6hOiWhRVwaVXKy8L8uYb86PjAXXT7uRdWPiqinfdKl3jZ605r5mtnwY9fZ6cZ8tbOVCMSn/Ck7lvNYC3ki0oNrVNrzd9KCVlgw1BPH179k359JMz8reCAcGD3euhOEkTF7k0kXClPYl2fuT6I78brqdn4uDNSHpu9jH+fEo0HZit7cbCNBdz9S6DWXSAmPNkvmDfQylhxtRkhmcLEh9L4lj90HUQRxyihw7DyCvt7nkRHofTRnM787LTWxJcz5mrMaqhcpwukrVhTlQa1IGcNOgC2WB9b8Q1sTlVyPoy1iqTcJ3pO2RZSnR+HRPTchmhhY9CJfsX5CMfQs2cX55rbsOVyKOwcis7NxB2qwKIXvFNlwqeuGYMqkvxkfH4qUMj 9KZLP+Yu VUb7bRHgtlh1mTsuIjV/XOu4P/IwyeRg4iOraHy2wuYpG4qP2Xl6qVFpfyeZGCKuYfsehl5P7exvWZTkiBQWshaTFyxyLHlstd/NjKjFtgfB50GuomwESiM3QGxA4TGis9P4+C744mzc18O1nRx8pEmouf1oOjk3Ff4Nw6Gt+47Pw6la6/lLXY7kq4nr2CZk3XqYKMVR/UEcNLlX5zqHV/iUGpqbbMRHmotWKT4hqEu1S9YhnHah7RT2o6xkKK9kwhaievKEAP6e9caNutAnhgSStub8KHJN1cFv/6uyb8H0SB1sEzqE4OgEH3jtER+AfG8FH1+9t8oIlWO613/sUdf7qb6x5FJh231Vo5jYsfaQLFPDXQmNmwKyydJYEWs5Zue8HPEDEgDc6GzCBsX78195ik2lyEPEoSyxsd0zDFGamgpeOscCM5cIT7InZh0tbVliS64tKhl1JttEyGAEnuyantfuZ2eL6Hrs9IFZqbJejgys4eElBV+4PKN930Uu9H3vDQM3Mka6qjBu0AczeGMRd6UOE0SJMlXsSctcZwXy0QHBZpuMyXNJJmjoSU3Jtt1YR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov In global bpf functions recognize btf_decl_tag("arg:arena") as PTR_TO_ARENA. Note, when the verifier sees: __weak void foo(struct bar *p) it recognizes 'p' as PTR_TO_MEM and 'struct bar' has to be a struct with scalars. Hence the only way to use arena pointers in global functions is to tag them with "arg:arena". Signed-off-by: Alexei Starovoitov --- include/linux/bpf.h | 1 + kernel/bpf/btf.c | 19 +++++++++++++++---- kernel/bpf/verifier.c | 15 +++++++++++++++ 3 files changed, 31 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 82f7727e434a..401c0031090d 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -715,6 +715,7 @@ enum bpf_arg_type { * on eBPF program stack */ ARG_PTR_TO_MEM, /* pointer to valid memory (stack, packet, map value) */ + ARG_PTR_TO_ARENA, ARG_CONST_SIZE, /* number of bytes accessed from memory */ ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */ diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index f7725cb6e564..6d2effb65943 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7053,10 +7053,11 @@ static int btf_get_ptr_to_btf_id(struct bpf_verifier_log *log, int arg_idx, } enum btf_arg_tag { - ARG_TAG_CTX = 0x1, - ARG_TAG_NONNULL = 0x2, - ARG_TAG_TRUSTED = 0x4, - ARG_TAG_NULLABLE = 0x8, + ARG_TAG_CTX = BIT_ULL(0), + ARG_TAG_NONNULL = BIT_ULL(1), + ARG_TAG_TRUSTED = BIT_ULL(2), + ARG_TAG_NULLABLE = BIT_ULL(3), + ARG_TAG_ARENA = BIT_ULL(4), }; /* Process BTF of a function to produce high-level expectation of function @@ -7168,6 +7169,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) tags |= ARG_TAG_NONNULL; } else if (strcmp(tag, "nullable") == 0) { tags |= ARG_TAG_NULLABLE; + } else if (strcmp(tag, "arena") == 0) { + tags |= ARG_TAG_ARENA; } else { bpf_log(log, "arg#%d has unsupported set of tags\n", i); return -EOPNOTSUPP; @@ -7222,6 +7225,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) sub->args[i].btf_id = kern_type_id; continue; } + if (tags & ARG_TAG_ARENA) { + if (tags & ~ARG_TAG_ARENA) { + bpf_log(log, "arg#%d arena cannot be combined with any other tags\n", i); + return -EINVAL; + } + sub->args[i].arg_type = ARG_PTR_TO_ARENA; + continue; + } if (is_global) { /* generic user data pointer */ u32 mem_size; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6bd5a0f30f72..07b8eec2f006 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -9348,6 +9348,18 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, bpf_log(log, "arg#%d is expected to be non-NULL\n", i); return -EINVAL; } + } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { + /* + * Can pass any value and the kernel won't crash, but + * only PTR_TO_ARENA or SCALAR make sense. Everything + * else is a bug in the bpf program. Point it out to + * the user at the verification time instead of + * run-time debug nightmare. + */ + if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) { + bpf_log(log, "R%d is not a pointer to arena or scalar.\n", regno); + return -EINVAL; + } } else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) { ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0); if (ret) @@ -20321,6 +20333,9 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) reg->btf = bpf_get_btf_vmlinux(); /* can't fail at this point */ reg->btf_id = arg->btf_id; reg->id = ++env->id_gen; + } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { + /* caller can pass either PTR_TO_ARENA or SCALAR */ + mark_reg_unknown(env, regs, i); } else { WARN_ONCE(1, "BUG: unhandled arg#%d type %d\n", i - BPF_REG_1, arg->arg_type); From patchwork Tue Feb 6 22:04:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1981BC48297 for ; Tue, 6 Feb 2024 22:05:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 965986B009B; Tue, 6 Feb 2024 17:05:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8EECA6B009C; Tue, 6 Feb 2024 17:05:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 742966B009D; Tue, 6 Feb 2024 17:05:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5CD176B009B for ; Tue, 6 Feb 2024 17:05:29 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 10B3BA1A19 for ; Tue, 6 Feb 2024 22:05:28 +0000 (UTC) X-FDA: 81762761136.24.4C3083D Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf22.hostedemail.com (Postfix) with ESMTP id 345C5C0022 for ; Tue, 6 Feb 2024 22:05:25 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GeO0oPoN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257126; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kyKA4wCImIZZ5xJE04KmfmUkjYhSBJW7+tRHKBcOaBI=; b=VLtJEiIo6enIguduBAJd7vsXOz2U8CBv3XNKzmdjYc0G9+smB8SKxgZQoHfZbkalRFTcnB OqtoFuR4X1oTUsjcBLdPMR0VJfotM9D5KN/XsbjHG6YWv0Y8YVqHxAf/+70GkPHq573bib rZCWsVPkh67AoUVBmx+aKauZ40BYUMw= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GeO0oPoN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257126; a=rsa-sha256; cv=none; b=dkN6mQXsK8a6ac1mtFsrUZAv072atGt6tXfiagvSwO49TKPq/dKhg26Cw+d6QmRpb9oqkp nV74SFLPMfIHxMyzWPE2/sTrikv5aV96TGUON6iiqVnelqxbGJ+QjattvGeXmTGd9sEGGJ NyTEK5SRxOeruYX26KSLDInl8wIc9UM= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1d953fa3286so32145ad.2 for ; Tue, 06 Feb 2024 14:05:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257125; x=1707861925; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kyKA4wCImIZZ5xJE04KmfmUkjYhSBJW7+tRHKBcOaBI=; b=GeO0oPoN7ofBTQT8xy50nISAFl0wPH6j87L6HdaOmL6imY8TU6VtoUDaGgQbHiJLFr dpAwohXvUHz8lgPrzBTG+uy0fabyQUW6xK3JxePrOqITlSZ9vJeNYabD4/GzMl3ZzrQM WmuuZhu4HI3AXHF+4hMWTKM8XQts12w0vtkCRZpXAz/AhFL0b6otTKWkBsqDkYwquAb4 0qt/ukpCGewU1qfrG9EdjxD9DeHYfGYikTUx743ubutkhVJf8uYth5My/xE44CEs9iSp WM9/8TpT9N5qUj1FRhUZf4UOLXZHQaHIq2C/e4Vos+Dyfu6JiWV99mguM9S7nn+zcpnr p29w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257125; x=1707861925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kyKA4wCImIZZ5xJE04KmfmUkjYhSBJW7+tRHKBcOaBI=; b=UGMLrmeb2Q/gBWYWxLUpf0wQebntnvI2x+QobqV+fprqWMJVH38T9jar6wcURQ9MiV vjI+SeZp+W61CUf4ytChd2nJSds7SirppCE7z1BqhKKzjnGxCVptfPPy+XV4fUkRuLPe MZqLM5zsCu5vJjbzCEDVdBvJO0CGgTPI6Qklq1qqPtiOfRkgWV9Zz4/kUO+/OoD0hGFR Fp70ER1hzVZ3stHH7uvC7kE5MBjoaLZDVQwSdB1c0l97KTxfX2DGoLJb0llao5bJlbCL VZwEBn165VgmIB0g9p+OrtfAnNR0pf+yBhhH0zHDtrXqiQFmpeUZU2eP5K6clS5gkLFV 77OA== X-Gm-Message-State: AOJu0Yy6Cr2hKGvc5mwdRhMSZOgb5HCjs0fsS/eP5z9c4grjzhqkJPVr hwSLAKVvDv9MOXrblQV08zuEyqgHAXhUmEd3BvDxhGYO22OnB3wj X-Google-Smtp-Source: AGHT+IFCXZo1wWd12w2djsRFUVDKXDBOg++Oi4WlWVJBWuRKgnQggkaBieLi7zmDvrE2yA+j9R0Veg== X-Received: by 2002:a17:903:2344:b0:1d8:94e4:770a with SMTP id c4-20020a170903234400b001d894e4770amr2939440plh.51.1707257125081; Tue, 06 Feb 2024 14:05:25 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXTg5DTgGf7yruJqB/WI8D3C35qukvbUy1OcPrsmlNmRj3bMZLrv6ZKXl+1ZliwY5Yh+RSrbBH5jJH4p9m8Da+qCe4R7jQ30NGMqR05TYm6hYqiEn8rHQLlMXL6NgE36kXCg7GqxrsrdtD2zHMjjnwBdDAYT6ffHki+e2V66HSHGIdS3pm2Bi07FjJ0b371OAyhqlw/auRDkPE0BfpAnLuFVa4mCeL/MsiaPAHWAYGJs0SX24t1cZkpz2mX/gCglBvOUJsHd/acn6fNCiNO1QmPODy2sogSJAE7 Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id w20-20020a170902d11400b001d91b617718sm8619plw.98.2024.02.06.14.05.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:24 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 10/16] libbpf: Add __arg_arena to bpf_helpers.h Date: Tue, 6 Feb 2024 14:04:35 -0800 Message-Id: <20240206220441.38311-11-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 345C5C0022 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: iyeo96gwxaf9n8w817n57znohwbdyrys X-HE-Tag: 1707257125-126692 X-HE-Meta: U2FsdGVkX1/sabn63z6lFPOH8qjESGy9oqnGzCtk1gza1TyQllKY37I81J3jOVPKtwz0aPtqghpvsVkk8f35s7bMSkCxA7slxiiEPffjruVdr3Pef9B/mwMoF+sufyFAKxJb7OuxVCcpFCYjjZlL6rdOfcHoVnqQlWtpl1+HTvPwFl1avrX7tYM5+LsSmNeljT1E5jsOdTxOaUn7M2MrWVCR9CMo2zv3sofne94X1D+8R0jz60NQAL6yWA3/9MYFB0aV3wtBQFX9aIINVvLAF979zNzdXOvCbBFtyJ6R4n/fldPXCyJA6mYM4rCisn64Dd8meY51uHmxq8wbM/RPSvjIZs7WDDwnhb8epd+WK6WdZnRj6z16ZQKmf/7v8z9CJbai/PENHWMjfd+QlL+Ro7/DNg2Cy9BhJxE8od5hkP+L+tgl6Z+TUD+q5IkAC3SXOWCSCYvhrrsrpVM8mVOCuhuk2Cd8ZvkoJUN843tyWR2uiacmiBxsBq7nOn5oYRAEKci/GxGdNKK0WRWm4v9JfdXdc4wuYNAW3+CeWiJtyMssswSY5Y4gQSdxb3hhd2dHfyg1qnIHPoQDQQZxQnc/CH8kFFWv97+xQd2j7z4b4C1Hp6W6G2tr278hTdAMbWTPIhScbOJYhPS8/agVBTQ+fR3fVM+lcOWgK2AVvBU5HFaaHSMl3YpmLrqg2FoBdo0geaGBsB06xtJqH8Mf6qdJ1daTvhU8FUn0W7cfKXdUO+Mw3hrFY0Y/ZrHstRnmjVg3zp8i6B+p7V6EPIDC9RWdZya7PD+SAxXTB6Vl178h3XoKciw9JLRAaLWMULYcQgqw3ozrIJOfBkKDY7dwEhPjzWRRwyvp0yEk+5X7uarsRPD3htrOL9kP0Mk+5P3oLW7kCKSLG7wPlc7be/g/mtEEB1CD5YJFk8+VQRPmA2dXJpFqS4Nkn4HwoRkVXGpP41aBkE2+dZWvV6MlTpu4WTV bcuNQzGn XocjI2/cLcO9FDX5iU9cLPtb2E9mXCSkQRy+FsKGZ4s5o2e0W134mRZRy5a8e/pMilmZHFzQrnAbPizgg2/xlaVUsfSrz2touIVMHoT6jspKznNVRrfRHRexG+VhtuT5OqS6y95OTX3Bg+K1AaI50AuOmInhuqiww61FKyiWfnE+qEdq3yGWmG5DZixI6b+hnJ8ydajpUtKM6kCv/9xtCAVNgK/slVHsBhUTc/izy96At1j4JhYgNoIXe/GZykHkkQTcGoi+G8g+l9RJuWK+LnlPihTPpdpFOpiMuTmnkHwe6u2WxpOtXMqinjOIjufDEmjpSmaH7i88w6URAT6c8gC9rAFP8lY/H19lKNENtLIKrKx2fpxz9GA5VvL4aQ4p/gis/KpgQnhEc6mvoHcMhF3Ca5aDjS8M5vetWKvWIgqhXx26t3PxKCOyZkSCypsZCgTjHkbss+7e5FK9f0lwRsm2FK1QqZ3kMXMSsB8za4K9QSD//S2zF0HXUfcoApTw9JbuZgt8X9zhaPA572rR8Cvq1LJoPLcFmnqIICrjgLxdCLqUy00IBjkCxCMUjurV4FiIQ1KQU1b2KWUHIAQ0BP2NLRC4gSIRnmxe1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001851, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Add __arg_arena to bpf_helpers.h Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/bpf_helpers.h | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h index 79eaa581be98..9c777c21da28 100644 --- a/tools/lib/bpf/bpf_helpers.h +++ b/tools/lib/bpf/bpf_helpers.h @@ -192,6 +192,7 @@ enum libbpf_tristate { #define __arg_nonnull __attribute((btf_decl_tag("arg:nonnull"))) #define __arg_nullable __attribute((btf_decl_tag("arg:nullable"))) #define __arg_trusted __attribute((btf_decl_tag("arg:trusted"))) +#define __arg_arena __attribute((btf_decl_tag("arg:arena"))) #ifndef ___bpf_concat #define ___bpf_concat(a, b) a ## b From patchwork Tue Feb 6 22:04:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36618C4828D for ; Tue, 6 Feb 2024 22:05:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B88C76B009D; Tue, 6 Feb 2024 17:05:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B12116B009E; Tue, 6 Feb 2024 17:05:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 964B16B009F; Tue, 6 Feb 2024 17:05:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7CB306B009D for ; Tue, 6 Feb 2024 17:05:33 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 02A6CA15AA for ; Tue, 6 Feb 2024 22:05:31 +0000 (UTC) X-FDA: 81762761304.17.693CAE3 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf11.hostedemail.com (Postfix) with ESMTP id 286D240014 for ; Tue, 6 Feb 2024 22:05:29 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nkcIzgmb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257130; a=rsa-sha256; cv=none; b=HK9MK/VjtGSXa1TaWN+PTHodrEIkG8lkiWzk5E2kOJ6ocPuBmRUrJn78jaa2484vGXdl39 JRy/v2GguVh+9ZFhhQQqG+Q/HozQtKGAfAj0xF25RbNTGf9TAyXktfSZLe0pJmbJ06Lyys VcLPkTYXDVOPiX9r2AXpv/iZOvyX4jU= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nkcIzgmb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257130; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RiXImkPg8/G5+2DOGxZ3xDCLE7/uYVswnoMrJHCmlE0=; b=c2FYNxrLTIqSP3nodjWyWIu4zGFUmVfSna8rouZ9UIkjPegH3th/T7POAQwnoBTQZUx5sa bfQ91bV1U32xREYROtxks5EbbSqUXYvvEOWjHFgDpMjZumWU7dBwwAwHodxF/vn/v5Mnh0 HgkA76FcNwqRfvLd+ZNzW3AD5FFYPIo= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6e054f674b3so12054b3a.1 for ; Tue, 06 Feb 2024 14:05:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257129; x=1707861929; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RiXImkPg8/G5+2DOGxZ3xDCLE7/uYVswnoMrJHCmlE0=; b=nkcIzgmbnz3kk8yJMdggsemiuzxA1czEqmUoiqzPpCWK7ABoiuuruBYnlw3UL/jB8W 2IGeFIJDAsLL92P/Ere/X3J5Z0ykMG5lmGIKfQosgTM6v1K+vs2VGNUC/0MRM8wdeAhV dYBX6bviIYcF5g+rE3xbVW9eUal1kp0eZ0QUQTzh4cMlSWH4dqqGpvtizb171BQguJO/ ZX5UuaOGLsIHnC1v/z7TmADSUJrbeAboSdpuNOkXDIKHmN2TOY5M+KYciY3zvTqISjhj qPc8iELyFbv7XGMBU18h4mmz63NLaKoP4+J48AUlB4X4zfgvXa25BJAPV6maN+3RsyxW ckXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257129; x=1707861929; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RiXImkPg8/G5+2DOGxZ3xDCLE7/uYVswnoMrJHCmlE0=; b=SUNohmpwtRAzQHL/LqhPoo2HJLHOmYY+060S37MrBF5EGUT6MEBm10H8XukLoGLcpy nVlP4I/MQBNxLp9JS8y9Be0rCY8GWDr5bealZ/9BNl9mTNQ6IUxsqGErn7kruSqH1CQ3 OcfGjDJsZSt9eXE3hq3xJF5iH03+VKv0wS0og+92melV4AcKL/dzsZPIKlq3PxvbncUr NktW9wlBsNQEX9EIne770VZHzZzlAvQ/9IDp1RXYM+rK2RlI9tZKP1WtM/6Di5jpT38l jzQA/XdhdoEPGye79sk/veO+fBgYqmndflryEkdh2fh1PHAutvy6YpU4il8UuB6o74GM i8Uw== X-Gm-Message-State: AOJu0YyjZjVf7XYexQoTPahAK/gN8FJzGAGM0OjklIFJjfrXGGmbwXVr 5miUSX/2NDfmB5hfU+Gte4wXeebjArpFymaHSr/HHBHj3jV3l076 X-Google-Smtp-Source: AGHT+IEIl4BNu5aTiVeud6z0/oB6Ycj6jmTdFR/O0kus8+9trSKYb2Mt0v8czJwLnLT6tYLaUsGdUQ== X-Received: by 2002:a62:c144:0:b0:6e0:4d28:b3a with SMTP id i65-20020a62c144000000b006e04d280b3amr813386pfg.21.1707257129006; Tue, 06 Feb 2024 14:05:29 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCVDvb+r0mk8Q2LDEv9B9JMgXfjhY0opj4ilgwOds4APWie51YIRNJnQsC39i+blOZty5ZIcj+KLRBiH6UTqBDlQcNNR2QBOypbk1SEMOeJJvz51o7vEk16399WLfVIf0km+ypgZ7200F6hdBlSZEkCOCdwTvjt+Egj6l8giEFYADjPdyFEjDSXDrkCv67crZA5GJY52dcWfEwzqtvxfTXFf7LbRxgpF28eLdOexz/BcKOiXaX4VDefpgoWZqFjoHApuj3rGrSfywcJymCoRt8xCDNcKkAZJihbB Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id k138-20020a628490000000b006d0d90edd2csm2588957pfd.42.2024.02.06.14.05.27 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:28 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 11/16] libbpf: Add support for bpf_arena. Date: Tue, 6 Feb 2024 14:04:36 -0800 Message-Id: <20240206220441.38311-12-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 286D240014 X-Stat-Signature: 75y8qutptpin5m6w7swiotwzxw1ccn6r X-HE-Tag: 1707257129-893632 X-HE-Meta: U2FsdGVkX1/5KN15f5VhdFbBk9hMa4v3DqpxIBS09NxWx+CidW7ahttMyV2esOCpYwda6j6+7QENML8pQQhjsTY4vwBQY8YNPtAtwaoi2XCt7LzkLyJyoqqvzY3ERxi2NA1Tlr4tcRXCjGnSNjL/dvCsV0hEQ25q/8RzlJFxpZtjOHVzRlfPpDXK5p3s0u8f4Ya2VeLLp2hDPlafpmEx2NXK7MvFQrN6egs5FEagzcw4MoqbCnr7x7cKwHhTcx8N5SEJM7c5p+w3gRmsKo6L+CTXEQC57+V7uzy7C1gXIfud7vEJkFvNX8TsrqkMfCSHWBCGkIqm8kDzwkd/OwjIcxYRJRmO+WCampZA917d6tS1cIer/9ADs285plaOovfYGOf45dO+w8uHp6a9kEbrGIQxCn9sI+znyOqgyvom+s+RcLXnQuQgZo5YorMc+pen4DplU5Y0uZACZkzA9eT2lmO/2My042cHC2QMhDqajSwDlVTRjeyc7Xp8zfxhgP3n/F1nnecn+px0S98yh7TPwoQoKVhQBRalKikMfZs9vXzFRxKT7ZECXGk8rFKrGDkhR+fU3YdlAbtKKypBLShRkRdMRFMLFFjEwu7/J4+SDRMAV/drqMw/FvUSfC0Jz9IrmmhinetuOuSQEHiZRigkb3fBtM1fc0wxpeqW8uAqxVonblwOs2RoMipywdwGSdCnEmUjSrJyjC5pssbes9Hc9ET7X79mH3C42A38zYyEgqV+V2wCXVzE3HRkYyPJonMQV78ib7TLWys1nzcvPN1HiZ5NZQxibUDZo0n8GvKW8pxCPWmQx9wOPPeijAZbYWehwaMTlgFGs0KMPqgusBSBMUhM96XhQAgYLvlM6LQwtRhiAZ0JmNfwNp41TThW9RltLzzWxY9+mZ73btFatHU0WEZgCNyasZ/DeXK+prd6MsBPpqZL0BUrFbo9GJZORpGLNjmKqVP9EoX9etNskhF L9fn12d+ iQfYdRPL47vjrpnG20Gly9vci7adg1CL9zEeh3sqh/hTQpBrDOndWFXKu10Aygj/u4nSxhb/IzcmVJkbfU3rFNp4CIS38uqNQACTD7wpFcuvvRZnLSgNRoT/t6hzSsJQEsEffs4egKrPh6JYbIVrRd3mVrEDa+mdFYoIKP7qym/m1CtDiE5eEwKYA1o8UnwCnjQVdUSdmZyIelEPYqngthZO8BryDHEYyJitAKbmDQIiJZs1n9JkXLbFX34dubDr3iMDWOD4UqdFght93V/YO1bKF1A9qrmevCTzdzzdPD3z062JEr/uiIfic9eIMOCjKmz6K8izNLN0mQt5q0iU+LyovTYcgmwRc5ABx/cIKy0x1V+WukehOvfuYj/ujwohs60AoA3SzIU5yOWf0I4PmlA1PVhIu+AY/B19paBMBQ0j8RV6Lzr23/SeNSfD9TuiPEWDNqSDe05s32hfk0aUvsvG21Ha2W/iWnjZKID0NpAhTFWAirIm3hUFpsYOXd2y6nEXDmxPMf2R6Cbo+x+3sBhU9GlN0AU0DjlrkC6Xcc22/0rwHP2DFfGoCf4g4E4fpyMLqF57BXb/JU4MK8ZjcvNRz8TW6xx++1T+w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov mmap() bpf_arena right after creation, since the kernel needs to remember the address returned from mmap. This is user_vm_start. LLVM will generate bpf_arena_cast_user() instructions where necessary and JIT will add upper 32-bit of user_vm_start to such pointers. Use traditional map->value_size * map->max_entries to calculate mmap sz, though it's not the best fit. Also don't set BTF at bpf_arena creation time, since it doesn't support it. Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/libbpf.c | 18 ++++++++++++++++++ tools/lib/bpf/libbpf_probes.c | 6 ++++++ 2 files changed, 24 insertions(+) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 01f407591a92..c5ce5946dc6d 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -185,6 +185,7 @@ static const char * const map_type_name[] = { [BPF_MAP_TYPE_BLOOM_FILTER] = "bloom_filter", [BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf", [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", + [BPF_MAP_TYPE_ARENA] = "arena", }; static const char * const prog_type_name[] = { @@ -4852,6 +4853,7 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b case BPF_MAP_TYPE_SOCKHASH: case BPF_MAP_TYPE_QUEUE: case BPF_MAP_TYPE_STACK: + case BPF_MAP_TYPE_ARENA: create_attr.btf_fd = 0; create_attr.btf_key_type_id = 0; create_attr.btf_value_type_id = 0; @@ -4908,6 +4910,22 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b if (map->fd == map_fd) return 0; + if (def->type == BPF_MAP_TYPE_ARENA) { + size_t mmap_sz; + + mmap_sz = bpf_map_mmap_sz(def->value_size, def->max_entries); + map->mmaped = mmap((void *)map->map_extra, mmap_sz, PROT_READ | PROT_WRITE, + map->map_extra ? MAP_SHARED | MAP_FIXED : MAP_SHARED, + map_fd, 0); + if (map->mmaped == MAP_FAILED) { + err = -errno; + map->mmaped = NULL; + pr_warn("map '%s': failed to mmap bpf_arena: %d\n", + bpf_map__name(map), err); + return err; + } + } + /* Keep placeholder FD value but now point it to the BPF map object. * This way everything that relied on this map's FD (e.g., relocated * ldimm64 instructions) will stay valid and won't need adjustments. diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index ee9b1dbea9eb..cbc7f4c09060 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -338,6 +338,12 @@ static int probe_map_create(enum bpf_map_type map_type) key_size = 0; max_entries = 1; break; + case BPF_MAP_TYPE_ARENA: + key_size = sizeof(__u64); + value_size = sizeof(__u64); + opts.map_extra = 0; /* can mmap() at any address */ + opts.map_flags = BPF_F_MMAPABLE; + break; case BPF_MAP_TYPE_HASH: case BPF_MAP_TYPE_ARRAY: case BPF_MAP_TYPE_PROG_ARRAY: From patchwork Tue Feb 6 22:04:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92847C4829A for ; Tue, 6 Feb 2024 22:05:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1EBC76B009E; Tue, 6 Feb 2024 17:05:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 129366B009F; Tue, 6 Feb 2024 17:05:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBB956B00A0; Tue, 6 Feb 2024 17:05:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CFDE66B009E for ; Tue, 6 Feb 2024 17:05:35 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B1B291A063A for ; Tue, 6 Feb 2024 22:05:35 +0000 (UTC) X-FDA: 81762761430.11.9246646 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf29.hostedemail.com (Postfix) with ESMTP id 0BCE112000C for ; Tue, 6 Feb 2024 22:05:33 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=R6l9dQOD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257134; a=rsa-sha256; cv=none; b=poDg3Qe9YWZDGUj9qou8MqGuZnZvFvCFseWEkQF6bK5JCLhB6HiSmvBTXYH6CVTAas9c1j hevX76II+TYyp/bC6YLnCjF2fHx7yrNOUiXClFLjhcY/YBjYt3R3zoRVEtdxtFYvgr6UYP 2eUnH965GDhBiII1nLgMpNg72c/Hdxc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=R6l9dQOD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257134; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wpvr7dgsK6u0sIpxAzSJWUuWXV5TlyIi0UfrrFuy1/I=; b=e1OFBY8eiPCBS4qcvHxsRzh6OjM9QbzZR8oiLeshUoJjj58MlVVmOpcH1gKjPSejBOlssq 4zSVVLCe4liPTgEGju2O5QeaHiLBHO3kMVttxPitVHqf2z91yy0LcWCuslegKM7gfn1ztG KJGmNa/8cGAhL3ceMc+2PqNaHnD/a3E= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6e03f7f427aso6219b3a.2 for ; Tue, 06 Feb 2024 14:05:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257133; x=1707861933; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wpvr7dgsK6u0sIpxAzSJWUuWXV5TlyIi0UfrrFuy1/I=; b=R6l9dQODLMIb0i3w7OxwWiHlh7lrWwgmCqLQ+D70oDzJOyYwo49uvX/saACx2UY3Sc 29w+bbIQ5D0tOePIoppU329Ps6LblA7wElh1c5bFJgtlekgjvT9bJ7ldMPQCUBwp/XTW NbMY5+3azXgBc4Agfzu1+tPmRjKJeMXqu2Sq8F0GLT/ub3uhB9sgp8MKHTwNGNnnYjkX QRQaRxzJFP5Kc9ocWCBzzSg9bmNRxarAPSSZhPPC2FS1lPBSuN9fDgJtc2GFFyvO7S8v jmF/O7LIpwg1A14d6s2aGkiU+8YRIV8Rwmx9GJP9rEoTyyMBD1fxefZD6z46hTUdjXFx STzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257133; x=1707861933; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wpvr7dgsK6u0sIpxAzSJWUuWXV5TlyIi0UfrrFuy1/I=; b=hSYzf25SANYgTUVeaKc0XzLEzHf7sVFlbXZTiVkXsXom9dzIUzG5tlReF161wAN2KL AzJZEjNAitXGd1ZJjYRlLDp0YIJO1eg1yrvF3jzSfYcW/ZXeVnjq+eEEvjJ8UR+eSxcP jYzAvmslBadgbAeyd7vLrqnPrU4AK/8G+MaxN1JNPX0/ccjSe/ujCAW4y7gBDu762ZfC vLthN3v+JilkkTOcUm2kOS91aAd1yDApLJCyn3OcJdTaFHbRqFk64zOkR/Mcu99qnv07 r5nZsEk62N/xYQ/uFi+Plh4OdgyKCOhNVvBMWeMuaXMl+au0ZFv3RLeIOWSP3K7mMkib nUkA== X-Gm-Message-State: AOJu0Yzt52Fwumw85K7IN9Pa1/VUnsLG/oQ5fQi97gAqfvPOzTaNT3m2 RMEVSwEJv1UErb8c7pTuQicTMngvzrlrhjzznazy9CDDnO1+DAWa51Rgr9El X-Google-Smtp-Source: AGHT+IEY4Ho/vNriwoPjJ4S75OBCY2kHP9G7jpkOFNGGYsmw17IByMEq936vsD27kQ2v+hwLcnpchA== X-Received: by 2002:a05:6a00:c83:b0:6db:d4f8:bb1d with SMTP id a3-20020a056a000c8300b006dbd4f8bb1dmr1301419pfv.2.1707257132982; Tue, 06 Feb 2024 14:05:32 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCU2OGvHfmq89UQzCp1rwME6GQJLViCec64yB9gCcsghB2SWpFWsKo6/ZGOtD8X8cwbT65OVFmOToCbMFo8mIFHt5dOdX6bHawLpFq78B2yQpO2SQ+CAyDi7JP85wFUDSkEALC6cY8Jt2uzOJxzQANZN9nuIkX26PjvSgZmc7qjDGyu1p3St4AYihBbq9XC9KxRjpCsu9huR4K68LTgrecJJNvFuvorikOF9KmU1EEzO123aAdBH3wpLRjJAe2ZyON/Sjy5Gv2v9eyZ3yOVxj/zYKjqBFSWlngGz Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id o1-20020a056a001b4100b006ddc7ed6edfsm2497587pfv.51.2024.02.06.14.05.31 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:32 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 12/16] libbpf: Allow specifying 64-bit integers in map BTF. Date: Tue, 6 Feb 2024 14:04:37 -0800 Message-Id: <20240206220441.38311-13-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 0BCE112000C X-Stat-Signature: ei4uaai1ggwxd1duqgogikuof4x7qrrq X-HE-Tag: 1707257133-260588 X-HE-Meta: U2FsdGVkX19k2b4vy0MM+7qlhrkC+EvRukEhtUC/MwAL5sjrB7ydDHkpCXNgIu9afub4v7f2/r3urQPOxc57N4zW7RIC1Du32wMdAHBX3LEXAaw3suXFliavchQtpo3JcvNrMBGlm70PNeohuXtLooafb2z1v+J+sz2EJBB/DROn4Qc4uJwecQC1cHUK8yOJgAiWrOBSbj1OABMLTW5AXaHWalWTTLRERr05PfAXK/RPlcArZeKVNOF22PFYZzGu4Z0L1JOqfxtDb6tMLkYJBU5ySRC6dGY/+EaNbEvzz6NPIiVeFYsF8U/ZNO2lZiT7LUSgCqcfRwrDkUuH5B52yK0BB5N5+HdI6W3csxUlSWN42lO4aRL0WUCc6OufJ9cVsj865zbf72/95nNRr8Y7qjpkfQC+RRLsrwgrbhVvVGwSVNy3YaEYVEZWs2t1eRML1qKgV3roZqZ0GF6Tf3J3vUf0swnrajgwAwpjyqeNG2no+QMLCbCQMV4foJkb0i1NPqd/kyrZNNHpZ39WAInphKP4q0DqIcsZo5OIiccPy8HOnGEvUDEr0Yz/7pMXRYlGxdkaT98ggv9fYxTtcsPUp3lUM5LTwCn8AVX72O1BKt3IqzFi/Q06jnwhjDfvbDoP8LkY70Iehl/lwl1w19cL6sjKH20yQcQChYEaKyQqXSsf/vHWr+r3grqvdajFN2VcLu+UlS8Ebx5akHjMcs7pdPBfH6YGwMc1jtC2iHHnPMDCqPkN6Tcy1ZUco9oGP6xJk/SEgZ1C4WjYPTeDgVlY/I672d3rdgUYf4wMiOwutXE+LmLDXSlLrVfx+xKUHc2/1G8jDbNTu3QHg8PnU/bpEOGL5TjGIA7J0sQCFEaKpQTrqSK1cllkWvOh8ZacudV31xzzfMgM/13llcPpZjo0ijVT28xiekcyzw46yZjyzBE4I7Mr/D6cXXewv6c08ZpH+4sW9JeNRlzA+RyhJ4G 0Cfl0rXX lyAp3n4EDq8rMDB9kGtutYOqMJUBWRYZCd3VA2oHMzkPzthPnHm27EDUl87iVD0rHtjLP+PmqnyM701trDSdVufAloe4L/USbfuFTW5TwQY/n5K1SDA7q9tFbNl6vMy4oWBrxBnXOF+aHrLNZ3gMzaNhZrg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov __uint() macro that is used to specify map attributes like: __uint(type, BPF_MAP_TYPE_ARRAY); __uint(map_flags, BPF_F_MMAPABLE); is limited to 32-bit, since BTF_KIND_ARRAY has u32 "number of elements" field. Introduce __ulong() macro that allows specifying values bigger than 32-bit. In map definition "map_extra" is the only u64 field. Signed-off-by: Alexei Starovoitov --- tools/lib/bpf/bpf_helpers.h | 1 + tools/lib/bpf/libbpf.c | 44 ++++++++++++++++++++++++++++++++++--- 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h index 9c777c21da28..fb909fc6866d 100644 --- a/tools/lib/bpf/bpf_helpers.h +++ b/tools/lib/bpf/bpf_helpers.h @@ -13,6 +13,7 @@ #define __uint(name, val) int (*name)[val] #define __type(name, val) typeof(val) *name #define __array(name, val) typeof(val) *name[] +#define __ulong(name, val) enum name##__enum { name##__value = val } name /* * Helper macro to place programs, maps, license in diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index c5ce5946dc6d..a8c89b2315cd 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -2229,6 +2229,39 @@ static bool get_map_field_int(const char *map_name, const struct btf *btf, return true; } +static bool get_map_field_long(const char *map_name, const struct btf *btf, + const struct btf_member *m, __u64 *res) +{ + const struct btf_type *t = skip_mods_and_typedefs(btf, m->type, NULL); + const char *name = btf__name_by_offset(btf, m->name_off); + + if (btf_is_ptr(t)) + return false; + + if (!btf_is_enum(t) && !btf_is_enum64(t)) { + pr_warn("map '%s': attr '%s': expected enum or enum64, got %s.\n", + map_name, name, btf_kind_str(t)); + return false; + } + + if (btf_vlen(t) != 1) { + pr_warn("map '%s': attr '%s': invalid __ulong\n", + map_name, name); + return false; + } + + if (btf_is_enum(t)) { + const struct btf_enum *e = btf_enum(t); + + *res = e->val; + } else { + const struct btf_enum64 *e = btf_enum64(t); + + *res = btf_enum64_value(e); + } + return true; +} + static int pathname_concat(char *buf, size_t buf_sz, const char *path, const char *name) { int len; @@ -2462,10 +2495,15 @@ int parse_btf_map_def(const char *map_name, struct btf *btf, map_def->pinning = val; map_def->parts |= MAP_DEF_PINNING; } else if (strcmp(name, "map_extra") == 0) { - __u32 map_extra; + __u64 map_extra; - if (!get_map_field_int(map_name, btf, m, &map_extra)) - return -EINVAL; + if (!get_map_field_long(map_name, btf, m, &map_extra)) { + __u32 map_extra_u32; + + if (!get_map_field_int(map_name, btf, m, &map_extra_u32)) + return -EINVAL; + map_extra = map_extra_u32; + } map_def->map_extra = map_extra; map_def->parts |= MAP_DEF_MAP_EXTRA; } else { From patchwork Tue Feb 6 22:04:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 951FCC48297 for ; Tue, 6 Feb 2024 22:05:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 287336B00A1; Tue, 6 Feb 2024 17:05:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 236A26B00A2; Tue, 6 Feb 2024 17:05:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B1666B00A3; Tue, 6 Feb 2024 17:05:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E71936B00A1 for ; Tue, 6 Feb 2024 17:05:39 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C859512011E for ; Tue, 6 Feb 2024 22:05:39 +0000 (UTC) X-FDA: 81762761598.07.58275F8 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf07.hostedemail.com (Postfix) with ESMTP id EDB6940019 for ; Tue, 6 Feb 2024 22:05:37 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VCfs+39g; spf=pass (imf07.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mCIDfn6Q/Sj02IoZMnsUeC+cwQq+R0K4Vy6avN3EcgA=; b=SsodAz+ajZrOlyaFRgRYYZxhvj1WSwBr5ixIQ4LpAwenrpbMWW9Ilq1f/mJq4PkRD9Ui3Q dI83QAc9R7S0OttSccgS3RCwUhxx7zvvIgNMP2515PSXFE43JgpOD0xKByAfY24h9u/t7G pxFC6xi96chEJ5pTbV+Lv3drpaUyje4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VCfs+39g; spf=pass (imf07.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257138; a=rsa-sha256; cv=none; b=kB650Q1osjucBuhktN5SaANgbMLu3yu9pdnnOUGDYlM1FVrTYHrEQAoVPI0tUiQundOYm6 e39BOdjotCa8ydVAo5JhZuPnGkgGG4Gug/g/6V+Xc279XXt4k8oaihJeFYj6wHyBIvDvs6 Te3pIo/+Ov2hNn2bQX1k8v9vlfD3ubk= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d934c8f8f7so42785ad.2 for ; Tue, 06 Feb 2024 14:05:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257137; x=1707861937; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mCIDfn6Q/Sj02IoZMnsUeC+cwQq+R0K4Vy6avN3EcgA=; b=VCfs+39gtXZv7paXd8uS5F2YNdABMnVkNiXjYchCREKl8dLNc/Kj6FzM3Su5hFCVFs No588FJgYYOuQVN7HPvDfCp/4NhNLmG2y59NzkYS8bwtSon64s0xoFPydX6qg5ifW9dB aCj/w3VihXKl4abNMDncivD+XY4p9Mvd3HcjwhL97UtSyVQitAByC0ukNM5c2qUgIj+b pDfY87kZJeqpR0du1/VBGObiftmr7UusvyL3Czru7v+ZDFliqTv76vTcXL2fC9JEX41/ MAb7Pr1S1AS7Bl4pItUDV+zA55iJCIrtbsC44nYQJXOIAgKYkob0+9itEKrecrc8FHJS I+WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257137; x=1707861937; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mCIDfn6Q/Sj02IoZMnsUeC+cwQq+R0K4Vy6avN3EcgA=; b=RCK2S0kt/31t9Wj3AGcjT+kuY6ZnBQOZw5AhidVB3ffZBIetLN3F0r1uIMZALCM4zR /udq+x4xXXy4HLjbA0gQFl9kKycQGlA9iW4CLG4ip/u5n7paf/VkXVVULkqKKulyLvJ5 p1XPJwLNFGIFtcvZra2p+G0dL5y4P+6RQqM5R63M1+h0p4mwIjhy7PfcPqyb+SSEML8+ pwdshPFJfRTXbHs+Cvtk1xYbySn9ufmAWMOob0ofLhniz+jnA3zj4xYYMPvB+LFeg4vy ycpaxuYJOgJnRupZqL9QSQioPjwAjVVXr/YPC63x4F5Medgx3AXY1Gukk58DIZ7zSUf1 fJ8g== X-Forwarded-Encrypted: i=1; AJvYcCXgGbRaA62pCS2AySS0ysOQ7LtIJS/Jii0GCR1IBzD16dvOoPEPLSTKCZaU12I+TBZmSYp1bRYCaQowGKpU7/R8a8Y= X-Gm-Message-State: AOJu0YwoagoL0nJClbq12cNWstaOz99Ymmoa4dh2ejvsvot6XxAcL+eR TUZivNA/pavAUgSpbOGmGll0hoV6rLXcA/oNlQ5nBb3AafxkTJ8t X-Google-Smtp-Source: AGHT+IFC/8gaSn2ezfySvQY9fE+LjbzC6tGU3b69/lMAgjDuoKzugaxeFxuGwhXQP6FF6XydFy7cZQ== X-Received: by 2002:a17:902:d505:b0:1d9:1322:5b0d with SMTP id b5-20020a170902d50500b001d913225b0dmr3444099plg.65.1707257136843; Tue, 06 Feb 2024 14:05:36 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCV0x5GbXUCf5LB0OoOZH0U0NIvJ4KFVm6E2YVqM7e7OdGPW+05tZ5DAXSvvES4yBjccZ+t5IUAp2y3ibsYajhe4yp9SVObP53SFLmqQJAkx7FyJme2n9iRr5USNt4OJlJqM8O2xXtKU+496radnUUCnB5P/KQPT6zgteLeYt5oGrVMYnpbH4/Dh7vDvJGTDyB8HeVf+n7M4FLc534D7uDs7Skwll/6R5u6HMOZl0XgdXeTx12OGxEAN58tNuRpgmiZPbXSDoBOkb+iBJSRun85q1Mu9578UYxyZ Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id y11-20020a170902e18b00b001d6f29c12f7sm4613pla.135.2024.02.06.14.05.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:36 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 13/16] bpf: Tell bpf programs kernel's PAGE_SIZE Date: Tue, 6 Feb 2024 14:04:38 -0800 Message-Id: <20240206220441.38311-14-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EDB6940019 X-Rspam-User: X-Stat-Signature: ngozfkafcfa867rnppodknns3fusts6z X-Rspamd-Server: rspam01 X-HE-Tag: 1707257137-39049 X-HE-Meta: U2FsdGVkX19oGKqNjlRuif96WdHiruIJPDY3zXnGOGYIxCJf98gX8Ao5V2eXPYDMN5XjpdzyFDfUIXx6pud24JrY4ntjoeXB3o74Lx6yCk2yLq/qoALCJVh3HVU3dJmHlQmFAL2YmwmsBWlzbauVBFo6goKMM8IPVgT0Vyie8bIKGiKRiGBGEoY/gcqc1RtYoiXRX4nsXPo/Wgh83mP6ZYk0QSx5YNeyxqkHqlNE6AJx/Mws/eK2JgUZcb5gNBdop78EV2ga7FB/xxAJ9RvbFFtyg0nXkPda85461hS7buqnTsHcSw16ePu9TnV0o0ih5DjmHU2xJX9/v3NWRDbCU0S9nPmTR8hLb1agsvWa+Ii7XmZBc/NTE0H7wfwZrRs9vkgUdTela0qwl85Bg6Bua2G1n4y/I2BTF7tEw+CzlLyBVh6dGVd/cY6oYMrPhh2vHHz/2oONnBCIS7MUMfIHAtOSkqGyPXlcUb7ybqgC5iWdcl/1bPrUmbmWEBdeVg79PDsFqpDcIOddaz2aNB1ZWl349lQmFUqUDT+ksUBTkydpjG2IWBJ52ocLPHjbe6yyU41KVLuVbT6tgLrBCL+3DeTONijOR+79t6oBLU8n3CrgUTdv6fnqX49O8WVtYlHFVhhZEabKYPI+uy/2uiNCFxjDpBz2Hlni5ek7G86imcwRv91zxVW2AcL70C3OJqFvLRVPxvkrm0CBoPanM/tTxRraE+cSDIHNVRHqTxVyO+RNimjJdlCgfPlSZgXc43xDTXu8BPt3Hkop0roQIfR5pJ2eohnl551DPUcNLonCADS8KskNMjpDaeAIhP/eXbzqo4438WC/NrDLCW2Ae3DsCRtARl7eG+ezGS/0f3FAwolQte/98SLtIrMqv6Mm3hngT9WcYczpUdoO4LEueop7n+kuJOMHyFDDm/rLFa5cYkxqBxoOPlyxB68m0ParYbIMvFugpHQj4TgyAjpTOLW KGq7zJFE yPfCoj+pOc2LV7oJdMsgqPGW456ySAlpKeIjKiw5GJhR5wD0DCGOw0mA7nChtFFOzw1OWg6/M8Ij0aYu5l6EC2FPeaEk/JWJNiXV8hRUS2jnXYvbv6461gDFiEVFy2xkxTKjEqAIp0icv1jwbDx0GpVH1XcEYxDruO/VCJprBYcFiE10ayo5VH87yviSg0k8wOL2KKGtWFJAUEExnPZCW13rL//hLULe8mieDhU/wxsfnnY6drWu0FEBWn2sS8N8DfRvkuMeVV3M3n2KS52GA/5iLf0xVilymyR11EKLh9NbEj8YdhJ9YerKx1RHvtGc6gl8JqlwXPAM9VkkepZ1x5eC58RXcmmx8Ac0pZMQmXm+kgRCyAPnXoIhEVDZ/ONpXFB3adZO7iGxFrKyMy+gmJNH9oS2/JorqEv5JE8BCNysbn5fZpKXluFwVnltsSNEr2a+Vw89nvrUti9oyK4MkSiS8fOO7tANj8PQZhLuQRf465GJKt1vv5ko2vmXQq9g32cOAXq5eSrCEs/l8PeWHFpE8nP2992EfoNF3TXokHpkFNp2sfZSUry+nHElCo1XSg+oRRxG1mYUQ0zZ7XbfUtHeCZbuPBROcYCekTWucIv5wrW9OWsw0cvT6Nw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000023, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov vmlinux BTF includes all kernel enums. Add __PAGE_SIZE = PAGE_SIZE enum, so that bpf programs that include vmlinux.h can easily access it. Signed-off-by: Alexei Starovoitov --- kernel/bpf/core.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2829077f0461..3aa3f56a4310 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -88,13 +88,18 @@ void *bpf_internal_load_pointer_neg_helper(const struct sk_buff *skb, int k, uns return NULL; } +/* tell bpf programs that include vmlinux.h kernel's PAGE_SIZE */ +enum page_size_enum { + __PAGE_SIZE = PAGE_SIZE +}; + struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flags) { gfp_t gfp_flags = bpf_memcg_flags(GFP_KERNEL | __GFP_ZERO | gfp_extra_flags); struct bpf_prog_aux *aux; struct bpf_prog *fp; - size = round_up(size, PAGE_SIZE); + size = round_up(size, __PAGE_SIZE); fp = __vmalloc(size, gfp_flags); if (fp == NULL) return NULL; From patchwork Tue Feb 6 22:04:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEB8BC48297 for ; Tue, 6 Feb 2024 22:05:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56EC46B00A3; Tue, 6 Feb 2024 17:05:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 51D0B6B00A4; Tue, 6 Feb 2024 17:05:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 370836B00A5; Tue, 6 Feb 2024 17:05:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 240186B00A3 for ; Tue, 6 Feb 2024 17:05:44 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E79AF16070A for ; Tue, 6 Feb 2024 22:05:43 +0000 (UTC) X-FDA: 81762761766.29.2FCEB99 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf21.hostedemail.com (Postfix) with ESMTP id 2115E1C001A for ; Tue, 6 Feb 2024 22:05:41 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZUXqDAHy; spf=pass (imf21.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257142; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MsJBrUUPtruHWtQsS6zgKVxqPHg8IzuE7qrpmwhQpng=; b=SUFuxw/A+3wuyKpbFScbjH/6+5JzEuueF/cdMrLiujzj/+ylDH5SWXH7fQF8PSwWu/Wy47 cg24+ek11La254oFurvtJR4+FSDycTtDbjOFFaA8yoh3Ndq+g+nFd1eJsH0T2JgMYGBB/3 4RNlW6i3yjkUAU0Iqp/sHpWGODcx7Zg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257142; a=rsa-sha256; cv=none; b=IM5ltOQXJ3evZvzPYibXAtiN8EBQZgFDSkOAUmVqTwvCSDiq04KFJWlmkUhz8JEbs/BlE1 8chAGzAGU22CNfymUhRJcsgwWYMceAwn06rH1JfG53y8M9RD/iq9hKbBCYPaE16TRI3wUB 9c1ztfzHa4gGviyXTLdwGj35FJSS9wQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ZUXqDAHy; spf=pass (imf21.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-5cdf76cde78so5486857a12.1 for ; Tue, 06 Feb 2024 14:05:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257141; x=1707861941; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MsJBrUUPtruHWtQsS6zgKVxqPHg8IzuE7qrpmwhQpng=; b=ZUXqDAHy08h8zJF2f0BbF4QaJ8RMATJHQUIp1PBgJ/gi3t8haODg17PoNkW9DOaGPW MmJ+BtX+V928mm93sr5S75iWU6ZM/emkikJQu7U45DHbCL00nBTaJ61cR9E9qtCJI1qr uOAyV5KCYO7a69CpbD7BVjI5djat7GX25613k9kgsJmYh/b+U56eQfqiu+QGvkR2MvLt 2ZIjCRq8l0ANVYNY5cOFEtRQ/Dwd1/jH1QFLAoZZ3O3kA5fZbugn/ZdZy+ROL1dJYUBP cCVCfDCTYOjrWsJkHspUrcy9pzDxnTmVui81YYspbWqztzukOLDZlxNGBrj60vNZPcWf Fo0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257141; x=1707861941; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MsJBrUUPtruHWtQsS6zgKVxqPHg8IzuE7qrpmwhQpng=; b=JISFIuhlLst88xqofLbPggyy/TC0JPaca0fT644g5RT47zJpj51GPtSi/4saXLCTUH P2fA/Gpf07QJOQ9fK/R+dIIGNacfZA+OlTyrNQUEZaxP8xLJv4wEe+XF1SzbooKcnkJb Z4idkhQWUkzocUUF3pSQOq5nVxrGfjRaHwTHDsTNo7d1mgbgof05LUxFI4BvKpseN9w0 bpUYao2WHjZmbKPWAx2DE7v7geAq9FTXHPv4wWVG8Pb+Z+eCmOtqPwDsLM9LRKvdFEUb WstzmuPUt4XQmmvCIe5W1pp4mtfu/e2i+AHuH7+AwL9ym1dHtpZoKHhUuvjpcSjNBNZx 8XSw== X-Gm-Message-State: AOJu0YydN9fo2teANouMnq7pNx1NFYqgYAThfoOVav36n9V8Xay0Mql1 PKoP07mu2AP3A4juUBY5qIy+27IgWc/AwpcqMPqDzLUVn2/F1OTV X-Google-Smtp-Source: AGHT+IGdHBHn8MfZOBdjDr1wev3fr0slDMYt9TgRf3fpt6EWXX4IriOrk3M4U7i1fJwsabnbpdMnnw== X-Received: by 2002:a05:6a21:1798:b0:19e:a1a1:5360 with SMTP id nx24-20020a056a21179800b0019ea1a15360mr1112802pzb.23.1707257141056; Tue, 06 Feb 2024 14:05:41 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCXz5G5HRMjkZjOv6ZNWk6IaBtAGKiStm6/yVXkGspRuSxmD8HC/H4mL6JLFh1XIx4+3PUuzmNG1hY1hX1FugKNofjuX0OghF/O+bK3DIKUNmz9/KnpQdtT8//WXOCO2GXI8KGJN+EmACq9yWmsUXs7bFR9BI0sxDJuQnL9vbW+jHQa5a3qDh4DEh2y7YXrvFIQ58H5MaP8nMeLs3gE0dbpirRALVTKUDLfUP/yiW6/QSk3qNwqYmu3W8DUazqg05AWtkQyQJbdVkyiE0bsIoonUbE2V78dSYFLF Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id sz14-20020a17090b2d4e00b00295b93bfb24sm7888pjb.22.2024.02.06.14.05.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:40 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 14/16] bpf: Add helper macro bpf_arena_cast() Date: Tue, 6 Feb 2024 14:04:39 -0800 Message-Id: <20240206220441.38311-15-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2115E1C001A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: rujzmts8ch9mxbifcnxpjhb3h7365zoq X-HE-Tag: 1707257141-95073 X-HE-Meta: U2FsdGVkX18t726J+I2cw8QPdOaqYvtWdkfszlXp+89gHFaJxn8fjMeJPCzLzK2LUW8p1cnCd4Yt3K7y5JCegmDHbhYqKutj9wvGJ48yzEtJE5GX9HdOtatRLXj17yd8CJfRzPaS83Ubcy6hIIpYtfyCmy2TUfpB1b/qf9RK6Hp7zcJRUrc3CbrLi7CAf+mzHpIj2vwcBDHDadm2GNcRGgtfYEIA86MUU5bn3hIxyjmTOGD3/tCUtbr1KVIlChzyuwPXPg+cH3WLSCn5rWIkGdZJy0wEbO4MIDUhvI/jK097c5a9b+HRK4Wd4cQ7uLCFA/TqX8B7doQY7MAKzwSFspbxFMh3CxgW7U5GUgKEDHFxW5NVKmiPpNbOHud0gipIoJ1MvLwi2ONDFC72gRVMLQJ4W2RXC0uLTvkdVb/BFPVoxy+BiFVjJcQUuzkoGB2kCV6I1LxLqs12St/WKs1u1GE9SCz+Z/Oxzfm2VwJ8r32ht0ad2ugiUqwVicsxiunRb3w4FQdRIJcBAy9O2N33v79MTbP4H38bs3krH786SbU1d/8ZUintXdLOty5gM/A+QxtKLNAMUCxEZWu3+CC2bejq9ib67GOMJ5Pgfc3rUE+iO1QQ8Jq6gOfEpjxHvOqydoWHa2SEh9xhgPRqsmdKy35YDDSsM7qnUwdQncLE/Qs8pMznIiQxSsdz8ZqcowaJQZduLCXXM9PRGWcJe9BbYDyBWQA2YiKcKL0uH2GV0/hZPmMH6jGr3yO79ZWXho/SDunZ/l/bFGD5GGjy5tjlu4o9EVGZewlcwKHbr/ewcneREJlbAMMktaHDlnAlZjYuFs/4n0bVoCkiyrqzFkTgs1XRYxaoCy9XluEmp2W+Nj0ZJVPXWhFXNPFERFqzJbb4XOwlNzmtvnzcXrWJv413NObJVllClobDmsFyygJrkQ6CgxcHTOa4wnO+gBvTKPFbzK6Wz1WbX3sFJiB98+h 6mjszHVz I9UuFsxqK3zCOXVYVH1Ljj3EQmKkI7GJDYXQNSteG+0cXGVCt5oQEk0n0weEElIgZR9NM321zzxcK8+JTNbcpG36B7VKa6dpNl7SslZjth3PAxXaip49FTyu7BCwLHOi+fnD/lVaxhfRYBL7Q7V8IfZ+ZyaF9vKSOILKX6b1B6EfwtzCGMVVeJPPXrcolFVLElC/QJQi4CpvSjGjpQlm35f5ePjbGfE3sm33wH8bEwfufDek4fnJz/eZfYBt/zHFT3lsmZhJtNu1SRxX0Ptwm8v9492WePt8TMnPzDm7vpBnMdH7/Aw+CW5uRj7TXNvXYtIBWmaXh5xu234sSLlsqBUp0q9rzkOCrbJjx8ZO2FwOw5dGp56tkAakIHX4vMT4dP5hxtr7l/HvFdbIh62P//qZMEop9760KWZTyFdW8Djwynt9zaeHi91cmRQtd39sr/ya3yUaCFm/lFpbkKDribs0QICJrtaZ2a/xsYRjh7y1f4feANqlybKOk3EQwfe3tcbM9i7PWg2E9yC9D355HhyOND1F+BA0Dnla8FTHNtC9MCckF2oJTYAv1zHvnvU4MmVIrQCNOUitPtK6PNxFPbiRaCLi/YvW4U05O X-Bogosity: Ham, tests=bogofilter, spamicity=0.000296, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Introduce helper macro bpf_arena_cast() that emits: rX = rX instruction with off = BPF_ARENA_CAST_KERN or off = BPF_ARENA_CAST_USER and encodes address_space into imm32. It's useful with older LLVM that doesn't emit this insn automatically. Signed-off-by: Alexei Starovoitov --- .../testing/selftests/bpf/bpf_experimental.h | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 0d749006d107..e73b7d48439f 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -331,6 +331,47 @@ l_true: \ asm volatile("%[reg]=%[reg]"::[reg]"r"((short)var)) #endif +/* emit instruction: rX=rX .off = mode .imm32 = address_space */ +#ifndef bpf_arena_cast +#define bpf_arena_cast(var, mode, addr_space) \ + ({ \ + typeof(var) __var = var; \ + asm volatile(".byte 0xBF; \ + .ifc %[reg], r0; \ + .byte 0x00; \ + .endif; \ + .ifc %[reg], r1; \ + .byte 0x11; \ + .endif; \ + .ifc %[reg], r2; \ + .byte 0x22; \ + .endif; \ + .ifc %[reg], r3; \ + .byte 0x33; \ + .endif; \ + .ifc %[reg], r4; \ + .byte 0x44; \ + .endif; \ + .ifc %[reg], r5; \ + .byte 0x55; \ + .endif; \ + .ifc %[reg], r6; \ + .byte 0x66; \ + .endif; \ + .ifc %[reg], r7; \ + .byte 0x77; \ + .endif; \ + .ifc %[reg], r8; \ + .byte 0x88; \ + .endif; \ + .ifc %[reg], r9; \ + .byte 0x99; \ + .endif; \ + .short %[off]; .long %[as]" \ + :: [reg]"r"(__var), [off]"i"(mode), [as]"i"(addr_space)); __var; \ + }) +#endif + /* Description * Assert that a conditional expression is true. * Returns From patchwork Tue Feb 6 22:04:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC71BC48297 for ; Tue, 6 Feb 2024 22:05:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E6716B00A5; Tue, 6 Feb 2024 17:05:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3965D6B00A6; Tue, 6 Feb 2024 17:05:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C2BA6B00A7; Tue, 6 Feb 2024 17:05:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E95FF6B00A5 for ; Tue, 6 Feb 2024 17:05:47 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C254C1A0ACF for ; Tue, 6 Feb 2024 22:05:47 +0000 (UTC) X-FDA: 81762761934.22.69DCC63 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf18.hostedemail.com (Postfix) with ESMTP id 161A71C0004 for ; Tue, 6 Feb 2024 22:05:45 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="cERWB/ej"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257146; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LE9MS9eGwDxxFPhO8TsHXptAYT9bfyhTWbVoEpeoqYo=; b=IVjF4OTSjH5d713aK1atsVxHIv3pFGd1bw6nhkEcG5w9fCLaEZF/SxziN9bpvI1SGaAdDB v11EBs/1rovAK1MYIDIPUdKZQCuAYG8T/rly+akFP+uv95v8yZcFIhsA1JJDHjPuCAKOau G7lV38OrUGrfI5fQ+cmjNfsFao/zYGs= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="cERWB/ej"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257146; a=rsa-sha256; cv=none; b=lYUAPYsNL7A0S2bgRuP9ZiEiMl39R2TyoFJJgCo4Kq09UKfVqOKBjVK5i80idDWwAZyOHc AC3LGQdtm8Em2drPPNvn8ZsmArijrePHISxOkAXsdrJhGTeuL0amWGyjDMsX/jRD5P1TLI o++GmOzaFxxCB5Ph7G5Ei13kYQCaKjA= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d751bc0c15so57825ad.2 for ; Tue, 06 Feb 2024 14:05:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257145; x=1707861945; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LE9MS9eGwDxxFPhO8TsHXptAYT9bfyhTWbVoEpeoqYo=; b=cERWB/ejAafQj5tqpQKPHZ0RY19lMFvWv+Ife5A17AaaFTxKrHbWFL7YB4DipNtj1E 4GXUdWzVVMxqnTnfsKdWtLsBKaEVn15+svhqWIlkY1DoRH96Xoo5HZTJFMgfqpU9Ukwx yXDQ/vP+zJS6101Ap/riMZc60A8xvBSCfrfgTbhZUVQU3VhSHQOs4EUPk/JXedgD1pwe TFTVz2R5jAz9gSEuiua5siA7XsXtYuuTKpjMqJBmxkMJpRzKuR7ucNOqjPQC+rBSvX76 gdWyxn4YXsOfaWhCg/EV3lPLG9dbIe+K7UAmT5RwGeFT3/DYKEweMil7H8TKzqyR/K9C kmIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257145; x=1707861945; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LE9MS9eGwDxxFPhO8TsHXptAYT9bfyhTWbVoEpeoqYo=; b=JrQFvvy/O0I17ywsHBphLSw8rvNiNmxeerJ/xYFrxSQOdFmrE6QPas9w647zFFxIMg M/GEqCRE5pgSKQtjJeZVSXHAmrCAbcDVZbpvt7PGPHEq/SlmNP7vlKbbbxvATQ8STYbA QulF4cTSsCJDCed5tATieKNdRfnGmYaGB5dxB0l9dPz1GKBecVrAtEO2/Q7HhQKPQtgm ncOaRPJX4HsLbC5tfBf/xZn74ROS3X587KwBw5zjgVIj4azosDK0LpD8etqMgbSb4Lm0 qHLndB9O2uEQS+mWHRNcKGrEhTMPgYDHCKg2AZ86ezLtbnbL2YStgYQt6bVLlWctzOae rQhA== X-Gm-Message-State: AOJu0YwvSoOvI/Ro+McXW0txRsUt+WdA1mbkAiKAjrPKE2h81DBd6Kj8 F35RyesiONM/pYpyvT654+FizUzlMMqmRfcabvCHg8g+D1xDgxcQ X-Google-Smtp-Source: AGHT+IGG4t9RdwkiKkRY2M19m/mS/AAjGHnAhoQCng8LQI6dq4CJB9Jc9qt3c31BNL7jwZ3IWRcjFQ== X-Received: by 2002:a17:902:7448:b0:1d9:bbc2:87e7 with SMTP id e8-20020a170902744800b001d9bbc287e7mr2885407plt.36.1707257144956; Tue, 06 Feb 2024 14:05:44 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVEDKC/l6eMEMtNlzEQS5nYwT0OU5E3cdDptT/bk60YVGSKo/zQyR2ggEBvMoXC+IGYAYwNexnNHzYKcK6Sy1nflFapKyb6QjD9fh6dKatzP9UaH207UZqQkqxAPhLpJ9gE4n8DSenkdqE6Hx7xzVlkC0yZzzMBg+zbd00GLiEiyZE3ZCbRK1jHdn2TTU4BYf91E/uepkReQz42tma3AhYwsh1vxCHafwJkY55srTZRYvlXGxlMA83+0cIWjdTdB8SDJPrz9RM8r0kHkKlT/LYjXrqgBXOFJ2Dl Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id h4-20020a170902748400b001d8e5a3be8asm1277pll.259.2024.02.06.14.05.43 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:44 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 15/16] selftests/bpf: Add bpf_arena_list test. Date: Tue, 6 Feb 2024 14:04:40 -0800 Message-Id: <20240206220441.38311-16-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 161A71C0004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 974o6ex8kouuqyarz6ftkzrfwfq6w1pi X-HE-Tag: 1707257145-474168 X-HE-Meta: U2FsdGVkX1/WZoPYi3XMbykaoNkuFTqKMifwHjlu26VS8QJWxDP+HkMkC7OvC1zznYF+D1ZkAqTf1nI6s6e8Zs+OgBV7y2UDXiW8rKsMv2fvn05WlD+MYFyuM99nP2PubZqsZtBhTdt/OB5g+UasC8X7Z68SxMC9qT55eafbeSzv3S6KwDhWpIADtFn6sKr+qoIJpLf4fyChx8mejQHAUa75jCCZa1/UVZoQ/FuOBHb5w+a/yuIRHf30ojNxzYDgeJXTapFmzjWgdZEF5GT/hktGKEnKFfCTSB1BRE0SqGvvFgjXhzuvsWUCV/VtbzPzFQrvMXEkNTR71eccdNu4IQAGmey2CIq9W5cMpfLzm4hX7/u4rzEFJjyXxqptsyeb1q41budt6yxzmcU1q8WX6R9Bp00kUva7uP7m97dBg6RKP0BVLjroqliJ5PF8ZJ7EB2NkgKTOXMIK29fgfJrE+XSWTErLShETjwnHlhJoTtl57QvFIetZMJEtEVvaG7DCe+zVu8JVqD3GkmbDKJOPYzjj2IHMtGvJN1hTASAsC7s4cbo7U0InbDSjyhxEuTBz/5SUCOBFkxxG2myRouS0Hgc/0EMaBaXtkj4En7P61pNRdLWvVnk1vEXL82mpP13EOoLnMBCSdsAWAXVcAOuo22Haf1WMxdETpyvgXWIfOp5QivpahJ73BWPrNuOX7W/XDwCzMXDv6pL9+hdKy1HkTrTkdAXmjkdfQlUv7DcBsIBmNoydxeI+fd6XCaN0ZK8/xPBMj+yReW0tAiMedEKxtd9wyQrlHpAp1085q1ZrwkTi/X39SidZ6w+pYvEH9cYDdLjQR/xF9kAayt9V4eFV14bEbDUvNe9/70tWzUtC+E0i8b305Pa39f+kZI6m7j+VsM6ahfBw7B4rJiUD73QgrXjO2kXGf10Ab45/XJvaMx+K/PAzb7klOe1FOaWSUh8IOncY0PrG8oWexAKbJ6d CvIHm6ns +zPU03YE1OL+PIkbKfP3TZ6okK3j3eBj5xZ0lpeczVJ1rjb59wxRklq9sAwn9o+6vMbk8CkZF8pU6u7IkK4530ovyjGSelAMStBzInY/WwdqM/Adm+YjdkS3qLDTC2U5TQutbgONg7x/AOA7VkPAyknDHy7JyQ/lIMX+a0++fmZmgh0RmcPCcnNp1+nUZqnoq30lV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov bpf_arena_common.h - common helpers and macros bpf_arena_alloc.h - implements page_frag allocator as a bpf program. bpf_arena_list.h - doubly linked link list as a bpf program. Compiled as a bpf program and as native C code. Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + tools/testing/selftests/bpf/bpf_arena_alloc.h | 58 +++++++++++ .../testing/selftests/bpf/bpf_arena_common.h | 70 ++++++++++++++ tools/testing/selftests/bpf/bpf_arena_list.h | 95 +++++++++++++++++++ .../selftests/bpf/prog_tests/arena_list.c | 65 +++++++++++++ .../testing/selftests/bpf/progs/arena_list.c | 75 +++++++++++++++ 7 files changed, 365 insertions(+) create mode 100644 tools/testing/selftests/bpf/bpf_arena_alloc.h create mode 100644 tools/testing/selftests/bpf/bpf_arena_common.h create mode 100644 tools/testing/selftests/bpf/bpf_arena_list.h create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_list.c create mode 100644 tools/testing/selftests/bpf/progs/arena_list.c diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 5c2cc7e8c5d0..7759cff95b6f 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -11,3 +11,4 @@ fill_link_info/kprobe_multi_link_info # bpf_program__attach_kprobe_mu fill_link_info/kretprobe_multi_link_info # bpf_program__attach_kprobe_multi_opts unexpected error: -95 fill_link_info/kprobe_multi_invalid_ubuff # bpf_program__attach_kprobe_multi_opts unexpected error: -95 missed/kprobe_recursion # missed_kprobe_recursion__attach unexpected error: -95 (errno 95) +arena # JIT does not support arena diff --git a/tools/testing/selftests/bpf/DENYLIST.s390x b/tools/testing/selftests/bpf/DENYLIST.s390x index 1a63996c0304..11f7b612f967 100644 --- a/tools/testing/selftests/bpf/DENYLIST.s390x +++ b/tools/testing/selftests/bpf/DENYLIST.s390x @@ -3,3 +3,4 @@ exceptions # JIT does not support calling kfunc bpf_throw (exceptions) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?) +arena # JIT does not support arena diff --git a/tools/testing/selftests/bpf/bpf_arena_alloc.h b/tools/testing/selftests/bpf/bpf_arena_alloc.h new file mode 100644 index 000000000000..0f4cb399b4c7 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_alloc.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once +#include "bpf_arena_common.h" + +#ifndef __round_mask +#define __round_mask(x, y) ((__typeof__(x))((y)-1)) +#endif +#ifndef round_up +#define round_up(x, y) ((((x)-1) | __round_mask(x, y))+1) +#endif + +void __arena *cur_page; +int cur_offset; + +/* Simple page_frag allocator */ +static inline void __arena* bpf_alloc(unsigned int size) +{ + __u64 __arena *obj_cnt; + void __arena *page = cur_page; + int offset; + + size = round_up(size, 8); + if (size >= PAGE_SIZE - 8) + return NULL; + if (!page) { +refill: + page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0); + if (!page) + return NULL; + cast_kern(page); + cur_page = page; + cur_offset = PAGE_SIZE - 8; + obj_cnt = page + PAGE_SIZE - 8; + *obj_cnt = 0; + } else { + cast_kern(page); + obj_cnt = page + PAGE_SIZE - 8; + } + + offset = cur_offset - size; + if (offset < 0) + goto refill; + + (*obj_cnt)++; + cur_offset = offset; + return page + offset; +} + +static inline void bpf_free(void __arena *addr) +{ + __u64 __arena *obj_cnt; + + addr = (void __arena *)(((long)addr) & ~(PAGE_SIZE - 1)); + obj_cnt = addr + PAGE_SIZE - 8; + if (--(*obj_cnt) == 0) + bpf_arena_free_pages(&arena, addr, 1); +} diff --git a/tools/testing/selftests/bpf/bpf_arena_common.h b/tools/testing/selftests/bpf/bpf_arena_common.h new file mode 100644 index 000000000000..07849d502f40 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_common.h @@ -0,0 +1,70 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once + +#ifndef WRITE_ONCE +#define WRITE_ONCE(x, val) ((*(volatile typeof(x) *) &(x)) = (val)) +#endif + +#ifndef NUMA_NO_NODE +#define NUMA_NO_NODE (-1) +#endif + +#ifndef arena_container_of +#define arena_container_of(ptr, type, member) \ + ({ \ + void __arena *__mptr = (void __arena *)(ptr); \ + ((type *)(__mptr - offsetof(type, member))); \ + }) +#endif + +#ifdef __BPF__ /* when compiled as bpf program */ + +#ifndef PAGE_SIZE +#define PAGE_SIZE __PAGE_SIZE +/* + * for older kernels try sizeof(struct genradix_node) + * or flexible: + * static inline long __bpf_page_size(void) { + * return bpf_core_enum_value(enum page_size_enum___l, __PAGE_SIZE___l) ?: sizeof(struct genradix_node); + * } + * but generated code is not great. + */ +#endif + +#if defined(__BPF_FEATURE_ARENA_CAST) && !defined(BPF_ARENA_FORCE_ASM) +#define __arena __attribute__((address_space(1))) +#define cast_kern(ptr) /* nop for bpf prog. emitted by LLVM */ +#define cast_user(ptr) /* nop for bpf prog. emitted by LLVM */ +#else +#define __arena +#define cast_kern(ptr) bpf_arena_cast(ptr, BPF_ARENA_CAST_KERN, 1) +#define cast_user(ptr) bpf_arena_cast(ptr, BPF_ARENA_CAST_USER, 1) +#endif + +void __arena* bpf_arena_alloc_pages(void *map, void __arena *addr, __u32 page_cnt, + int node_id, __u64 flags) __ksym __weak; +void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) __ksym __weak; + +#else /* when compiled as user space code */ + +#define __arena +#define __arg_arena +#define cast_kern(ptr) /* nop for user space */ +#define cast_user(ptr) /* nop for user space */ +__weak char arena[1]; + +#ifndef offsetof +#define offsetof(type, member) ((unsigned long)&((type *)0)->member) +#endif + +static inline void __arena* bpf_arena_alloc_pages(void *map, void *addr, __u32 page_cnt, + int node_id, __u64 flags) +{ + return NULL; +} +static inline void bpf_arena_free_pages(void *map, void __arena *ptr, __u32 page_cnt) +{ +} + +#endif diff --git a/tools/testing/selftests/bpf/bpf_arena_list.h b/tools/testing/selftests/bpf/bpf_arena_list.h new file mode 100644 index 000000000000..9f34142b0f65 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_list.h @@ -0,0 +1,95 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once +#include "bpf_arena_common.h" + +struct arena_list_node; + +typedef struct arena_list_node __arena arena_list_node_t; + +struct arena_list_node { + arena_list_node_t *next; + arena_list_node_t * __arena *pprev; +}; + +struct arena_list_head { + struct arena_list_node __arena *first; +}; +typedef struct arena_list_head __arena arena_list_head_t; + +#define list_entry(ptr, type, member) arena_container_of(ptr, type, member) + +#define list_entry_safe(ptr, type, member) \ + ({ typeof(*ptr) * ___ptr = (ptr); \ + ___ptr ? ({ cast_kern(___ptr); list_entry(___ptr, type, member); }) : NULL; \ + }) + +#ifndef __BPF__ +static inline void *bpf_iter_num_new(struct bpf_iter_num *, int, int) { return NULL; } +static inline void bpf_iter_num_destroy(struct bpf_iter_num *) {} +static inline bool bpf_iter_num_next(struct bpf_iter_num *) { return true; } +#endif + +/* Safely walk link list of up to 1M elements. Deletion of elements is allowed. */ +#define list_for_each_entry(pos, head, member) \ + for (struct bpf_iter_num ___it __attribute__((aligned(8), \ + cleanup(bpf_iter_num_destroy))), \ + * ___tmp = ( \ + bpf_iter_num_new(&___it, 0, (1000000)), \ + pos = list_entry_safe((head)->first, \ + typeof(*(pos)), member), \ + (void)bpf_iter_num_destroy, (void *)0); \ + bpf_iter_num_next(&___it) && pos && \ + ({ ___tmp = (void *)pos->member.next; 1; }); \ + pos = list_entry_safe((void __arena *)___tmp, typeof(*(pos)), member)) + +static inline void list_add_head(arena_list_node_t *n, arena_list_head_t *h) +{ + arena_list_node_t *first = h->first, * __arena *tmp; + + cast_user(first); + cast_kern(n); + WRITE_ONCE(n->next, first); + cast_kern(first); + if (first) { + tmp = &n->next; + cast_user(tmp); + WRITE_ONCE(first->pprev, tmp); + } + cast_user(n); + WRITE_ONCE(h->first, n); + + tmp = &h->first; + cast_user(tmp); + cast_kern(n); + WRITE_ONCE(n->pprev, tmp); +} + +static inline void __list_del(arena_list_node_t *n) +{ + arena_list_node_t *next = n->next, *tmp; + arena_list_node_t * __arena *pprev = n->pprev; + + cast_user(next); + cast_kern(pprev); + tmp = *pprev; + cast_kern(tmp); + WRITE_ONCE(tmp, next); + if (next) { + cast_user(pprev); + cast_kern(next); + WRITE_ONCE(next->pprev, pprev); + } +} + +#define POISON_POINTER_DELTA 0 + +#define LIST_POISON1 ((void __arena *) 0x100 + POISON_POINTER_DELTA) +#define LIST_POISON2 ((void __arena *) 0x122 + POISON_POINTER_DELTA) + +static inline void list_del(arena_list_node_t *n) +{ + __list_del(n); + n->next = LIST_POISON1; + n->pprev = LIST_POISON2; +} diff --git a/tools/testing/selftests/bpf/prog_tests/arena_list.c b/tools/testing/selftests/bpf/prog_tests/arena_list.c new file mode 100644 index 000000000000..ca3ce8abefc4 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/arena_list.c @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include + +#define PAGE_SIZE 4096 + +#include "bpf_arena_list.h" +#include "arena_list.skel.h" + +struct elem { + struct arena_list_node node; + __u64 value; +}; + +static int list_sum(struct arena_list_head *head) +{ + struct elem __arena *n; + int sum = 0; + + list_for_each_entry(n, head, node) + sum += n->value; + return sum; +} + +static void test_arena_list_add_del(int cnt) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct arena_list *skel; + int expected_sum = (u64)cnt * (cnt - 1) / 2; + int ret, sum; + + skel = arena_list__open_and_load(); + if (!ASSERT_OK_PTR(skel, "arena_list__open_and_load")) + return; + + skel->bss->cnt = cnt; + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_add), &opts); + ASSERT_OK(ret, "ret_add"); + ASSERT_OK(opts.retval, "retval"); + if (skel->bss->skip) { + printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__); + test__skip(); + goto out; + } + sum = list_sum(skel->bss->list_head); + ASSERT_EQ(sum, expected_sum, "sum of list elems"); + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_list_del), &opts); + ASSERT_OK(ret, "ret_del"); + sum = list_sum(skel->bss->list_head); + ASSERT_EQ(sum, 0, "sum of list elems after del"); + ASSERT_EQ(skel->bss->list_sum, expected_sum, "sum of list elems computed by prog"); +out: + arena_list__destroy(skel); +} + +void test_arena_list(void) +{ + if (test__start_subtest("arena_list_1")) + test_arena_list_add_del(1); + if (test__start_subtest("arena_list_1000")) + test_arena_list_add_del(1000); +} diff --git a/tools/testing/selftests/bpf/progs/arena_list.c b/tools/testing/selftests/bpf/progs/arena_list.c new file mode 100644 index 000000000000..1acdec9dadde --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_list.c @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include "bpf_experimental.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 1u << 24); /* max_entries * value_size == size of mmap() region */ + __ulong(map_extra, 2ull << 44); /* start of mmap() region */ + __type(key, __u64); + __type(value, __u64); +} arena SEC(".maps"); + +#include "bpf_arena_alloc.h" +#include "bpf_arena_list.h" + +struct elem { + struct arena_list_node node; + __u64 value; +}; + +struct arena_list_head __arena *list_head; +int list_sum; +int cnt; +bool skip = false; + +SEC("syscall") +int arena_list_add(void *ctx) +{ +#ifdef __BPF_FEATURE_ARENA_CAST + __u64 i; + + list_head = bpf_alloc(sizeof(*list_head)); + + bpf_for(i, 0, cnt) { + struct elem __arena *n = bpf_alloc(sizeof(*n)); + + n->value = i; + list_add_head(&n->node, list_head); + } +#else + skip = true; +#endif + return 0; +} + +SEC("syscall") +int arena_list_del(void *ctx) +{ +#ifdef __BPF_FEATURE_ARENA_CAST + struct elem __arena *n; + int sum = 0; + + list_for_each_entry(n, list_head, node) { + sum += n->value; + list_del(&n->node); + bpf_free(n); + } + list_sum = sum; + + /* triple free will not crash the kernel */ + bpf_free(list_head); + bpf_free(list_head); + bpf_free(list_head); +#else + skip = true; +#endif + return 0; +} + +char _license[] SEC("license") = "GPL"; From patchwork Tue Feb 6 22:04:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13547851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72AFDC4828D for ; Tue, 6 Feb 2024 22:05:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0171D6B00A7; Tue, 6 Feb 2024 17:05:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE1246B00A8; Tue, 6 Feb 2024 17:05:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0CEC6B00A9; Tue, 6 Feb 2024 17:05:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B95646B00A7 for ; Tue, 6 Feb 2024 17:05:51 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 94D921C1047 for ; Tue, 6 Feb 2024 22:05:51 +0000 (UTC) X-FDA: 81762762102.22.AB24FDD Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf29.hostedemail.com (Postfix) with ESMTP id BBA6312000C for ; Tue, 6 Feb 2024 22:05:49 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EDCzCIb6; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707257149; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z5sXl6wfPkVpJxXcJSxvXuOvb+1sMHHR6BvnXTDIECA=; b=nVmn/+ADqSaVCTPShLvMfBV/i9OqQhwpFKAl1HIiWisnep2oUKCoEb3NY/CrSCCk7qmKqs FWINSJhUGt7IJJD+SLp2drB3NE3bVQFKQ0WQZP7kE+C4Sbz9FI+SoduyGXW5MNeLXuOY4S BeYRiH8NSVJuDCzDOUD/KpPBDF44ug8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707257149; a=rsa-sha256; cv=none; b=yTQbANnJ4C0gUG3ddNPSauM0BwkdjomyMo3UGP3f8pCBFmsqCo6JgGhh4uYvWIEndpk4uc OFum1AkbnEPRAQQf+16HdwyIjj2X9aH/FyG61biW69pXJ0W0hYy+cekKJQwBvmjDWgw4CS jGS9/0iheS/FmOqeH1W+eiOiRs4bnoc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EDCzCIb6; spf=pass (imf29.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1d93edfa76dso135495ad.1 for ; Tue, 06 Feb 2024 14:05:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1707257149; x=1707861949; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z5sXl6wfPkVpJxXcJSxvXuOvb+1sMHHR6BvnXTDIECA=; b=EDCzCIb657eGigimPthC0rMybLVFId5JHq4ZRVJLUDKTugUzmjNEqYuj+3Cy6otWcL DEuC3Hf9uA7rqHMp4/hKV26dcE5d8GlASUSXT/V/lCbx0HGH8kW1jCGXd6nZRDSMqBWg k0DaMJ4C/tZa4ISRZD82J6fozP580JfkGepBpk3rMTXwwE/T4F91hJIFd4wSi7pRqNzk jQFz9d7AA6kVOzffX6Qkj9BC3ueirlJhdR5M5ruf3F/eAppT6nJKuDM8otxToqk7zgTr WSgsux4UxrtetxP8eHQWnu4qiYFmVrAmbrVfxGaF10w8G7SIKl/keTzzyOpGsqvDO3XB lAIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707257149; x=1707861949; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z5sXl6wfPkVpJxXcJSxvXuOvb+1sMHHR6BvnXTDIECA=; b=fBgIFiSw+pPuK5dUbxo7gpBQNzaTffZAykr3SwYlZoYrrBTJmghPnPtKo+AFJ4X6RP tAeSKeS3gp50Qt+OAMu6SE2Pgsft7yfD0joGBAkxHEC2OnFTEMgGdKP25ROreqeZw/iu hq8dPebaHNOGRcFFMc7utGOY1CDzJBwSB61PuvLQTIXkqWHqc3z4iHKinBStkjkDwCss d/heu6jLGS6PQjOhpJXcK6922UC7Oi/757fnBkSDZJ0ZcSReqIDXRY9KlZHYAYjM53Tx cTlV53uULz0lCN8ft241rWHrPPlIR+qjyKIUJoZ2bupo/pBI87KSGZOtbMHOGbwh0uxv CXtQ== X-Gm-Message-State: AOJu0YwKx42nTawkCua4LlxTFmxDGLCfaNiTBTmxNFgGWliURDl10kIR CxvA7KMWg5ZL8rb7lBPKq/NCHJJv6DWJQSvCqNuzWQqo3wCdyv07 X-Google-Smtp-Source: AGHT+IF9U+BtdYCt6M8dYZtgQt/dJeYpcJAyzhX1n//KSQC1NX0NDBwdE7WV9Rte853ceTYT7aKzoQ== X-Received: by 2002:a17:902:650f:b0:1d9:edf5:c86b with SMTP id b15-20020a170902650f00b001d9edf5c86bmr59997plk.67.1707257148740; Tue, 06 Feb 2024 14:05:48 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCUwrnDaQTz88yxH7R10WszU4AcGQqpT0AnT3dwaaN4OHwpmdUYYkayQ0PtrkBsBSO9Wjx5ZBjmxmqDBYbHr2gYsJUzr8QjyyEEQvgAvG79SyCGECLR+KK1BoD8xCyc5PEBoX01aS0vrV5oCmEn0oAq8Cv79v4fg0YQg3/5TU0uAa9YA/zGgqQufx4ZMBU3oiWF3pbssnjT3Rt3yOG3GlgiyLxeTvaM8xicjLTdO62TKE10RjLGBtI7VQRG0Ytm9FHSObaQjTVnSsh5LfPEi9eM4mstvz9dj7/i0 Received: from localhost.localdomain ([2620:10d:c090:400::4:27bf]) by smtp.gmail.com with ESMTPSA id r1-20020a170903410100b001d8f81ecebesm3718pld.192.2024.02.06.14.05.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 06 Feb 2024 14:05:48 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, memxor@gmail.com, eddyz87@gmail.com, tj@kernel.org, brho@google.com, hannes@cmpxchg.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next 16/16] selftests/bpf: Add bpf_arena_htab test. Date: Tue, 6 Feb 2024 14:04:41 -0800 Message-Id: <20240206220441.38311-17-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-145) In-Reply-To: <20240206220441.38311-1-alexei.starovoitov@gmail.com> References: <20240206220441.38311-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BBA6312000C X-Rspam-User: X-Stat-Signature: ig7ieo7qdx3dikcbexsr9shwxj4g7tf9 X-Rspamd-Server: rspam03 X-HE-Tag: 1707257149-160980 X-HE-Meta: U2FsdGVkX1+zuOoJYjDPXJUnYlZOnZ8CD4D7IlTGb9y/3miX02K+QPawue954wtKSnmauuGOcVUgU1OHydN209hVkZCgSTpk3L/vLTFL9m7GSF0SoqOEs8J3N69tK1Ft2Df7gBv2V99umk39BcluJjXvkDWkD/hKw9eMWd5RKUZHGr3MCW/iRBFo9Bb31qFwstpQ6Qtf/2uLtbllgeevgM3E/3hvqX+U7w92AqWr+v1ofYzimiCabInZbLXosfhZOW8f38YJU8y7IBXKzGidzU5XO2tGUr5Y5pZs+vZmseKJv2X8Yws7ZKltZOM27jCRzdn5xGmpPDC6lQ1BSY9I4s98E4YUCfo875z8z/x9PX1yBLK7ylmiwXOvjMxudCKsYfSEI5ESsJNNLzoMyXHWtL6elmY/qf2jpCNfUYeb0Okf8JvNw4bnROCPWSMzU0XUU/NaQam5fsyR6SKb0LUCzypAJfOfFBCXbVGD9V33hsSaM+fY5WcSXpQ2OmP4rMwZirrxkJszjZOoFrJdED9sB/k739U4ECZY82SG/675FVtufhRLMPEeyBNfJPpQSATcwHIwyNdFH6zm7Sr9w2261Pk+RY6LaU3SjUGLRRFFHympRrsASN0Uu0a3l0vDDmvqFEQmao75J1rx7HBlbiMNKWHRasDtoVV+HdRECCWMbYBJiQv4SSi5ysaOphpgDsHTOLyUSQjqq7QCIbUEpFal942USlYoxFi3PQXDmIduzpl+MMjc2aP9+RN97/r60GuXvl56RiuZA/4g1tBL5rWXCczAbS1mzYA/RP2uz2jkZ82EfTkSivj7OpAkqrCwKlf29X+A4vUBKZsTYXYFL/sxLyLTKky9vFjCCiZ+d4TrnV76+YPkcHfdEJEY/XOm+vm9Xlqny2ri5YgjqJR+WraH8Pe/jugFHS0Kw0xkKzE0AYIYtOqIS1rIs1BDWqbF4kI7w7yw2eN/T0A3uTesZAb bIpmkRf0 G8suS6wRG53Ao9D7n7cPBYu4M3kZY4oZdf7C7Mxy6/XVnCRgGsHD4ysuxM1zHX/gW75sjRXjgHGd7ZUB5BFpHklRhcEg0mYVA1RC4lJHDvUL7X6yxzD+xp2Dx62GzI3CwFAOpodhDlkIYJ9tSSsaqTLfIUghqLja1V0C6gAMUvrHFYpeRHJIPDfd5GokJo329FGZMM+MgrAJNH4p8jwcCdoADG18MruUovWRttXKrWZ7h2QIpkmbI0z6Wm7l7vcf7enmccXMzCYlMR5XaZcG+TkLi6x6b8muFYpQFfhS/H3yfhKjDKsDIToWoI4ecqdtBTpmssrt9JKrMNYd1UAq3D3q4Ey/DHR8SndPqEe8dXIcLKgbLV61QttCOWLpHDLVZ4AM3A3jFC1bDs45JNcOu9EBz+10d/oHFqlUS3P054eQ2ZcPxKEf4i/BRncuL6XyE5wxBuhrHAnPEwWQll3lxjuY9RkfxCcvBgePD4GU1oFHq4IyER4Pl4Sdz6hIYxIHBezNJPnqDkvoK/0m3fA3gM9SaRjCn/gOMwuP8XD0/NT+WtlrcYYiEhkwhFgaUkf1H+E2heQquF2/Wl/cuut6CfJGBocDhTSmuN76V X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov bpf_arena_htab.h - hash table implemented as bpf program Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/bpf_arena_htab.h | 100 ++++++++++++++++++ .../selftests/bpf/prog_tests/arena_htab.c | 88 +++++++++++++++ .../testing/selftests/bpf/progs/arena_htab.c | 48 +++++++++ .../selftests/bpf/progs/arena_htab_asm.c | 5 + 4 files changed, 241 insertions(+) create mode 100644 tools/testing/selftests/bpf/bpf_arena_htab.h create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_htab.c create mode 100644 tools/testing/selftests/bpf/progs/arena_htab.c create mode 100644 tools/testing/selftests/bpf/progs/arena_htab_asm.c diff --git a/tools/testing/selftests/bpf/bpf_arena_htab.h b/tools/testing/selftests/bpf/bpf_arena_htab.h new file mode 100644 index 000000000000..acc01a876668 --- /dev/null +++ b/tools/testing/selftests/bpf/bpf_arena_htab.h @@ -0,0 +1,100 @@ +/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */ +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#pragma once +#include +#include "bpf_arena_alloc.h" +#include "bpf_arena_list.h" + +struct htab_bucket { + struct arena_list_head head; +}; +typedef struct htab_bucket __arena htab_bucket_t; + +struct htab { + htab_bucket_t *buckets; + int n_buckets; +}; +typedef struct htab __arena htab_t; + +static inline htab_bucket_t *__select_bucket(htab_t *htab, __u32 hash) +{ + htab_bucket_t *b = htab->buckets; + + cast_kern(b); + return &b[hash & (htab->n_buckets - 1)]; +} + +static inline arena_list_head_t *select_bucket(htab_t *htab, __u32 hash) +{ + return &__select_bucket(htab, hash)->head; +} + +struct hashtab_elem { + int hash; + int key; + int value; + struct arena_list_node hash_node; +}; +typedef struct hashtab_elem __arena hashtab_elem_t; + +static hashtab_elem_t *lookup_elem_raw(arena_list_head_t *head, __u32 hash, int key) +{ + hashtab_elem_t *l; + + list_for_each_entry(l, head, hash_node) + if (l->hash == hash && l->key == key) + return l; + + return NULL; +} + +static int htab_hash(int key) +{ + return key; +} + +__weak int htab_lookup_elem(htab_t *htab __arg_arena, int key) +{ + hashtab_elem_t *l_old; + arena_list_head_t *head; + + cast_kern(htab); + head = select_bucket(htab, key); + l_old = lookup_elem_raw(head, htab_hash(key), key); + if (l_old) + return l_old->value; + return 0; +} + +__weak int htab_update_elem(htab_t *htab __arg_arena, int key, int value) +{ + hashtab_elem_t *l_new = NULL, *l_old; + arena_list_head_t *head; + + cast_kern(htab); + head = select_bucket(htab, key); + l_old = lookup_elem_raw(head, htab_hash(key), key); + + l_new = bpf_alloc(sizeof(*l_new)); + if (!l_new) + return -ENOMEM; + l_new->key = key; + l_new->hash = htab_hash(key); + l_new->value = value; + + list_add_head(&l_new->hash_node, head); + if (l_old) { + list_del(&l_old->hash_node); + bpf_free(l_old); + } + return 0; +} + +void htab_init(htab_t *htab) +{ + void __arena *buckets = bpf_arena_alloc_pages(&arena, NULL, 2, NUMA_NO_NODE, 0); + + cast_user(buckets); + htab->buckets = buckets; + htab->n_buckets = 2 * PAGE_SIZE / sizeof(struct htab_bucket); +} diff --git a/tools/testing/selftests/bpf/prog_tests/arena_htab.c b/tools/testing/selftests/bpf/prog_tests/arena_htab.c new file mode 100644 index 000000000000..0766702de846 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/arena_htab.c @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include + +#include "arena_htab_asm.skel.h" +#include "arena_htab.skel.h" + +#define PAGE_SIZE 4096 + +#include "bpf_arena_htab.h" + +static void test_arena_htab_common(struct htab *htab) +{ + int i; + + printf("htab %p buckets %p n_buckets %d\n", htab, htab->buckets, htab->n_buckets); + ASSERT_OK_PTR(htab->buckets, "htab->buckets shouldn't be NULL"); + for (i = 0; htab->buckets && i < 16; i += 4) { + /* + * Walk htab buckets and link lists since all pointers are correct, + * though they were written by bpf program. + */ + int val = htab_lookup_elem(htab, i); + + ASSERT_EQ(i, val, "key == value"); + } +} + +static void test_arena_htab_llvm(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct arena_htab *skel; + struct htab *htab; + size_t arena_sz; + void *area; + int ret; + + skel = arena_htab__open_and_load(); + if (!ASSERT_OK_PTR(skel, "arena_htab__open_and_load")) + return; + + area = bpf_map__initial_value(skel->maps.arena, &arena_sz); + /* fault-in a page with pgoff == 0 as sanity check */ + *(volatile int *)area = 0x55aa; + + /* bpf prog will allocate more pages */ + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_htab_llvm), &opts); + ASSERT_OK(ret, "ret"); + ASSERT_OK(opts.retval, "retval"); + if (skel->bss->skip) { + printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__); + test__skip(); + goto out; + } + htab = skel->bss->htab_for_user; + test_arena_htab_common(htab); +out: + arena_htab__destroy(skel); +} + +static void test_arena_htab_asm(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct arena_htab_asm *skel; + struct htab *htab; + int ret; + + skel = arena_htab_asm__open_and_load(); + if (!ASSERT_OK_PTR(skel, "arena_htab_asm__open_and_load")) + return; + + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.arena_htab_asm), &opts); + ASSERT_OK(ret, "ret"); + ASSERT_OK(opts.retval, "retval"); + htab = skel->bss->htab_for_user; + test_arena_htab_common(htab); + arena_htab_asm__destroy(skel); +} + +void test_arena_htab(void) +{ + if (test__start_subtest("arena_htab_llvm")) + test_arena_htab_llvm(); + if (test__start_subtest("arena_htab_asm")) + test_arena_htab_asm(); +} diff --git a/tools/testing/selftests/bpf/progs/arena_htab.c b/tools/testing/selftests/bpf/progs/arena_htab.c new file mode 100644 index 000000000000..51a9eeb3df5a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_htab.c @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include "bpf_experimental.h" + +struct { + __uint(type, BPF_MAP_TYPE_ARENA); + __uint(map_flags, BPF_F_MMAPABLE); + __uint(max_entries, 1u << 20); /* max_entries * value_size == size of mmap() region */ + __type(key, __u64); + __type(value, __u64); +} arena SEC(".maps"); + +#include "bpf_arena_htab.h" + +void __arena *htab_for_user; +bool skip = false; + +SEC("syscall") +int arena_htab_llvm(void *ctx) +{ +#if defined(__BPF_FEATURE_ARENA_CAST) || defined(BPF_ARENA_FORCE_ASM) + struct htab __arena *htab; + __u64 i; + + htab = bpf_alloc(sizeof(*htab)); + cast_kern(htab); + htab_init(htab); + + /* first run. No old elems in the table */ + bpf_for(i, 0, 1000) + htab_update_elem(htab, i, i); + + /* should replace all elems with new ones */ + bpf_for(i, 0, 1000) + htab_update_elem(htab, i, i); + cast_user(htab); + htab_for_user = htab; +#else + skip = true; +#endif + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/arena_htab_asm.c b/tools/testing/selftests/bpf/progs/arena_htab_asm.c new file mode 100644 index 000000000000..6cd70ea12f0d --- /dev/null +++ b/tools/testing/selftests/bpf/progs/arena_htab_asm.c @@ -0,0 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#define BPF_ARENA_FORCE_ASM +#define arena_htab_llvm arena_htab_asm +#include "arena_htab.c"