From patchwork Wed Nov 23 17:41:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Maguire X-Patchwork-Id: 13054045 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C451CC4332F for ; Wed, 23 Nov 2022 17:42:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237496AbiKWRmg (ORCPT ); Wed, 23 Nov 2022 12:42:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236639AbiKWRmd (ORCPT ); Wed, 23 Nov 2022 12:42:33 -0500 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C1C1F78 for ; Wed, 23 Nov 2022 09:42:31 -0800 (PST) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHeffX006310; Wed, 23 Nov 2022 17:42:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2022-7-12; bh=bg/zdPZMnoM6xrcIe1D3mIvFscb7+6Jxb0xZHy15ijg=; b=AEk9SDLQxZgf+SiTsQZHXh642nx5VK2e8uxFsyZDwgfrqbwSZ1MyQILbZGZAR2JAWLVv p3ILf/h3tF/REPOSwjXodqQYBef1sQfQ2q0EwHzy+VctoQor3mvv+Q0p1oabW1dF1Brl GPCoiL4BhCSySjM6zH8HglMDMQISEeGi7ouwqFaM+hwQVX+tK/V8Ueo8I3y/g05cSkRE YyKkF4Rh6Hjn+LTJ5fDwKl0bginq3L4GL38WIZ6plBAW1JH2EEq+ez6ATZFEfZDkID5p OSxoNERpablNgh+jc8HiFQyMIPX3sFM8dfTo54W9aG+HJk+dwrbKXWhD1/3xyXsL6IXM 9g== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3m1nd88h35-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:03 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2ANH74tV015641; Wed, 23 Nov 2022 17:42:02 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kxnk74a9t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:01 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2ANHfvqC028233; Wed, 23 Nov 2022 17:42:01 GMT Received: from myrouter.uk.oracle.com (dhcp-10-175-201-76.vpn.oracle.com [10.175.201.76]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kxnk74a4g-2; Wed, 23 Nov 2022 17:42:01 +0000 From: Alan Maguire To: andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, mykolal@fb.com, haiyue.wang@intel.com, bpf@vger.kernel.org, Alan Maguire Subject: [RFC bpf-next 1/5] bpf: add kind/metadata prefixes to uapi/linux/btf.h Date: Wed, 23 Nov 2022 17:41:48 +0000 Message-Id: <1669225312-28949-2-git-send-email-alan.maguire@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> References: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_10,2022-11-23_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211230130 X-Proofpoint-GUID: MhMso8frcz_nrSERh6X5EfskHZ63VBpO X-Proofpoint-ORIG-GUID: MhMso8frcz_nrSERh6X5EfskHZ63VBpO Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC This allows us to share them with kernel and userspace so that both libbpf and the kernel can parse BTF kind information. Signed-off-by: Alan Maguire --- include/uapi/linux/btf.h | 7 +++++++ tools/include/uapi/linux/btf.h | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/include/uapi/linux/btf.h b/include/uapi/linux/btf.h index ec1798b..68628e9 100644 --- a/include/uapi/linux/btf.h +++ b/include/uapi/linux/btf.h @@ -197,4 +197,11 @@ struct btf_enum64 { __u32 val_hi32; }; +/* Prefixes used for names encoding BTF kind information via structs; + * a "struct __BTF_KIND_ARRAY" represents how BTF_KIND_ARRAY is encoded, + * while a "struct __BTF_KIND_META_ARRAY" represents the metadata encoding. + */ +#define BTF_KIND_PFX "__BTF_KIND_" +#define BTF_KIND_META_PFX "__BTF_KIND_META_" + #endif /* _UAPI__LINUX_BTF_H__ */ diff --git a/tools/include/uapi/linux/btf.h b/tools/include/uapi/linux/btf.h index ec1798b..68628e9 100644 --- a/tools/include/uapi/linux/btf.h +++ b/tools/include/uapi/linux/btf.h @@ -197,4 +197,11 @@ struct btf_enum64 { __u32 val_hi32; }; +/* Prefixes used for names encoding BTF kind information via structs; + * a "struct __BTF_KIND_ARRAY" represents how BTF_KIND_ARRAY is encoded, + * while a "struct __BTF_KIND_META_ARRAY" represents the metadata encoding. + */ +#define BTF_KIND_PFX "__BTF_KIND_" +#define BTF_KIND_META_PFX "__BTF_KIND_META_" + #endif /* _UAPI__LINUX_BTF_H__ */ From patchwork Wed Nov 23 17:41:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Maguire X-Patchwork-Id: 13054047 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D1D6C3A59F for ; Wed, 23 Nov 2022 17:42:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236639AbiKWRmh (ORCPT ); Wed, 23 Nov 2022 12:42:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236642AbiKWRmd (ORCPT ); Wed, 23 Nov 2022 12:42:33 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FBBAFC6 for ; Wed, 23 Nov 2022 09:42:31 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHQCm3026690; Wed, 23 Nov 2022 17:42:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2022-7-12; bh=Tat7TXc76ejD15NJSRFk75FXUDE/KqLs+Lc9jbkd9xc=; b=LchBXtpd+Ui6Xy5fi9Dtx+DCoGF8mGLOONhP4nCr+b1tWOxajTZlFRAvPocMvo0miYM0 5NoiOkWsTVU9HEz+wmqlX59T3vtq8jxRDeE/9HwuxRzdicZM/s6LSp9ruwnd+GO6w4V2 PO77Lj32L1z69UNH+9Wk8tdLhLOHj0TYryf5alYA10h7XfHYvYdvkyW+o/124mXBMQkL 8vTkad64CXYJ47QnZHCdBhDF9EPcCPPOqp51xlX1yEeQQucw0qnoTYtpP0kHqYn4HcHN FmwDdrUPoR4/CfEkPongzROwomyrPfK3nw5JxPUradV9xsiWJ6BTSsOZNd0QjGTrl5ED Ig== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3m1qwt01h6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:06 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2ANHBHQa015572; Wed, 23 Nov 2022 17:42:05 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kxnk74aca-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:05 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2ANHfvqE028233; Wed, 23 Nov 2022 17:42:05 GMT Received: from myrouter.uk.oracle.com (dhcp-10-175-201-76.vpn.oracle.com [10.175.201.76]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kxnk74a4g-3; Wed, 23 Nov 2022 17:42:04 +0000 From: Alan Maguire To: andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, mykolal@fb.com, haiyue.wang@intel.com, bpf@vger.kernel.org, Alan Maguire Subject: [RFC bpf-next 2/5] libbpf: provide libbpf API to encode BTF kind information Date: Wed, 23 Nov 2022 17:41:49 +0000 Message-Id: <1669225312-28949-3-git-send-email-alan.maguire@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> References: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_10,2022-11-23_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211230130 X-Proofpoint-ORIG-GUID: CN34q6Il8makO0sbkyT7j1j766SrvMpe X-Proofpoint-GUID: CN34q6Il8makO0sbkyT7j1j766SrvMpe Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC This can be used by BTF parsers to handle kinds they do not know about; this is useful when the encoding libbpf is more recent than the parsing BTF; the parser can then skip over the encoded types it does not know about. We use BTF to encode the BTF kinds that are known at the time of BTF encoding; the use of basic BTF kinds (structs, arrays, base types) to describe each kind and any associated metadata allows BTF parsing to handle new kinds that the parser (in libbpf or the kernel) does not know about. These kinds will not be used, but since we know their format they can be skipped over and the rest of the BTF can be parsed. This means we can encode BTF without worrying about the kinds a BTF parser knows about, and means we can avoid using --skip_new_kind solutions. This is valuable, as if kernel BTF encodes everything it can, something as simple as a libbpf package update then unlocks that encoded information, whereas if we encode pessimistically and drop representations of new kinds, this is not possible. So, in short, by carrying a representation of all the kinds encoded, parsers can parse all of the encoded kinds, even if they cannot use them all. We use BTF itself to carry this representation because this approach does not require BTF parsing to understand a new BTF header format; BTF parsing simply sees some additional types it does not do anything with. However, a BTF parser that knows about the encoding of kind information can use this information to guide parsing. The process works by explicitly adding btf structs for each kind. Each struct consists of a "struct __btf_type" followed by an array of metadata structs representing the following metadata (for those kinds that have it). For kinds where a single metadata structure is used, the metadata array has one element. For kinds where the number of metadata elements varies as per the info.vlen field, a zero-element array is encoded. For a given kind, we add a struct __BTF_KIND_. For example, struct __BTF_KIND_INT { struct __btf_type type; }; For a type with one metadata element, the representation looks like this: struct __BTF_KIND_META_ARRAY { __u32 type; __u32 index_type; __u32 nelems; }; struct __BTF_KIND_ARRAY { struct __btf_type type; struct __BTF_KIND_META_ARRAY meta[1]; }; For a type with an info.vlen-determined number of following metadata objects, a zero-length array is used: struct __BTF_KIND_STRUCT { struct __btf_type type; struct __BTF_KIND_META_STRUCT meta[0]; }; In order to link kind numeric kind values to the appropriate struct, a typedef is added; for example: typedef struct __BTF_KIND_INT __BTF_KIND_1; When BTF parsing encounters a kind that is not known, the typedef __BTF_KIND_ is looked up, and we find which struct type id it points to. So 1 -> typedef __BTF_KIND_1 -> struct __BTF_KIND_INT This approach is preferred, since it ensures the structs representing BTF kinds have names which match their associated kind rather than an opaque number. From there, BTF parsing can look up that struct and determine - its basic size; - if it has metadata; and if so - how many array instances are present; - if 0, we know it is a vlen-determined number; i.e. vlen * meta_size - if > 0, simply use the overall struct size; Based upon that information, BTF parsing can proceed for such unknown kinds, since sufficient information was provided at encoding time to skip over them. Note that this assumes that the above kind-related data structures are represented in BTF _prior_ to any kinds that are new to the parser. It also assumes the basic kinds required to represent kinds + metadata; base types, structs, arrays, etc. Signed-off-by: Alan Maguire --- tools/lib/bpf/btf.c | 281 +++++++++++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/btf.h | 10 ++ tools/lib/bpf/libbpf.map | 1 + 3 files changed, 292 insertions(+) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index 71e165b..e3cea44 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -28,6 +28,16 @@ static struct btf_type btf_void; +/* info used to encode/decode an unrecognized kind */ +struct btf_kind_desc { + int kind; + const char *struct_name; /* __BTF_KIND_ARRAY */ + const char *typedef_name; /* __BTF_KIND_2 */ + const char *meta_name; /* __BTF_KIND_META_ARRAY */ + int nr_meta; + int meta_size; +}; + struct btf { /* raw BTF data in native endianness */ void *raw_data; @@ -5011,3 +5021,274 @@ int btf_ext_visit_str_offs(struct btf_ext *btf_ext, str_off_visit_fn visit, void return 0; } + +/* Here we use BTF to encode the BTF kinds that are known at the time of + * BTF encoding; the use of basic BTF kinds (structs, arrays, base types) + * to describe each kind and any associated metadata allows BTF parsing + * to handle new kinds that the parser (in libbpf or the kernel) does + * not know about. These kinds will not be used, but since we know + * their format they can be skipped over and the rest of the BTF can + * be parsed. This means we can encode BTF without worrying about the + * kinds a BTF parser knows about, and means we can avoid using + * --skip_new_kind solutions. This is valuable, as if kernel BTF encodes + * everything it can, something as simple as a libbpf package update + * then unlocks that encodeded information, whereas if we encode + * pessimistically and drop representations of new kinds, this is not + * possible. + * + * So, in short, by carrying a representation of all the kinds encoded, + * parsers can parse all of the encoded kinds, even if they cannot use + * them all. + * + * We use BTF itself to carry this representation because this approach + * does not require BTF parsing to understand a new BTF header format; + * BTF parsing simply sees some additional types it does not do anything + * with. A BTF parser that knows about the encoding of kind information + * however can use this information in parsing. + * + * The process works by explicitly adding btf structs for each kind. + * Each struct consists of a struct __btf_type followed by an array of + * metadata structs representing the following metadata (for those kinds + * that have it). For kinds where a single metadata structure is used, + * the metadata array has one element. For kinds where the number + * of metadata elements varies as per the info.vlen field, a zero-element + * array is encoded. + * + * For a given kind, we add a struct __BTF_KIND_. For example, + * + * struct __BTF_KIND_INT { + * struct __btf_type type; + * }; + * + * For a type with one metadata element, the representation looks like + * this: + * + * struct __BTF_KIND_META_ARRAY { + * __u32 type; + * __u32 index_type; + * __u32 nelems; + * }; + * + * struct __BTF_KIND_ARRAY { + * struct __btf_type type; + * struct __BTF_KIND_META_ARRAY meta[1]; + * }; + * + * + * For a type with an info.vlen-determined number of following metadata + * objects, a zero-length array is used: + * + * struct __BTF_KIND_STRUCT { + * struct __btf_type type; + * struct __BTF_KIND_META_STRUCT meta[0]; + * }; + * + * In order to link kind numeric kind values to the appropriate struct, + * a typedef is added; for example: + * + * typedef struct __BTF_KIND_INT __BTF_KIND_1; + * + * When BTF parsing encounters a kind that is not known, the + * typedef __BTF_KIND_ is looked up, and we find which + * struct type id it points to. So + * + * 1 -> typedef __BTF_KIND_1 -> struct __BTF_KIND_INT + * + * This approach is preferred, since it ensures the structs representing + * BTF kinds have names which match their associated kind rather than + * an opaque number. + * + * From there, BTF parsing can look up that struct and determine + * - its basic size; + * - if it has metadata; and if so + * - how many array instances are present; + * - if 0, we know it is a vlen-determined number; + * - if > 0, simply use the overall struct size; + * + * Based upon that information, BTF parsing can proceed for such + * unknown kinds, since sufficient information was provided + * at encoding time. + * + * Note that this assumes that the above kind-related data + * structures are represented in BTF _prior_ to any kinds that + * are new to the parser. It also assumes the basic kinds + * required to represent kinds + metadata; base types, structs, + * arrays, etc. + */ + +/* info used to encode a kind metadata field */ +struct btf_meta_field { + const char *type; + const char *name; + int size; + int type_id; +}; + +#define BTF_MAX_META_FIELDS 10 + +#define BTF_META_FIELD(__type, __name) \ + { .type = #__type, .name = #__name, .size = sizeof(__type) } + +#define BTF_KIND_STR(__kind) #__kind + +struct btf_kind_encoding { + struct btf_kind_desc kind; + struct btf_meta_field meta[BTF_MAX_META_FIELDS]; +}; + +#define BTF_KIND(__name, __nr_meta, __meta_size, ...) \ + { .kind = { \ + .kind = BTF_KIND_##__name, \ + .struct_name = BTF_KIND_PFX#__name, \ + .meta_name = BTF_KIND_META_PFX #__name, \ + .nr_meta = __nr_meta, \ + .meta_size = __meta_size, \ + }, .meta = { __VA_ARGS__ } } + +struct btf_kind_encoding kinds[] = { + BTF_KIND(UNKN, 0, 0), + + BTF_KIND(INT, 0, 0), + + BTF_KIND(PTR, 0, 0), + + BTF_KIND(ARRAY, 1, sizeof(struct btf_array), + BTF_META_FIELD(__u32, type), + BTF_META_FIELD(__u32, index_type), + BTF_META_FIELD(__u32, nelems)), + + BTF_KIND(STRUCT, 0, sizeof(struct btf_member), + BTF_META_FIELD(__u32, name_off), + BTF_META_FIELD(__u32, type), + BTF_META_FIELD(__u32, offset)), + + BTF_KIND(UNION, 0, sizeof(struct btf_member), + BTF_META_FIELD(__u32, name_off), + BTF_META_FIELD(__u32, type), + BTF_META_FIELD(__u32, offset)), + + BTF_KIND(ENUM, 0, sizeof(struct btf_enum), + BTF_META_FIELD(__u32, name_off), + BTF_META_FIELD(__s32, val)), + + BTF_KIND(FWD, 0, 0), + + BTF_KIND(TYPEDEF, 0, 0), + + BTF_KIND(VOLATILE, 0, 0), + + BTF_KIND(CONST, 0, 0), + + BTF_KIND(RESTRICT, 0, 0), + + BTF_KIND(FUNC, 0, 0), + + BTF_KIND(FUNC_PROTO, 0, sizeof(struct btf_param), + BTF_META_FIELD(__u32, name_off), + BTF_META_FIELD(__u32, type)), + + BTF_KIND(VAR, 1, sizeof(struct btf_var), + BTF_META_FIELD(__u32, linkage)), + + BTF_KIND(DATASEC, 0, sizeof(struct btf_var_secinfo), + BTF_META_FIELD(__u32, type), + BTF_META_FIELD(__u32, offset), + BTF_META_FIELD(__u32, size)), + + + BTF_KIND(FLOAT, 0, 0), + + BTF_KIND(DECL_TAG, 1, sizeof(struct btf_decl_tag), + BTF_META_FIELD(__s32, component_idx)), + + BTF_KIND(TYPE_TAG, 0, 0), + + BTF_KIND(ENUM64, 0, sizeof(struct btf_enum64), + BTF_META_FIELD(__u32, name_off), + BTF_META_FIELD(__u32, val_lo32), + BTF_META_FIELD(__u32, val_hi32)), +}; + +/* Try to add representations of the kinds supported to BTF provided. This will allow parsers + * to decode kinds they do not support and skip over them. + */ +int btf__add_kinds(struct btf *btf) +{ + int btf_type_id, __u32_id, __s32_id, struct_type_id; + char name[64]; + int i; + + /* should have base types; if not bootstrap them. */ + __u32_id = btf__find_by_name(btf, "__u32"); + if (__u32_id < 0) { + __s32 unsigned_int_id = btf__find_by_name(btf, "unsigned int"); + + if (unsigned_int_id < 0) + unsigned_int_id = btf__add_int(btf, "unsigned int", 4, 0); + __u32_id = btf__add_typedef(btf, "__u32", unsigned_int_id); + } + __s32_id = btf__find_by_name(btf, "__s32"); + if (__s32_id < 0) { + __s32 int_id = btf__find_by_name_kind(btf, "int", BTF_KIND_INT); + + if (int_id < 0) + int_id = btf__add_int(btf, "int", 4, BTF_INT_SIGNED); + __s32_id = btf__add_typedef(btf, "__s32", int_id); + } + + /* add "struct __btf_type" if not already present. */ + btf_type_id = btf__find_by_name(btf, "__btf_type"); + if (btf_type_id < 0) { + __s32 union_id = btf__add_union(btf, NULL, sizeof(__u32)); + + btf__add_field(btf, "size", __u32_id, 0, 0); + btf__add_field(btf, "type", __u32_id, 0, 0); + + btf_type_id = btf__add_struct(btf, "__btf_type", sizeof(struct btf_type)); + btf__add_field(btf, "name_off", __u32_id, 0, 0); + btf__add_field(btf, "info", __u32_id, sizeof(__u32) * 8, 0); + btf__add_field(btf, NULL, union_id, sizeof(__u32) * 16, 0); + } + + for (i = 0; i < ARRAY_SIZE(kinds); i++) { + struct btf_kind_encoding *kind = &kinds[i]; + int meta_id, array_id = 0; + + if (btf__find_by_name(btf, kind->kind.struct_name) > 0) + continue; + + if (kind->kind.meta_size != 0) { + struct btf_meta_field *field; + __u32 bit_offset = 0; + int j; + + meta_id = btf__add_struct(btf, kind->kind.meta_name, kind->kind.meta_size); + + for (j = 0; bit_offset < kind->kind.meta_size * 8; j++) { + field = &kind->meta[j]; + + field->type_id = btf__find_by_name(btf, field->type); + if (field->type_id < 0) { + pr_debug("cannot find type '%s' for kind '%s' field '%s'\n", + kind->meta[j].type, kind->kind.struct_name, + kind->meta[j].name); + } else { + btf__add_field(btf, field->name, field->type_id, bit_offset, 0); + } + bit_offset += field->size * 8; + } + array_id = btf__add_array(btf, __u32_id, meta_id, + kind->kind.nr_meta); + + } + struct_type_id = btf__add_struct(btf, kind->kind.struct_name, + sizeof(struct btf_type) + + (kind->kind.nr_meta * kind->kind.meta_size)); + btf__add_field(btf, "type", btf_type_id, 0, 0); + if (kind->kind.meta_size != 0) + btf__add_field(btf, "meta", array_id, sizeof(struct btf_type) * 8, 0); + snprintf(name, sizeof(name), BTF_KIND_PFX "%u", i); + btf__add_typedef(btf, name, struct_type_id); + } + return 0; +} diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h index 8e6880d..a054082 100644 --- a/tools/lib/bpf/btf.h +++ b/tools/lib/bpf/btf.h @@ -219,6 +219,16 @@ LIBBPF_API int btf__add_datasec_var_info(struct btf *btf, int var_type_id, LIBBPF_API int btf__add_decl_tag(struct btf *btf, const char *value, int ref_type_id, int component_idx); +/** + * @brief **btf__add_kinds()** adds BTF representations of the kind encoding for + * all of the kinds known to libbpf. This ensures that when BTF is encoded, it + * will include enough information for parsers to decode (and skip over) kinds + * that the parser does not know about yet. This ensures that an older BTF + * parser can read newer BTF, and avoids the need for the BTF encoder to limit + * which kinds it emits to make decoding easier. + */ +LIBBPF_API int btf__add_kinds(struct btf *btf); + struct btf_dedup_opts { size_t sz; /* optional .BTF.ext info to dedup along the main BTF info */ diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 71bf569..6121ff1 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -375,6 +375,7 @@ LIBBPF_1.1.0 { bpf_link_get_fd_by_id_opts; bpf_map_get_fd_by_id_opts; bpf_prog_get_fd_by_id_opts; + btf__add_kinds; user_ring_buffer__discard; user_ring_buffer__free; user_ring_buffer__new; From patchwork Wed Nov 23 17:41:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Maguire X-Patchwork-Id: 13054049 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C1EEC4332F for ; Wed, 23 Nov 2022 17:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238197AbiKWRml (ORCPT ); Wed, 23 Nov 2022 12:42:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237815AbiKWRmk (ORCPT ); Wed, 23 Nov 2022 12:42:40 -0500 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BA62F78 for ; Wed, 23 Nov 2022 09:42:35 -0800 (PST) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANHQDdZ026697; Wed, 23 Nov 2022 17:42:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2022-7-12; bh=ub1tNgwn+SgnwRZt1w1K7dsBYSmgg+ChnDekeIQMf54=; b=pWVsdN+3E921RSVZj6Fc3Wjm/OjlDFbCiZTPZrb5ZEFdLsZ/dHaL+dIg18Aos6a2bCwH A5zubb73eeVR66E2CsAgDha4z3dumrl87EK2RJQzcc+5rgGCGrvPev87LnzskPALverD SgTAwg80moCMnPBqtcxY3waRGoNmvv4O0e32IKV7gSlJUjCbpSYHcUOFAjbRn+GF1Qss mw2cU6CC+SoKssR/VPdsK4daNju8To9AtHAUaUZ82WIv4MAqimec5xWl2Hwpab2PIYxb /11YdxUecRrkQGuSHb2E1kWcTH9XdctsQQONwSDiebCf4JD2t+scM4XKkCUHGEdYWtXb jg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3m1qwt01h9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:10 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2ANHFO89015584; Wed, 23 Nov 2022 17:42:10 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kxnk74af1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:09 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2ANHfvqG028233; Wed, 23 Nov 2022 17:42:09 GMT Received: from myrouter.uk.oracle.com (dhcp-10-175-201-76.vpn.oracle.com [10.175.201.76]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kxnk74a4g-4; Wed, 23 Nov 2022 17:42:09 +0000 From: Alan Maguire To: andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, mykolal@fb.com, haiyue.wang@intel.com, bpf@vger.kernel.org, Alan Maguire Subject: [RFC bpf-next 3/5] libbpf: use BTF-encoded kind information to help parse unrecognized kinds Date: Wed, 23 Nov 2022 17:41:50 +0000 Message-Id: <1669225312-28949-4-git-send-email-alan.maguire@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> References: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_10,2022-11-23_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211230130 X-Proofpoint-ORIG-GUID: oiQvhlAHF_uEQGI4kUzPFwYHq4n8vTN9 X-Proofpoint-GUID: oiQvhlAHF_uEQGI4kUzPFwYHq4n8vTN9 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC btf__add_kinds() adds typedef/struct representations of the kinds supported at BTF encoding time. When decoding/parsing, we can then use information about unrecognized kinds to skip over them. This will be useful if the BTF encoder encoded info using a new kind, but the parser doesn't support it yet. Note that only size determinations of unrecognized kinds are supported; lookup, dedup and other features are not supported; the aim here is to not break BTF parsing if newer kinds are encountered. Signed-off-by: Alan Maguire --- tools/lib/bpf/btf.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index e3cea44..da719de 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -28,6 +28,8 @@ static struct btf_type btf_void; +#define NR_BTF_KINDS_POSSIBLE 0x20 + /* info used to encode/decode an unrecognized kind */ struct btf_kind_desc { int kind; @@ -131,6 +133,9 @@ struct btf { /* Pointer size (in bytes) for a target architecture of this BTF */ int ptr_sz; + + /* representations of unrecognized kinds are stored here */ + struct btf_kind_desc unrecognized_kinds[NR_BTF_KINDS_POSSIBLE - NR_BTF_KINDS]; }; static inline __u64 ptr_to_u64(const void *ptr) @@ -420,6 +425,75 @@ static int btf_bswap_type_rest(struct btf_type *t) } } +static int btf_unrecognized_kind_type_size(struct btf *btf, const struct btf_type *t) +{ + struct btf_kind_desc *unrec_kind = NULL; + __u16 kind = btf_kind(t); + int size = 0; + + if (kind >= NR_BTF_KINDS && kind < NR_BTF_KINDS_POSSIBLE) + unrec_kind = &btf->unrecognized_kinds[kind - NR_BTF_KINDS]; + if (!unrec_kind) { + pr_debug("No information about unrecognized kind:%u\n", kind); + return -EINVAL; + } + + if (unrec_kind->kind != kind) { + const char *kind_struct_name; + const struct btf_type *kt; + const struct btf_member *m; + const struct btf_array *a; + char typedef_name[64]; + __s32 id; + + /* we need to fill in information on this kind; it will be cached in struct btf + * for subsequent references. + */ + snprintf(typedef_name, sizeof(typedef_name), BTF_KIND_PFX "%u", kind); + id = btf__find_by_name_kind(btf, typedef_name, BTF_KIND_TYPEDEF); + if (id < 0) + return id; + kt = btf__type_by_id(btf, id); + kt = btf__type_by_id(btf, kt->type); + kind_struct_name = btf__str_by_offset(btf, kt->name_off); + /* struct should contain type + optional meta fields; otherwise unsupported */ + switch (btf_vlen(kt)) { + case 1: + unrec_kind->nr_meta = 0; + unrec_kind->meta_size = 0; + break; + case 2: + m = btf_members(kt); + kt = btf__type_by_id(btf, (++m)->type); + if (btf_kind(kt) != BTF_KIND_ARRAY) { + pr_debug("Unexpected kind %u for member in '%s'\n", + btf_kind(kt), kind_struct_name); + return -EINVAL; + } + a = btf_array(kt); + unrec_kind->nr_meta = a->nelems; + kt = btf__type_by_id(btf, a->type); + unrec_kind->meta_size = kt->size; + unrec_kind->meta_name = btf__str_by_offset(btf, kt->name_off); + break; + default: + pr_debug("unexpected nr of fields for '%s'(%u)\n", kind_struct_name, + kind); + return -EINVAL; + } + unrec_kind->kind = kind; + unrec_kind->struct_name = kind_struct_name; + } + size = sizeof(struct btf_type); + if (unrec_kind->meta_size > 0) { + if (unrec_kind->nr_meta == 0) + size += btf_vlen(t) * unrec_kind->meta_size; + else + size += unrec_kind->nr_meta * unrec_kind->meta_size; + } + return size; +} + static int btf_parse_type_sec(struct btf *btf) { struct btf_header *hdr = btf->hdr; @@ -433,6 +507,8 @@ static int btf_parse_type_sec(struct btf *btf) type_size = btf_type_size(next_type); if (type_size < 0) + type_size = btf_unrecognized_kind_type_size(btf, next_type); + if (type_size < 0) return type_size; if (next_type + type_size > end_type) { pr_warn("BTF type [%d] is malformed\n", btf->start_id + btf->nr_types); From patchwork Wed Nov 23 17:41:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Maguire X-Patchwork-Id: 13054046 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C67DBC43219 for ; Wed, 23 Nov 2022 17:42:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236642AbiKWRmi (ORCPT ); Wed, 23 Nov 2022 12:42:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236767AbiKWRme (ORCPT ); Wed, 23 Nov 2022 12:42:34 -0500 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2792C55BE for ; Wed, 23 Nov 2022 09:42:32 -0800 (PST) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANGTV5S015410; Wed, 23 Nov 2022 17:42:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2022-7-12; bh=kRQdqPZJLvnUf9AmGNLTWIW5o2boEbDaFN6o71qDERk=; b=ENmj9FRYFQ/F09s1ybDj7FFyK6kDHi0WYGZoVdi+q76udV0Fvyk2hhkh8mtFDBo+ln1S /2m1tXSAu2CLOlcrNNot+CmlbZ0q/rPms50D381PD7Md3u1U2JXEZtlSesu2pZMhg4vB GfYv2UqA996sP5SS6+FPLayG/VqAID1Tf8nRLog8cd2vnsef4waD4MCGjWdhXfXb9hwc c05gUqYGSFItBvPVTQ+m1+vmmcwTrS77IgBNhg5mX1tQWZADLBeAafEImC6mZfwNJrsj l191T4wZgj5j2hhbRX4CkC1qj7uPBD5lTeZImKhfTjGJWc3aAdafJMIOlz4QtxQSz+wz eg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3m1p5fgd4v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:14 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2ANHFO8A015584; Wed, 23 Nov 2022 17:42:13 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kxnk74agn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:13 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2ANHfvqI028233; Wed, 23 Nov 2022 17:42:12 GMT Received: from myrouter.uk.oracle.com (dhcp-10-175-201-76.vpn.oracle.com [10.175.201.76]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kxnk74a4g-5; Wed, 23 Nov 2022 17:42:12 +0000 From: Alan Maguire To: andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, mykolal@fb.com, haiyue.wang@intel.com, bpf@vger.kernel.org, Alan Maguire Subject: [RFC bpf-next 4/5] bpf: parse unrecognized kind info using encoded kind information (if present) Date: Wed, 23 Nov 2022 17:41:51 +0000 Message-Id: <1669225312-28949-5-git-send-email-alan.maguire@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> References: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_10,2022-11-23_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211230130 X-Proofpoint-ORIG-GUID: H6UFZBBfIIcxajFft0sncl0tqkQsPHCe X-Proofpoint-GUID: H6UFZBBfIIcxajFft0sncl0tqkQsPHCe Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC When BTF parsing encounters an unrecognized kind (> BTF_KIND_MAX), look for __BTF_KIND_ typedef which points at the associated kind struct; it tells us if there is metadata and how much. This all allows us to proceed with BTF parsing rather than bailing when hitting a kind we do not support. Signed-off-by: Alan Maguire --- kernel/bpf/btf.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 80 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 1a59cc7..ce00a4c5 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -222,6 +222,14 @@ struct btf_id_dtor_kfunc_tab { struct btf_id_dtor_kfunc dtors[]; }; +struct btf_kind_desc { + u16 kind; + u16 nr_meta; + u32 meta_size; +}; + +#define NR_BTF_KINDS_POSSIBLE 0x20 + struct btf { void *data; struct btf_type **types; @@ -246,6 +254,7 @@ struct btf { u32 start_str_off; /* first string offset (0 for base BTF) */ char name[MODULE_NAME_LEN]; bool kernel_btf; + struct btf_kind_desc unrecognized_kinds[NR_BTF_KINDS_POSSIBLE - NR_BTF_KINDS]; }; enum verifier_phase { @@ -4873,7 +4882,7 @@ static s32 btf_check_meta(struct btf_verifier_env *env, u32 meta_left) { u32 saved_meta_left = meta_left; - s32 var_meta_size; + s32 var_meta_size = 0; if (meta_left < sizeof(*t)) { btf_verifier_log(env, "[%u] meta_left:%u meta_needed:%zu", @@ -4888,12 +4897,80 @@ static s32 btf_check_meta(struct btf_verifier_env *env, return -EINVAL; } - if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX || - BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) { + if (BTF_INFO_KIND(t->info) == BTF_KIND_UNKN || + BTF_INFO_KIND(t->info) >= NR_BTF_KINDS_POSSIBLE) { btf_verifier_log(env, "[%u] Invalid kind:%u", env->log_type_id, BTF_INFO_KIND(t->info)); return -EINVAL; } + if (BTF_INFO_KIND(t->info) <= BTF_KIND_MAX) { + var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left); + if (var_meta_size < 0) + return var_meta_size; + } else { + struct btf_kind_desc *unrec_kind; + u8 kind = BTF_INFO_KIND(t->info); + struct btf *btf = env->btf; + + unrec_kind = &btf->unrecognized_kinds[kind - NR_BTF_KINDS]; + + if (unrec_kind->kind != kind) { + const struct btf_member *m; + const struct btf_type *kt; + const struct btf_array *a; + char name[64]; + s32 id; + + /* BTF may encode info about unrecognized kinds; check for this here. */ + snprintf(name, sizeof(name), BTF_KIND_PFX "%u", kind); + id = btf_find_by_name_kind(btf, name, BTF_KIND_TYPEDEF); + if (id > 0) { + kt = btf_type_by_id(btf, id); + if (kt) + kt = btf_type_by_id(btf, kt->type); + } + if (id < 0 || !kt) { + btf_verifier_log_type(env, t, "[%u] invalid kind:%u", + env->log_type_id, kind); + return -EINVAL; + } + switch (btf_type_vlen(kt)) { + case 1: + /* no metadata */ + unrec_kind->kind = kind; + unrec_kind->nr_meta = 0; + unrec_kind->meta_size = 0; + break; + case 2: + m = btf_members(kt); + kt = btf_type_by_id(btf, (++m)->type); + if (btf_kind(kt) != BTF_KIND_ARRAY) { + btf_verifier_log_type(env, t, "[%u] invalid metadata:%u", + env->log_type_id, kind); + return -EINVAL; + } + a = btf_array(kt); + kt = btf_type_by_id(btf, a->type); + if (!kt) { + btf_verifier_log_type(env, t, "[%u] invalid metadata:%u", + env->log_type_id, kind); + return -EINVAL; + } + unrec_kind->kind = kind; + unrec_kind->nr_meta = a->nelems; + unrec_kind->meta_size = kt->size; + break; + default: + btf_verifier_log_type(env, t, "[%u] invalid metadata:%u", + env->log_type_id, kind); + return -EINVAL; + } + } + if (!unrec_kind->nr_meta) + var_meta_size = btf_type_vlen(t) * unrec_kind->meta_size; + else + var_meta_size = unrec_kind->nr_meta * unrec_kind->meta_size; + } if (!btf_name_offset_valid(env->btf, t->name_off)) { btf_verifier_log(env, "[%u] Invalid name_offset:%u", @@ -4901,10 +4978,6 @@ static s32 btf_check_meta(struct btf_verifier_env *env, return -EINVAL; } - var_meta_size = btf_type_ops(t)->check_meta(env, t, meta_left); - if (var_meta_size < 0) - return var_meta_size; - meta_left -= var_meta_size; return saved_meta_left - meta_left; From patchwork Wed Nov 23 17:41:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alan Maguire X-Patchwork-Id: 13054050 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16AE4C433FE for ; Wed, 23 Nov 2022 17:42:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237811AbiKWRms (ORCPT ); Wed, 23 Nov 2022 12:42:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238261AbiKWRmm (ORCPT ); Wed, 23 Nov 2022 12:42:42 -0500 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34E39B1FD for ; Wed, 23 Nov 2022 09:42:38 -0800 (PST) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 2ANGTUxY015349; Wed, 23 Nov 2022 17:42:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2022-7-12; bh=p7hc4poq3vBizw9PlTqG0mo8eICza6D7lYdVu9fFq/Y=; b=bsn/co7Kcnk0tkO8K0v9dpZCyDqvyPpB2mJp3jYuYgOxKQuzaAy0C7YULzAH/8QdsFP3 sVVNYKu8D8Ay6WauaYtZvrawGuEtVyJsbi2bxKTPCr63C4wq31Qn6UAfJiR8uC9H7yUf zEtspHBDbpMrl817Si/uEUBnETlwhmfpMkxmH3zuysGbZZI199jkZTZceGirWzvnHzrt THtBLk7RdAmvi10laQUnMuY81oKMaDoMO/jq8bJ5iXJfngdFMV5yzCDDO270SWgec1eh e31fhR0sCFFJ1vDEoMfSLc/FlvZfd/2mtMb0iPIpwrIIqr/o716einofrFqCjUSj0sGB Bw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3m1p5fgd4y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:18 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.5/8.17.1.5) with ESMTP id 2ANHJFZs015679; Wed, 23 Nov 2022 17:42:17 GMT Received: from pps.reinject (localhost [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3kxnk74ajh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 23 Nov 2022 17:42:16 +0000 Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2ANHfvqK028233; Wed, 23 Nov 2022 17:42:16 GMT Received: from myrouter.uk.oracle.com (dhcp-10-175-201-76.vpn.oracle.com [10.175.201.76]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3kxnk74a4g-6; Wed, 23 Nov 2022 17:42:16 +0000 From: Alan Maguire To: andrii@kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, mykolal@fb.com, haiyue.wang@intel.com, bpf@vger.kernel.org, Alan Maguire Subject: [RFC bpf-next 5/5] selftests/bpf: test kind encoding/decoding Date: Wed, 23 Nov 2022 17:41:52 +0000 Message-Id: <1669225312-28949-6-git-send-email-alan.maguire@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> References: <1669225312-28949-1-git-send-email-alan.maguire@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-11-23_10,2022-11-23_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2211230130 X-Proofpoint-ORIG-GUID: fG4foOBT7iaCQjcNGs_6aIqhlF5cSS0f X-Proofpoint-GUID: fG4foOBT7iaCQjcNGs_6aIqhlF5cSS0f Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC verify btf__add_kinds() adds kind encodings for all kinds supported, and after adding kind-related types for unknown kinds, ensure that parsing uses this info when those kinds are encountered rather than giving up. Signed-off-by: Alan Maguire --- tools/testing/selftests/bpf/prog_tests/btf_kind.c | 234 ++++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_kind.c diff --git a/tools/testing/selftests/bpf/prog_tests/btf_kind.c b/tools/testing/selftests/bpf/prog_tests/btf_kind.c new file mode 100644 index 0000000..e0865ab --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/btf_kind.c @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022, Oracle and/or its affiliates. */ + +#include +#include +#include + +/* verify kind encoding exists for each kind; ensure it consists of a + * typedef __BTF_KIND_ pointing at a struct __BTF_KIND_, + * and the latter has a struct btf_type field and an optional metadata + * field. + */ +void test_btf_kind_encoding(struct btf *btf) +{ + int type_cnt; + __u16 i; + + type_cnt = btf__type_cnt(btf); + if (!ASSERT_GT(type_cnt, 0, "check_type_cnt")) + return; + + for (i = 0; i <= BTF_KIND_MAX; i++) { + const struct btf_type *t, *mt; + const struct btf_array *a; + struct btf_member *m; + char name[64]; + __s32 tid; + + snprintf(name, sizeof(name), BTF_KIND_PFX"%u", i); + + tid = btf__find_by_name(btf, name); + + if (!ASSERT_GT(tid, 0, "find_typedef")) + continue; + + t = btf__type_by_id(btf, tid); + if (!ASSERT_OK_PTR(t, "typedef_ptr")) + continue; + t = btf__type_by_id(btf, t->type); + if (!ASSERT_OK_PTR(t, "struct_ptr")) + continue; + if (!ASSERT_EQ(strncmp(btf__name_by_offset(btf, t->name_off), + BTF_KIND_PFX, + strlen(BTF_KIND_PFX)), + 0, "check_struct_name")) + continue; + if (!ASSERT_GT(btf_vlen(t), 0, "check_struct_vlen")) + continue; + m = btf_members(t); + mt = btf__type_by_id(btf, m->type); + if (!ASSERT_EQ(btf_kind(mt), BTF_KIND_STRUCT, "member_kind")) + continue; + if (!ASSERT_EQ(strcmp(btf__name_by_offset(btf, mt->name_off), + "__btf_type"), 0, "member_type_name")) + continue; + if (btf_vlen(t) == 1) + continue; + m++; + mt = btf__type_by_id(btf, m->type); + if (!ASSERT_EQ(btf_kind(mt), BTF_KIND_ARRAY, "member_kind")) + continue; + a = btf_array(mt); + mt = btf__type_by_id(btf, a->type); + if (!ASSERT_EQ(strncmp(btf__name_by_offset(btf, mt->name_off), + BTF_KIND_META_PFX, + strlen(BTF_KIND_META_PFX)), 0, + "check_meta_name")) + continue; + } +} + +static __s32 add_kind(struct btf *btf, __u16 unrec_kind, int nr_meta, int meta_size) +{ + __s32 btf_type_id, id, struct_id, array_id; + char name[64]; + int ret; + + /* fabricate unrecognized kind definitions that will be used + * when we add a type using the unrecognized kind later. + */ + btf_type_id = btf__find_by_name_kind(btf, "__btf_type", BTF_KIND_STRUCT); + if (!ASSERT_GT(btf_type_id, 0, "check_btf_type_id")) + return btf_type_id; + + if (meta_size > 0) { + __s32 __u32_id; + + __u32_id = btf__find_by_name(btf, "__u32"); + if (!ASSERT_GT(__u32_id, 0, "find_u32")) + return __u32_id; + + array_id = btf__add_array(btf, __u32_id, btf_type_id, nr_meta); + if (!ASSERT_GT(array_id, 0, "btf__add_array")) + return array_id; + } + + snprintf(name, sizeof(name), BTF_KIND_PFX "UNREC%d", unrec_kind); + struct_id = btf__add_struct(btf, name, sizeof(struct btf_type) + (nr_meta * meta_size)); + if (!ASSERT_GT(struct_id, 0, "check_struct_id")) + return struct_id; + + ret = btf__add_field(btf, "type", btf_type_id, 0, 0); + if (!ASSERT_EQ(ret, 0, "btf__add_field")) + return ret; + if (meta_size > 0) { + ret = btf__add_field(btf, "meta", array_id, 96, 0); + if (!ASSERT_EQ(ret, 0, "btf__add_field_meta")) + return ret; + } + snprintf(name, sizeof(name), BTF_KIND_PFX "%u", unrec_kind); + id = btf__add_typedef(btf, name, struct_id); + if (!ASSERT_GT(id, 0, "btf__add_typedef")) + return id; + + return btf_type_id; +} + +/* fabricate unrecognized kinds at BTF_KIND_MAX + 1/2/3, and after adding + * the appropriate struct/typedefs to the BTF such that it recognizes + * these kinds, ensure that parsing of BTF containing the unrecognized kinds + * can succeed. + */ +void test_btf_kind_decoding(struct btf *btf) +{ + const struct btf_type *old_unrecs[6]; + __s32 id, btf_type_id, unrec_ids[6]; + __u16 unrec_kind = BTF_KIND_MAX + 1; + struct btf_type *new_unrecs[6]; + const void *raw_data; + struct btf *raw_btf; + char btf_path[64]; + __u32 raw_size; + int i, fd; + + /* fabricate unrecognized kind definitions that will be used + * when we add a type using the unrecognized kind later. + * + * Kinds are created with + * - no metadata; + * - a single metadata object + * - a vlen-determined number of metadata objects. + */ + btf_type_id = add_kind(btf, unrec_kind, 0, 0); + if (!ASSERT_GT(btf_type_id, 0, "add_kind1")) + return; + if (!ASSERT_GT(add_kind(btf, unrec_kind + 1, 1, sizeof(struct btf_type)), 0, "add_kind2")) + return; + if (!ASSERT_GT(add_kind(btf, unrec_kind + 2, 0, sizeof(struct btf_type)), 0, "add_kind3")) + return; + + /* now create our types with unrecognized kinds by adding typedef kinds + * and overwriting them with our unrecognized kind values. + */ + for (i = 0; i < ARRAY_SIZE(unrec_ids); i++) { + char name[64]; + + snprintf(name, sizeof(name), "unrec_kind%d", i); + unrec_ids[i] = btf__add_typedef(btf, name, btf_type_id); + if (!ASSERT_GT(unrec_ids[i], 0, "btf__add_typedef")) + return; + old_unrecs[i] = btf__type_by_id(btf, unrec_ids[i]); + if (!ASSERT_OK_PTR(old_unrecs[i], "check_unrec_ptr")) + return; + new_unrecs[i] = (struct btf_type *)old_unrecs[i]; + } + + /* add an id after it that we will look up to verify we can parse + * beyond unrecognized kinds. + */ + id = btf__add_typedef(btf, "test_lookup", btf_type_id); + if (!ASSERT_GT(id, 0, "add_test_lookup_id")) + return; + /* ...but because we converted two BTF types into metadata, the id + * for lookup after modification will be two less. + */ + id -= 2; + + /* now modify typedefs added above to become unrecognized kinds */ + new_unrecs[0]->info = (unrec_kind << 24); + + /* unrecognized kind with 1 metadata object */ + new_unrecs[1]->info = ((unrec_kind + 1) << 24); + + /* unrecognized kind with vlen-determined number of metadata objects; vlen == 1 here */ + new_unrecs[3]->info = ((unrec_kind + 2) << 24) | 0x1; + + /* having another instance of the unrecognized kind allows us to test + * the caching codepath; we store unrecognized kind data the + * first time we encounter one to avoid the cost of typedef/struct + * lookups each time an unrecognized kind is seen. + */ + new_unrecs[5]->info = (unrec_kind << 24); + /* now write our BTF to a raw file, ready for parsing. */ + snprintf(btf_path, sizeof(btf_path), "/tmp/btf_kind.%d", getpid()); + raw_data = btf__raw_data(btf, &raw_size); + if (!ASSERT_OK_PTR(raw_data, "check_raw_data")) + return; + fd = open(btf_path, O_WRONLY | O_CREAT); + write(fd, raw_data, raw_size); + close(fd); + + /* verify parsing succeeds, and that we can read type info past + * the unrecognized kind. + */ + raw_btf = btf__parse_raw(btf_path); + if (!ASSERT_OK_PTR(raw_btf, "btf__parse_raw")) + goto unlink; + ASSERT_EQ(btf__find_by_name_kind(raw_btf, "test_lookup", + BTF_KIND_TYPEDEF), id, + "verify_id_lookup"); + + /* finally, verify the kernel can handle unrecognized kinds. */ + ASSERT_EQ(btf__load_into_kernel(raw_btf), 0, "btf_load_into_kernel"); +unlink: + unlink(btf_path); +} + +void test_btf_kind(void) +{ + struct btf *btf = btf__new_empty(); + + if (!ASSERT_OK_PTR(btf, "btf_new")) + return; + + if (!ASSERT_OK(btf__add_kinds(btf), "btf__add_kinds")) + goto cleanup; + + if (test__start_subtest("btf_kind_encoding")) + test_btf_kind_encoding(btf); + if (test__start_subtest("btf_kind_decoding")) + test_btf_kind_decoding(btf); +cleanup: + btf__free(btf); +}