From patchwork Wed Sep 7 15:45:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969154 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEDD2C38145 for ; Wed, 7 Sep 2022 15:45:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229484AbiIGPpM (ORCPT ); Wed, 7 Sep 2022 11:45:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229544AbiIGPpL (ORCPT ); Wed, 7 Sep 2022 11:45:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79B674A81D for ; Wed, 7 Sep 2022 08:45:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zzJ9iGRxFY7VwR0DTvK6zA/xNig8/8UFuA2SVUAPlTs=; b=SGapSvvOFS5WH2sHpp0zFD5OhtLzGqXLkCfMCjkix2kdGpn5GzK52E1MEiELr884JOd7Ea 7/PfT/1bNxneaCt8mS1H9P+OAZh9TRsl1r5+kBuX23YN7FI43goW/hcuiMY81Bor2/ZtUP 0IS66D3Hq8HnKo/+0+oAtlfRi7OQKh8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-378-Jo85jWahNLKQgkxiOFiz_Q-1; Wed, 07 Sep 2022 11:45:07 -0400 X-MC-Unique: Jo85jWahNLKQgkxiOFiz_Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D6419811E83; Wed, 7 Sep 2022 15:45:06 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 754704010D2A; Wed, 7 Sep 2022 15:45:06 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id B1D9430721A6C; Wed, 7 Sep 2022 17:45:05 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 01/18] libbpf: factor out BTF loading from load_module_btfs() From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:05 +0200 Message-ID: <166256550566.1434226.11682408188276569750.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Larysa Zaremba In order to be able to reuse BTF loading logics, move it to the new btf_load_next_with_info() and call it from load_module_btfs() instead. To still be able to get the ID, introduce the ID field to the userspace struct btf and return it via the new btf_obj_id(). To still be able to use bpf_btf_info::name as a string, locally add a counterpart to ptr_to_u64() - u64_to_ptr() and use it to filter vmlinux/module BTFs. Also, add a definition for easy bpf_btf_info name declaration and make btf_get_from_fd() static as it's now used only in btf.c. Signed-off-by: Larysa Zaremba Signed-off-by: Alexander Lobakin --- tools/lib/bpf/btf.c | 110 +++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf.c | 52 ++++-------------- tools/lib/bpf/libbpf_internal.h | 7 ++ 3 files changed, 126 insertions(+), 43 deletions(-) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index 361131518d63..cad11c56cf1f 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -121,6 +121,9 @@ struct btf { /* Pointer size (in bytes) for a target architecture of this BTF */ int ptr_sz; + + /* BTF object ID, valid for vmlinux and module BTF */ + __u32 id; }; static inline __u64 ptr_to_u64(const void *ptr) @@ -128,6 +131,11 @@ static inline __u64 ptr_to_u64(const void *ptr) return (__u64) (unsigned long) ptr; } +static inline const void *u64_to_ptr(__u64 val) +{ + return (const void *)(unsigned long)val; +} + /* Ensure given dynamically allocated memory region pointed to by *data* with * capacity of *cap_cnt* elements each taking *elem_sz* bytes has enough * memory to accommodate *add_cnt* new elements, assuming *cur_cnt* elements @@ -458,6 +466,11 @@ const struct btf *btf__base_btf(const struct btf *btf) return btf->base_btf; } +__u32 btf_obj_id(const struct btf *btf) +{ + return btf->id; +} + /* internal helper returning non-const pointer to a type */ struct btf_type *btf_type_by_id(const struct btf *btf, __u32 type_id) { @@ -814,6 +827,7 @@ static struct btf *btf_new_empty(struct btf *base_btf) btf->fd = -1; btf->ptr_sz = sizeof(void *); btf->swapped_endian = false; + btf->id = 0; if (base_btf) { btf->base_btf = base_btf; @@ -864,6 +878,7 @@ static struct btf *btf_new(const void *data, __u32 size, struct btf *base_btf) btf->start_id = 1; btf->start_str_off = 0; btf->fd = -1; + btf->id = 0; if (base_btf) { btf->base_btf = base_btf; @@ -1327,7 +1342,7 @@ const char *btf__name_by_offset(const struct btf *btf, __u32 offset) return btf__str_by_offset(btf, offset); } -struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf) +static struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf) { struct bpf_btf_info btf_info; __u32 len = sizeof(btf_info); @@ -1375,6 +1390,8 @@ struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf) } btf = btf_new(ptr, btf_info.btf_size, base_btf); + if (!IS_ERR_OR_NULL(btf)) + btf->id = btf_info.id; exit_free: free(ptr); @@ -4636,6 +4653,97 @@ static int btf_dedup_remap_types(struct btf_dedup *d) return 0; } +/** + * btf_load_next_with_info - get first BTF with ID bigger than the input one. + * @start_id: ID to start the search from + * @info: buffer to put BTF info to + * @base_btf: base BTF, can be %NULL if @vmlinux is true + * @vmlinux: true to look for the vmlinux BTF instead of a module BTF + * + * Obtains the first BTF with the ID bigger than the @start_id. @info::name and + * @info::name_len must be initialized by the caller. The default name buffer + * size is %BTF_NAME_BUF_LEN. + * FD must be closed after BTF is no longer needed. If @vmlinux is true, FD can + * be closed and set to -1 right away without preventing later usage. + * + * Returns pointer to the BTF loaded from the kernel or an error pointer. + */ +struct btf *btf_load_next_with_info(__u32 start_id, struct bpf_btf_info *info, + struct btf *base_btf, bool vmlinux) +{ + __u32 name_len = info->name_len; + __u64 name = info->name; + const char *name_str; + __u32 id = start_id; + + if (!name) + return ERR_PTR(-EINVAL); + + name_str = u64_to_ptr(name); + + while (true) { + __u32 len = sizeof(*info); + struct btf *btf; + int err, fd; + + err = bpf_btf_get_next_id(id, &id); + if (err) { + err = -errno; + if (err != -ENOENT) + pr_warn("failed to iterate BTF objects: %d\n", + err); + return ERR_PTR(err); + } + + fd = bpf_btf_get_fd_by_id(id); + if (fd < 0) { + err = -errno; + if (err == -ENOENT) + /* Expected race: non-vmlinux BTF was + * unloaded + */ + continue; + pr_warn("failed to get BTF object #%d FD: %d\n", + id, err); + return ERR_PTR(err); + } + + memset(info, 0, len); + info->name = name; + info->name_len = name_len; + + err = bpf_obj_get_info_by_fd(fd, info, &len); + if (err) { + err = -errno; + pr_warn("failed to get BTF object #%d info: %d\n", + id, err); + goto err_out; + } + + /* Filter BTFs */ + if (!info->kernel_btf || + !strcmp(name_str, "vmlinux") != vmlinux) { + close(fd); + continue; + } + + btf = btf_get_from_fd(fd, base_btf); + err = libbpf_get_error(btf); + if (err) { + pr_warn("failed to load module [%s]'s BTF object #%d: %d\n", + name_str, id, err); + goto err_out; + } + + btf->fd = fd; + return btf; + +err_out: + close(fd); + return ERR_PTR(err); + } +} + /* * Probe few well-known locations for vmlinux kernel image and try to load BTF * data out of it to use for target BTF. diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 3ad139285fad..ff0a2b026cd4 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -5341,11 +5341,11 @@ int bpf_core_add_cands(struct bpf_core_cand *local_cand, static int load_module_btfs(struct bpf_object *obj) { - struct bpf_btf_info info; + char name[BTF_NAME_BUF_LEN] = { }; struct module_btf *mod_btf; + struct bpf_btf_info info; struct btf *btf; - char name[64]; - __u32 id = 0, len; + __u32 id = 0; int err, fd; if (obj->btf_modules_loaded) @@ -5362,49 +5362,19 @@ static int load_module_btfs(struct bpf_object *obj) return 0; while (true) { - err = bpf_btf_get_next_id(id, &id); - if (err && errno == ENOENT) - return 0; - if (err) { - err = -errno; - pr_warn("failed to iterate BTF objects: %d\n", err); - return err; - } - - fd = bpf_btf_get_fd_by_id(id); - if (fd < 0) { - if (errno == ENOENT) - continue; /* expected race: BTF was unloaded */ - err = -errno; - pr_warn("failed to get BTF object #%d FD: %d\n", id, err); - return err; - } - - len = sizeof(info); memset(&info, 0, sizeof(info)); info.name = ptr_to_u64(name); info.name_len = sizeof(name); - err = bpf_obj_get_info_by_fd(fd, &info, &len); - if (err) { - err = -errno; - pr_warn("failed to get BTF object #%d info: %d\n", id, err); - goto err_out; - } - - /* ignore non-module BTFs */ - if (!info.kernel_btf || strcmp(name, "vmlinux") == 0) { - close(fd); - continue; - } - - btf = btf_get_from_fd(fd, obj->btf_vmlinux); + btf = btf_load_next_with_info(id, &info, obj->btf_vmlinux, + false); err = libbpf_get_error(btf); - if (err) { - pr_warn("failed to load module [%s]'s BTF object #%d: %d\n", - name, id, err); - goto err_out; - } + if (err) + return err == -ENOENT ? 0 : err; + + fd = btf__fd(btf); + btf__set_fd(btf, -1); + id = btf_obj_id(btf); err = libbpf_ensure_mem((void **)&obj->btf_modules, &obj->btf_module_cap, sizeof(*obj->btf_modules), obj->btf_module_cnt + 1); diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h index 377642ff51fc..02d8f544eade 100644 --- a/tools/lib/bpf/libbpf_internal.h +++ b/tools/lib/bpf/libbpf_internal.h @@ -367,9 +367,14 @@ int libbpf__load_raw_btf(const char *raw_types, size_t types_len, const char *str_sec, size_t str_len); int btf_load_into_kernel(struct btf *btf, char *log_buf, size_t log_sz, __u32 log_level); -struct btf *btf_get_from_fd(int btf_fd, struct btf *base_btf); void btf_get_kernel_prefix_kind(enum bpf_attach_type attach_type, const char **prefix, int *kind); +__u32 btf_obj_id(const struct btf *btf); + +#define BTF_NAME_BUF_LEN 64 + +struct btf *btf_load_next_with_info(__u32 start_id, struct bpf_btf_info *info, + struct btf *base_btf, bool vmlinux); struct btf_ext_info { /* From patchwork Wed Sep 7 15:45:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969155 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4ABAC54EE9 for ; Wed, 7 Sep 2022 15:45:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229604AbiIGPpV (ORCPT ); Wed, 7 Sep 2022 11:45:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229544AbiIGPpQ (ORCPT ); Wed, 7 Sep 2022 11:45:16 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EC101705E for ; Wed, 7 Sep 2022 08:45:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CfnpNVOn8bVeC3t2vM1lmC87z9StKK7QVfzUjHELd7M=; b=XnKGPbzVvTT+4J1dn0uLjBKdQxdsZFbQ0Wy3tM3khtycn15FMC7twSOzhS+VBCErDrZtGD kUVpvsiLXjwtgONM2+7js5m0raT34KBYG4lb2Aswk97S8rs4qqnxrScl+Lt4QzDgfAgDhu AqeInseJI3BNWWRUFbdiwudE+IkZYMg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-449-uTG5rkmhOmSngS_6zY5Nxw-1; Wed, 07 Sep 2022 11:45:12 -0400 X-MC-Unique: uTG5rkmhOmSngS_6zY5Nxw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0102B801231; Wed, 7 Sep 2022 15:45:12 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id B623290A04; Wed, 7 Sep 2022 15:45:11 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id BC87230721A6C; Wed, 7 Sep 2022 17:45:10 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 02/18] libbpf: try to load vmlinux BTF from the kernel first From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:10 +0200 Message-ID: <166256551073.1434226.11702276674514019182.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Larysa Zaremba Try to acquire vmlinux BTF the same way it's being done for module BTFs. Use btf_load_next_with_info() and resort to the filesystem lookup only if it fails. Also, adjust debug messages in btf__load_vmlinux_btf() to reflect that it actually tries to load vmlinux BTF. Signed-off-by: Larysa Zaremba Signed-off-by: Alexander Lobakin --- tools/lib/bpf/btf.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index cad11c56cf1f..1fd12a2e1b08 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -4744,6 +4744,25 @@ struct btf *btf_load_next_with_info(__u32 start_id, struct bpf_btf_info *info, } } +static struct btf *btf_load_vmlinux_from_kernel(void) +{ + char name[BTF_NAME_BUF_LEN] = { }; + struct bpf_btf_info info; + struct btf *btf; + + memset(&info, 0, sizeof(info)); + info.name = ptr_to_u64(name); + info.name_len = sizeof(name); + + btf = btf_load_next_with_info(0, &info, NULL, true); + if (!libbpf_get_error(btf)) { + close(btf->fd); + btf__set_fd(btf, -1); + } + + return btf; +} + /* * Probe few well-known locations for vmlinux kernel image and try to load BTF * data out of it to use for target BTF. @@ -4770,6 +4789,15 @@ struct btf *btf__load_vmlinux_btf(void) struct btf *btf; int i, err; + btf = btf_load_vmlinux_from_kernel(); + err = libbpf_get_error(btf); + pr_debug("loading vmlinux BTF from kernel: %d\n", err); + if (!err) + return btf; + + pr_info("failed to load vmlinux BTF from kernel: %d, will look through filesystem\n", + err); + uname(&buf); for (i = 0; i < ARRAY_SIZE(locations); i++) { @@ -4783,14 +4811,14 @@ struct btf *btf__load_vmlinux_btf(void) else btf = btf__parse_elf(path, NULL); err = libbpf_get_error(btf); - pr_debug("loading kernel BTF '%s': %d\n", path, err); + pr_debug("loading vmlinux BTF '%s': %d\n", path, err); if (err) continue; return btf; } - pr_warn("failed to find valid kernel BTF\n"); + pr_warn("failed to find valid vmlinux BTF\n"); return libbpf_err_ptr(-ESRCH); } From patchwork Wed Sep 7 15:45:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969156 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E19C5C54EE9 for ; Wed, 7 Sep 2022 15:45:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229612AbiIGPpY (ORCPT ); Wed, 7 Sep 2022 11:45:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229595AbiIGPpU (ORCPT ); Wed, 7 Sep 2022 11:45:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28E8F58B4B for ; Wed, 7 Sep 2022 08:45:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565518; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZZZ1CMV6frSOfS2BKbh9jMknLjABN4hNwTIR62lrj00=; b=PljEm0h5iFfzLyuZKiguA2J6hMCEqcgBalQvTIj/aKDtOWecyo+HC4+HUTH4nmlxK8Zymz XEti0PNUbyKtXie8GYtK9+4UuJyVDbmXL0yOTKP9FMr47MvQTWLnZIthgdSB1RYgt8859P kV+P3deOwBaLSx4/RrBOCObKCaFUyZw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-44-I-foaCp-OTi7PdC-mStS6w-1; Wed, 07 Sep 2022 11:45:17 -0400 X-MC-Unique: I-foaCp-OTi7PdC-mStS6w-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BD855101A56C; Wed, 7 Sep 2022 15:45:16 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 776682026D4C; Wed, 7 Sep 2022 15:45:16 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id C81B730721A6C; Wed, 7 Sep 2022 17:45:15 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 03/18] libbpf: patch module BTF obj+type ID into BPF insns From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:15 +0200 Message-ID: <166256551578.1434226.1711667214489185920.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Larysa Zaremba Return both BTF type id and BTF object id from bpf_core_type_id_kernel(). Earlier only type id was returned despite the fact that llvm has enabled the 64-bit return type for this instruction [1]. This was done as a preparation to the patch [2], which also strongly served as a inspiration for this implementation. [1] https://reviews.llvm.org/D91489 [2] https://lore.kernel.org/all/20201205025140.443115-1-andrii@kernel.org Signed-off-by: Larysa Zaremba Signed-off-by: Alexander Lobakin Signed-off-by: Jesper Dangaard Brouer --- tools/lib/bpf/bpf_core_read.h | 3 ++- tools/lib/bpf/relo_core.c | 8 +++++++- tools/lib/bpf/relo_core.h | 1 + 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/tools/lib/bpf/bpf_core_read.h b/tools/lib/bpf/bpf_core_read.h index 496e6a8ee0dc..f033ec65fc01 100644 --- a/tools/lib/bpf/bpf_core_read.h +++ b/tools/lib/bpf/bpf_core_read.h @@ -168,7 +168,8 @@ enum bpf_enum_value_kind { * Convenience macro to get BTF type ID of a target kernel's type that matches * specified local type. * Returns: - * - valid 32-bit unsigned type ID in kernel BTF; + * - valid 64-bit unsigned integer: the upper 32 bits is the BTF object ID + * and the lower 32 bits is the BTF type ID within the BTF object. * - 0, if no matching type was found in a target kernel BTF. */ #define bpf_core_type_id_kernel(type) \ diff --git a/tools/lib/bpf/relo_core.c b/tools/lib/bpf/relo_core.c index c4b0e81ae293..ca94f8e2c698 100644 --- a/tools/lib/bpf/relo_core.c +++ b/tools/lib/bpf/relo_core.c @@ -892,6 +892,7 @@ static int bpf_core_calc_relo(const char *prog_name, res->fail_memsz_adjust = false; res->orig_sz = res->new_sz = 0; res->orig_type_id = res->new_type_id = 0; + res->btf_obj_id = 0; if (core_relo_is_field_based(relo->kind)) { err = bpf_core_calc_field_relo(prog_name, relo, local_spec, @@ -942,6 +943,8 @@ static int bpf_core_calc_relo(const char *prog_name, } else if (core_relo_is_type_based(relo->kind)) { err = bpf_core_calc_type_relo(relo, local_spec, &res->orig_val, &res->validate); err = err ?: bpf_core_calc_type_relo(relo, targ_spec, &res->new_val, NULL); + if (!err && relo->kind == BPF_CORE_TYPE_ID_TARGET) + res->btf_obj_id = btf_obj_id(targ_spec->btf); } else if (core_relo_is_enumval_based(relo->kind)) { err = bpf_core_calc_enumval_relo(relo, local_spec, &res->orig_val); err = err ?: bpf_core_calc_enumval_relo(relo, targ_spec, &res->new_val); @@ -1133,7 +1136,10 @@ int bpf_core_patch_insn(const char *prog_name, struct bpf_insn *insn, } insn[0].imm = new_val; - insn[1].imm = new_val >> 32; + /* For type IDs, upper 32 bits are used for BTF object ID */ + insn[1].imm = relo->kind == BPF_CORE_TYPE_ID_TARGET ? + res->btf_obj_id : + (new_val >> 32); pr_debug("prog '%s': relo #%d: patched insn #%d (LDIMM64) imm64 %llu -> %llu\n", prog_name, relo_idx, insn_idx, (unsigned long long)imm, (unsigned long long)new_val); diff --git a/tools/lib/bpf/relo_core.h b/tools/lib/bpf/relo_core.h index 1c0566daf8e8..52de7c018fb8 100644 --- a/tools/lib/bpf/relo_core.h +++ b/tools/lib/bpf/relo_core.h @@ -66,6 +66,7 @@ struct bpf_core_relo_res { __u32 orig_type_id; __u32 new_sz; __u32 new_type_id; + __u32 btf_obj_id; }; int __bpf_core_types_are_compat(const struct btf *local_btf, __u32 local_id, From patchwork Wed Sep 7 15:45:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969157 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E142C54EE9 for ; Wed, 7 Sep 2022 15:45:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229652AbiIGPpd (ORCPT ); Wed, 7 Sep 2022 11:45:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229595AbiIGPpa (ORCPT ); Wed, 7 Sep 2022 11:45:30 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3277166A5D for ; Wed, 7 Sep 2022 08:45:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y6L2s1M+P9oqMPMrDrGBNL2SB10RA54bSv25oEqaLuI=; b=MFZOhudS1mbtAn/mmzs96ghFGuB+VaJwt77sOZZo6niClZXcpMNdgU4GWxP6BE73TGKbHK B0aARGbDedOVfKWaN4i2lvnNYR5fviWJg1Kp09AF1LlJ7HlGYkUOAE+1LD4SMNz5/X8zt9 q7cBkQ/e6oCHAIlED7jRJodMAufD9HY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-621-9n1vZWZgMluD8HSmHxRelQ-1; Wed, 07 Sep 2022 11:45:23 -0400 X-MC-Unique: 9n1vZWZgMluD8HSmHxRelQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 42BEC1C05EA8; Wed, 7 Sep 2022 15:45:22 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id CBA801410F38; Wed, 7 Sep 2022 15:45:21 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id D3F8D30721A6C; Wed, 7 Sep 2022 17:45:20 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 04/18] net: create xdp_hints_common and set functions From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:20 +0200 Message-ID: <166256552083.1434226.577215984964402996.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC XDP-hints via BTF are about giving drivers the ability to extend the common set of hardware offload hints in a flexible way. This patch start out with defining the common set, based on what is used available in the SKB. Having this as a common struct in core vmlinux makes it easier to implement xdp_frame to SKB conversion routines as normal C-code, see later patches. Drivers can redefine the layout of the entire metadata area, but are encouraged to use this common struct as the base, on which they can extend on top for their extra hardware offload hints. When doing so, drivers can mark the xdp_buff (and xdp_frame) with flags indicating this it compatible with the common struct. Patch also provides XDP-hints driver helper functions for updating the common struct. Helpers gets inlined and are defined for maximum performance, which does require some extra care in drivers, e.g. to keep track of flags to reduce data dependencies, see code DOC. Userspace and BPF-prog's MUST not consider the common struct UAPI. The common struct (and enum flags) are only exposed via BTF, which implies consumers must read and decode this BTF before using/consuming data layout. Signed-off-by: Jesper Dangaard Brouer --- include/net/xdp.h | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++++ net/core/xdp.c | 5 ++ 2 files changed, 152 insertions(+) diff --git a/include/net/xdp.h b/include/net/xdp.h index 04c852c7a77f..ea5836ccee82 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -8,6 +8,151 @@ #include /* skb_shared_info */ +/** + * struct xdp_hints_common - Common XDP-hints offloads shared with netstack + * @btf_full_id: The modules BTF object + type ID for specific struct + * @vlan_tci: Hardware provided VLAN tag + proto type in @xdp_hints_flags + * @rx_hash32: Hardware provided RSS hash value + * @xdp_hints_flags: see &enum xdp_hints_flags + * + * This structure contains the most commonly used hardware offloads hints + * provided by NIC drivers and supported by the SKB. + * + * Driver are expected to extend this structure by include &struct + * xdp_hints_common as part of the drivers own specific xdp_hints struct's, but + * at the end-of their struct given XDP metadata area grows backwards. + * + * The member @btf_full_id is populated by driver modules to uniquely identify + * the BTF struct. The high 32-bits store the modules BTF object ID and the + * lower 32-bit the BTF type ID within that BTF object. + */ +struct xdp_hints_common { + union { + __wsum csum; + struct { + __u16 csum_start; + __u16 csum_offset; + }; + }; + u16 rx_queue; + u16 vlan_tci; + u32 rx_hash32; + u32 xdp_hints_flags; + u64 btf_full_id; /* BTF object + type ID */ +} __attribute__((aligned(4))) __attribute__((packed)); + + +/** + * enum xdp_hints_flags - flags used by &struct xdp_hints_common + * + * The &enum xdp_hints_flags have reserved the first 16 bits for common flags + * and drivers can introduce use their own flags bits from BIT(16). For + * BPF-progs to find these flags (via BTF) drivers should define an enum + * xdp_hints_flags_driver. + */ +enum xdp_hints_flags { + HINT_FLAG_CSUM_TYPE_BIT0 = BIT(0), + HINT_FLAG_CSUM_TYPE_BIT1 = BIT(1), + HINT_FLAG_CSUM_TYPE_MASK = 0x3, + + HINT_FLAG_CSUM_LEVEL_BIT0 = BIT(2), + HINT_FLAG_CSUM_LEVEL_BIT1 = BIT(3), + HINT_FLAG_CSUM_LEVEL_MASK = 0xC, + HINT_FLAG_CSUM_LEVEL_SHIFT = 2, + + HINT_FLAG_RX_HASH_TYPE_BIT0 = BIT(4), + HINT_FLAG_RX_HASH_TYPE_BIT1 = BIT(5), + HINT_FLAG_RX_HASH_TYPE_MASK = 0x30, + HINT_FLAG_RX_HASH_TYPE_SHIFT = 0x4, + + HINT_FLAG_RX_QUEUE = BIT(7), + + HINT_FLAG_VLAN_PRESENT = BIT(8), + HINT_FLAG_VLAN_PROTO_ETH_P_8021Q = BIT(9), + HINT_FLAG_VLAN_PROTO_ETH_P_8021AD = BIT(10), + /* Flags from BIT(16) can be used by drivers */ +}; + +/** + * enum xdp_hints_csum_type - BTF exposing checksum defines + * + * This enum is primarily for BTF exposing ``CHECKSUM_*`` defines (as an enum) + * used by &struct skb->ip_summed (see Documentation/networking/skbuff.rst + * section "Checksum information"). + * + * These values are stored in &enum xdp_hints_flags as bit locations + * ``HINT_FLAG_CSUM_TYPE_BIT*`` + */ +enum xdp_hints_csum_type { + HINT_CHECKSUM_NONE = CHECKSUM_NONE, + HINT_CHECKSUM_UNNECESSARY = CHECKSUM_UNNECESSARY, + HINT_CHECKSUM_COMPLETE = CHECKSUM_COMPLETE, + HINT_CHECKSUM_PARTIAL = CHECKSUM_PARTIAL, +}; + +/** DOC: XDP hints driver helpers + * + * Helpers for drivers updating struct xdp_hints_common. + * + * Avoid creating a data dependency on xdp_hints_flags via returning the flags + * that need to be set. Drivers MUST update the xdp_hints_flags member + * themselves, which allows drivers to construct code with less data dependency + * between instructions by OR'ing the final flags together. + */ + +/* Drivers please use this simple helper to ease changes across drives */ +static __always_inline void xdp_hints_set_flags(struct xdp_hints_common *hints, + u32 flags) +{ + hints->xdp_hints_flags = flags; +} + +static __always_inline u32 xdp_hints_set_rx_csum( + struct xdp_hints_common *hints, + u16 type, u16 level) +{ + u32 flags; + + flags = type & HINT_FLAG_CSUM_TYPE_MASK; + flags |= (level << HINT_FLAG_CSUM_LEVEL_SHIFT) + & HINT_FLAG_CSUM_LEVEL_MASK; + + // TODO: handle CHECKSUM_PARTIAL and COMPLETE (needs updating *hints) + return flags; +} + +/* @type Must be &enum enum pkt_hash_types (PKT_HASH_TYPE_*) */ +static __always_inline u32 xdp_hints_set_rx_hash( + struct xdp_hints_common *hints, + u32 hash, u32 type) +{ + hints->rx_hash32 = hash; + return (type << HINT_FLAG_RX_HASH_TYPE_SHIFT) & + HINT_FLAG_RX_HASH_TYPE_MASK; +} + +static __always_inline u32 xdp_hints_set_rxq(struct xdp_hints_common *hints, + u16 q_idx) +{ + hints->rx_queue = q_idx; + return HINT_FLAG_RX_QUEUE; +} + +/* @proto Must be ETH_P_8021Q or ETH_P_8021AD in network order */ +static __always_inline u32 xdp_hints_set_vlan(struct xdp_hints_common *hints, + u16 vlan_tag, const u16 proto) +{ + u32 flags = HINT_FLAG_VLAN_PRESENT; + + hints->vlan_tci = vlan_tag; + if (proto == htons(ETH_P_8021Q)) + flags |= HINT_FLAG_VLAN_PROTO_ETH_P_8021Q; + if (proto == htons(ETH_P_8021AD)) + flags |= HINT_FLAG_VLAN_PROTO_ETH_P_8021AD; + + return flags; +} + /** * DOC: XDP RX-queue information * @@ -72,6 +217,8 @@ enum xdp_buff_flags { XDP_FLAGS_FRAGS_PF_MEMALLOC = BIT(1), /* xdp paged memory is under * pressure */ + XDP_FLAGS_HAS_HINTS = BIT(2), + XDP_FLAGS_HINTS_COMPAT_COMMON = BIT(3), }; struct xdp_buff { diff --git a/net/core/xdp.c b/net/core/xdp.c index 24420209bf0e..a57bd5278b47 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -33,6 +33,11 @@ static int mem_id_next = MEM_ID_MIN; static bool mem_id_init; /* false */ static struct rhashtable *mem_id_ht; +/* Make xdp_hints part of core vmlinux BTF */ +struct xdp_hints_common xdp_hints_common; +enum xdp_hints_flags xdp_hints_flags; +enum xdp_hints_csum_type xdp_hints_csum_type; + static u32 xdp_mem_id_hashfn(const void *data, u32 len, u32 seed) { const u32 *k = data; From patchwork Wed Sep 7 15:45:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969158 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60D25C54EE9 for ; Wed, 7 Sep 2022 15:45:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229764AbiIGPpm (ORCPT ); Wed, 7 Sep 2022 11:45:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229643AbiIGPpc (ORCPT ); Wed, 7 Sep 2022 11:45:32 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A058462AB1 for ; Wed, 7 Sep 2022 08:45:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cwk6GCxz2fKCIof6GdQC9jP34T0np9Gb9y3RuN9eduo=; b=acobb0sndIpqjOFh61DpaDjQ4mfoB61vmuEp+nl1Nh08iKnNUay/9aogdlSySRcUmZWfJn Y3ZIyjRaST1Go/V+JTJO6cGOoVjt/R44fCtDsrb0MbF4Nkgy7SUeJCMZR7izcxdCJyJaV5 fp9lXD66DB7yLdtIX6tl7Sy49pAe2/c= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-633-eux9cWw2Mh2tz_-Td3XKTQ-1; Wed, 07 Sep 2022 11:45:27 -0400 X-MC-Unique: eux9cWw2Mh2tz_-Td3XKTQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D3C3E1C05EAE; Wed, 7 Sep 2022 15:45:26 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 99697492C3B; Wed, 7 Sep 2022 15:45:26 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id E009630721A6C; Wed, 7 Sep 2022 17:45:25 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 05/18] net: add net_device feature flag for XDP-hints From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:25 +0200 Message-ID: <166256552588.1434226.9247778254069745452.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Make it possible to turnoff XDP-hints for a given net_device. It is recommended that drivers default turn on XDP-hints as the overhead is generally low, extracting these hardware hints, and the benefit is usually higher than this small overhead e.g. getting HW to do RX checksumming are usually a higher gain. Some XDP use-case are not ready to take this small overhead. Thus, the possibility to turn off XDP-hints is need to keep performance of these use-cases intact. Signed-off-by: Jesper Dangaard Brouer --- include/linux/netdev_features.h | 3 ++- net/ethtool/common.c | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/netdev_features.h b/include/linux/netdev_features.h index 7c2d77d75a88..713f04eab497 100644 --- a/include/linux/netdev_features.h +++ b/include/linux/netdev_features.h @@ -14,7 +14,7 @@ typedef u64 netdev_features_t; enum { NETIF_F_SG_BIT, /* Scatter/gather IO. */ NETIF_F_IP_CSUM_BIT, /* Can checksum TCP/UDP over IPv4. */ - __UNUSED_NETIF_F_1, + NETIF_F_XDP_HINTS_BIT, /* Populates XDP-hints metadata */ NETIF_F_HW_CSUM_BIT, /* Can checksum all the packets. */ NETIF_F_IPV6_CSUM_BIT, /* Can checksum TCP/UDP over IPV6 */ NETIF_F_HIGHDMA_BIT, /* Can DMA to high memory. */ @@ -168,6 +168,7 @@ enum { #define NETIF_F_HW_HSR_TAG_RM __NETIF_F(HW_HSR_TAG_RM) #define NETIF_F_HW_HSR_FWD __NETIF_F(HW_HSR_FWD) #define NETIF_F_HW_HSR_DUP __NETIF_F(HW_HSR_DUP) +#define NETIF_F_XDP_HINTS __NETIF_F(XDP_HINTS) /* Finds the next feature with the highest number of the range of start-1 till 0. */ diff --git a/net/ethtool/common.c b/net/ethtool/common.c index 566adf85e658..a9c62482220f 100644 --- a/net/ethtool/common.c +++ b/net/ethtool/common.c @@ -11,6 +11,7 @@ const char netdev_features_strings[NETDEV_FEATURE_COUNT][ETH_GSTRING_LEN] = { [NETIF_F_SG_BIT] = "tx-scatter-gather", [NETIF_F_IP_CSUM_BIT] = "tx-checksum-ipv4", + [NETIF_F_XDP_HINTS_BIT] = "xdp-hints", [NETIF_F_HW_CSUM_BIT] = "tx-checksum-ip-generic", [NETIF_F_IPV6_CSUM_BIT] = "tx-checksum-ipv6", [NETIF_F_HIGHDMA_BIT] = "highdma", From patchwork Wed Sep 7 15:45:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969159 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6869C38145 for ; Wed, 7 Sep 2022 15:45:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229674AbiIGPpo (ORCPT ); Wed, 7 Sep 2022 11:45:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229669AbiIGPpk (ORCPT ); Wed, 7 Sep 2022 11:45:40 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46D8B6D9C5 for ; Wed, 7 Sep 2022 08:45:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cdxGJ5dHoOldEJkDFP9p+jriAkljx0MtUoRMntLWj64=; b=M87bLyn9t4CAXIGypWRizU4Y1n37eiVl8+5fRNR5eamOveLFMTdurjDD8G5dAXdTfXCR4a TE0VLrUfppJatdqmhBqIROpggV0Jue/2mnxcKA5ptJx0enji3U0myohaD5pJXCCyVU+IIw dRVndaqGXzScb8mrx1WPLui3JVOEYUk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-195-2-S1a2WbOIy9vbKF2elLxw-1; Wed, 07 Sep 2022 11:45:33 -0400 X-MC-Unique: 2-S1a2WbOIy9vbKF2elLxw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 673DA85A58B; Wed, 7 Sep 2022 15:45:32 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF2671410F38; Wed, 7 Sep 2022 15:45:31 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 017C630721A6C; Wed, 7 Sep 2022 17:45:31 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 06/18] xdp: controlling XDP-hints from BPF-prog via helper From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:30 +0200 Message-ID: <166256553093.1434226.11320655607309623999.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC XDP BPF-prog's need a way to interact with the XDP-hints. This patch introduces a BPF-helper function, that allow XDP BPF-prog's to interact with the XDP-hints. Choosing BPF-helper to avoid directly exposing xdp_buff_flags as UAPI. BPF-prog can query if any XDP-hints have been setup and if this is compatible with the xdp_hints_common struct. Notice that XDP-hints are setup by the driver prior to calling the XDP BPF-prog, which is useful as a BPF software layer for adjusting the HW provided XDP-hints in-case of HW issues or missing HW features, for use-case like xdp2skb or AF_XDP. The BPF-prog might also prefer to use metadata area for other things, either disabling XDP-hints or updating with another XDP-hints layout that might still be compatible with common struct. Thus, helper have "update" and "delete" mode flags. Signed-off-by: Jesper Dangaard Brouer --- include/net/xdp.h | 41 ++++++++++++++++++++++++++++---- include/uapi/linux/bpf.h | 41 ++++++++++++++++++++++++++++++++ net/core/filter.c | 52 ++++++++++++++++++++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 43 ++++++++++++++++++++++++++++++++- 4 files changed, 172 insertions(+), 5 deletions(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index ea5836ccee82..c7cdcef83fa5 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -213,14 +213,19 @@ struct xdp_txq_info { }; enum xdp_buff_flags { - XDP_FLAGS_HAS_FRAGS = BIT(0), /* non-linear xdp buff */ - XDP_FLAGS_FRAGS_PF_MEMALLOC = BIT(1), /* xdp paged memory is under + XDP_FLAGS_HINTS_ENABLED = BIT(0),/* enum xdp_hint */ +#define XDP_FLAGS_HINTS_COMPAT_COMMON_ BIT(1) /* HINTS_BTF_COMPAT_COMMON */ + XDP_FLAGS_HINTS_COMPAT_COMMON = XDP_FLAGS_HINTS_COMPAT_COMMON_, + + XDP_FLAGS_HAS_FRAGS = BIT(2), /* non-linear xdp buff */ + XDP_FLAGS_FRAGS_PF_MEMALLOC = BIT(3), /* xdp paged memory is under * pressure */ - XDP_FLAGS_HAS_HINTS = BIT(2), - XDP_FLAGS_HINTS_COMPAT_COMMON = BIT(3), }; +#define XDP_FLAGS_HINTS_MASK (XDP_FLAGS_HINTS_ENABLED | \ + XDP_FLAGS_HINTS_COMPAT_COMMON) + struct xdp_buff { void *data; void *data_end; @@ -257,6 +262,34 @@ static __always_inline void xdp_buff_set_frag_pfmemalloc(struct xdp_buff *xdp) xdp->flags |= XDP_FLAGS_FRAGS_PF_MEMALLOC; } +static __always_inline bool xdp_buff_has_hints(struct xdp_buff *xdp) +{ + return !!(xdp->flags & XDP_FLAGS_HINTS_MASK); +} + +static __always_inline bool xdp_buff_has_hints_compat(struct xdp_buff *xdp) +{ + u32 flags = xdp->flags; + + if (!(flags & XDP_FLAGS_HINTS_COMPAT_COMMON)) + return false; + + return !!(flags & XDP_FLAGS_HINTS_MASK); +} + +static __always_inline void xdp_buff_set_hints_flags(struct xdp_buff *xdp, + bool is_compat_common) +{ + u32 common = is_compat_common ? XDP_FLAGS_HINTS_COMPAT_COMMON : 0; + + xdp->flags |= XDP_FLAGS_HINTS_ENABLED | common; +} + +static __always_inline void xdp_buff_clear_hints_flags(struct xdp_buff *xdp) +{ + xdp->flags &= ~XDP_FLAGS_HINTS_MASK; +} + static __always_inline void xdp_init_buff(struct xdp_buff *xdp, u32 frame_sz, struct xdp_rxq_info *rxq) { diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 934a2a8beb87..36ba104e612e 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -5355,6 +5355,37 @@ union bpf_attr { * Return * Current *ktime*. * + * long xdp_hints_btf(struct xdp_buff *xdp_md, u64 flags) + * Description + * Update and get info on XDP hints ctx state. + * + * Drivers can provide XDP-hints information via the metadata area, + * which defines the layout of this area via BTF. The *full* BTF ID + * is available as the last member. + * + * This **full** BTF ID is a 64-bit value, encoding the BTF + * **object** ID as the high 32-bit and BTF *type* ID as lower + * 32-bit. This is needed as the BTF **type** ID (32-bit) can + * originate from different BTF **object** sources, e.g. vmlinux, + * module or local BTF-object. + * + * In-case a BPF-prog want to redefine the layout of this area it + * should update the full BTF ID (last-member) and call this helper + * to specify if the layout is compatible with kernel struct + * xdp_hints_common. + * + * The **flags** are used to control the mode of the helper. + * See enum xdp_hints_btf_mode_flags. + * + * Return + * 0 if driver didn't populate XDP-hints. + * + * Flag **HINTS_BTF_ENABLED** (1) if driver populated hints. + * + * Flag **HINTS_BTF_COMPAT_COMMON** (2) if layout is compatible + * with kernel struct xdp_hints_common. Thus, return value 3 as + * both flags will be set. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -5566,6 +5597,7 @@ union bpf_attr { FN(tcp_raw_check_syncookie_ipv4), \ FN(tcp_raw_check_syncookie_ipv6), \ FN(ktime_get_tai_ns), \ + FN(xdp_hints_btf), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -5977,6 +6009,15 @@ struct xdp_md { __u32 egress_ifindex; /* txq->dev->ifindex */ }; +/* Mode flags for BPF_FUNC_xdp_hints_btf helper. */ +enum xdp_hints_btf_mode_flags { + HINTS_BTF_QUERY_ONLY = (1U << 0), + HINTS_BTF_ENABLED = (1U << 0), /* Return value */ + HINTS_BTF_COMPAT_COMMON = (1U << 1), /* Return and query value */ + HINTS_BTF_UPDATE = (1U << 2), + HINTS_BTF_DISABLE = (1U << 3), +}; + /* DEVMAP map-value layout * * The struct data-layout of map-value is a configuration interface. diff --git a/net/core/filter.c b/net/core/filter.c index 1acfaffeaf32..35f29990a67e 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -6094,6 +6094,56 @@ static const struct bpf_func_proto bpf_xdp_check_mtu_proto = { .arg5_type = ARG_ANYTHING, }; +/* flags type &enum xdp_hints_btf_mode_flags */ +BPF_CALL_2(bpf_xdp_hints_btf, struct xdp_buff *, xdp, u64, flags) +{ + bool is_compat_common; + s64 ret = 0; + + /* UAPI value HINTS_BTF_COMPAT_COMMON happens to match xdp_buff->flags + * XDP_FLAGS_HINTS_COMPAT_COMMON which makes below code easier + */ + BUILD_BUG_ON(HINTS_BTF_COMPAT_COMMON != XDP_FLAGS_HINTS_COMPAT_COMMON_); + + if (flags & HINTS_BTF_QUERY_ONLY) { + ret = xdp->flags & XDP_FLAGS_HINTS_MASK; + goto out; + } + if (flags & HINTS_BTF_DISABLE) { + xdp_buff_clear_hints_flags(xdp); + goto out; + } + if (flags & HINTS_BTF_UPDATE) { + is_compat_common = !!(flags & HINTS_BTF_COMPAT_COMMON); + + if (is_compat_common) { + unsigned long metalen = xdp_get_metalen(xdp); + + if (sizeof(struct xdp_hints_common) < metalen) + is_compat_common = false; + /* TODO: Can kernel validate if hints are BTF compat + * with common? + */ + } + /* TODO: Could BPF prog provide BTF as ARG_PTR_TO_BTF_ID to prove compat_common ? */ + xdp_buff_set_hints_flags(xdp, is_compat_common); + + ret = xdp->flags & XDP_FLAGS_HINTS_MASK; + goto out; + } + + out: + return ret; +} + +static const struct bpf_func_proto bpf_xdp_hints_btf_proto = { + .func = bpf_xdp_hints_btf, + .gpl_only = true, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg2_type = ARG_ANYTHING, +}; + #if IS_ENABLED(CONFIG_IPV6_SEG6_BPF) static int bpf_push_seg6_encap(struct sk_buff *skb, u32 type, void *hdr, u32 len) { @@ -7944,6 +7994,8 @@ xdp_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_xdp_fib_lookup_proto; case BPF_FUNC_check_mtu: return &bpf_xdp_check_mtu_proto; + case BPF_FUNC_xdp_hints_btf: + return &bpf_xdp_hints_btf_proto; #ifdef CONFIG_INET case BPF_FUNC_sk_lookup_udp: return &bpf_xdp_sk_lookup_udp_proto; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 1d6085e15fc8..36ba104e612e 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -79,7 +79,7 @@ struct bpf_insn { /* Key of an a BPF_MAP_TYPE_LPM_TRIE entry */ struct bpf_lpm_trie_key { __u32 prefixlen; /* up to 32 for AF_INET, 128 for AF_INET6 */ - __u8 data[0]; /* Arbitrary size */ + __u8 data[]; /* Arbitrary size */ }; struct bpf_cgroup_storage_key { @@ -5355,6 +5355,37 @@ union bpf_attr { * Return * Current *ktime*. * + * long xdp_hints_btf(struct xdp_buff *xdp_md, u64 flags) + * Description + * Update and get info on XDP hints ctx state. + * + * Drivers can provide XDP-hints information via the metadata area, + * which defines the layout of this area via BTF. The *full* BTF ID + * is available as the last member. + * + * This **full** BTF ID is a 64-bit value, encoding the BTF + * **object** ID as the high 32-bit and BTF *type* ID as lower + * 32-bit. This is needed as the BTF **type** ID (32-bit) can + * originate from different BTF **object** sources, e.g. vmlinux, + * module or local BTF-object. + * + * In-case a BPF-prog want to redefine the layout of this area it + * should update the full BTF ID (last-member) and call this helper + * to specify if the layout is compatible with kernel struct + * xdp_hints_common. + * + * The **flags** are used to control the mode of the helper. + * See enum xdp_hints_btf_mode_flags. + * + * Return + * 0 if driver didn't populate XDP-hints. + * + * Flag **HINTS_BTF_ENABLED** (1) if driver populated hints. + * + * Flag **HINTS_BTF_COMPAT_COMMON** (2) if layout is compatible + * with kernel struct xdp_hints_common. Thus, return value 3 as + * both flags will be set. + * */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -5566,6 +5597,7 @@ union bpf_attr { FN(tcp_raw_check_syncookie_ipv4), \ FN(tcp_raw_check_syncookie_ipv6), \ FN(ktime_get_tai_ns), \ + FN(xdp_hints_btf), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper @@ -5977,6 +6009,15 @@ struct xdp_md { __u32 egress_ifindex; /* txq->dev->ifindex */ }; +/* Mode flags for BPF_FUNC_xdp_hints_btf helper. */ +enum xdp_hints_btf_mode_flags { + HINTS_BTF_QUERY_ONLY = (1U << 0), + HINTS_BTF_ENABLED = (1U << 0), /* Return value */ + HINTS_BTF_COMPAT_COMMON = (1U << 1), /* Return and query value */ + HINTS_BTF_UPDATE = (1U << 2), + HINTS_BTF_DISABLE = (1U << 3), +}; + /* DEVMAP map-value layout * * The struct data-layout of map-value is a configuration interface. From patchwork Wed Sep 7 15:45:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969160 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B886C6FA89 for ; Wed, 7 Sep 2022 15:45:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229572AbiIGPpq (ORCPT ); Wed, 7 Sep 2022 11:45:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48228 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229614AbiIGPpn (ORCPT ); Wed, 7 Sep 2022 11:45:43 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E07096C775 for ; Wed, 7 Sep 2022 08:45:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565541; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sKfvlfICNlAsdN7VZEl9cExBvBRupUtsUYAWmXh19TA=; b=RJ5d1bkdh26irUK1Zv6KMlBZ8fF1aotGbde6sCv6FGvJUpH1xvF5dIcFugZfggsweziQMJ 1qrX2qflxyOwBtlC+Yy+shYD6rq7N5k5JurpNVYYF2viHKzKIHktUvp66I3E4UYsxULZF4 Kk07GJjKAG9pfaH6hIU5+48mTT/TORs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-646-isKiwfZUNPKBZ1SYdUw-Gw-1; Wed, 07 Sep 2022 11:45:38 -0400 X-MC-Unique: isKiwfZUNPKBZ1SYdUw-Gw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 664FA3800C2C; Wed, 7 Sep 2022 15:45:37 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0693740149B6; Wed, 7 Sep 2022 15:45:37 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 0E08030721A6C; Wed, 7 Sep 2022 17:45:36 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 07/18] i40e: Refactor i40e_ptp_rx_hwtstamp From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:36 +0200 Message-ID: <166256553602.1434226.14564390707008322763.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC No functional change, this is in preparation for later patches. Introduce i40e_ptp_rx_hwtstamp_raw() that doesn't depend on skb pointer as input. Keep i40e_ptp_rx_hwtstamp with same semantics as before. Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/i40e/i40e.h | 1 + drivers/net/ethernet/intel/i40e/i40e_ptp.c | 36 +++++++++++++++++++++------- 2 files changed, 28 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h index d86b6d349ea9..859e11f4e884 100644 --- a/drivers/net/ethernet/intel/i40e/i40e.h +++ b/drivers/net/ethernet/intel/i40e/i40e.h @@ -1262,6 +1262,7 @@ void i40e_ptp_rx_hang(struct i40e_pf *pf); void i40e_ptp_tx_hang(struct i40e_pf *pf); void i40e_ptp_tx_hwtstamp(struct i40e_pf *pf); void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index); +u64 i40e_ptp_rx_hwtstamp_raw(struct i40e_pf *pf, u8 index); void i40e_ptp_set_increment(struct i40e_pf *pf); int i40e_ptp_set_ts_config(struct i40e_pf *pf, struct ifreq *ifr); int i40e_ptp_get_ts_config(struct i40e_pf *pf, struct ifreq *ifr); diff --git a/drivers/net/ethernet/intel/i40e/i40e_ptp.c b/drivers/net/ethernet/intel/i40e/i40e_ptp.c index 2d3533f38d7b..ec33d783f6ee 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_ptp.c +++ b/drivers/net/ethernet/intel/i40e/i40e_ptp.c @@ -808,18 +808,16 @@ void i40e_ptp_tx_hwtstamp(struct i40e_pf *pf) } /** - * i40e_ptp_rx_hwtstamp - Utility function which checks for an Rx timestamp + * i40e_ptp_rx_hwtstamp_raw - Utility function which checks for an Rx timestamp * @pf: Board private structure - * @skb: Particular skb to send timestamp with * @index: Index into the receive timestamp registers for the timestamp * * The XL710 receives a notification in the receive descriptor with an offset - * into the set of RXTIME registers where the timestamp is for that skb. This + * into the set of RXTIME registers where the timestamp is for that pkt. This * function goes and fetches the receive timestamp from that offset, if a valid - * one exists. The RXTIME registers are in ns, so we must convert the result - * first. + * one exists, else zero is returned. **/ -void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index) +u64 i40e_ptp_rx_hwtstamp_raw(struct i40e_pf *pf, u8 index) { u32 prttsyn_stat, hi, lo; struct i40e_hw *hw; @@ -829,7 +827,7 @@ void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index) * doing Tx timestamping, check if Rx timestamping is configured. */ if (!(pf->flags & I40E_FLAG_PTP) || !pf->ptp_rx) - return; + return 0; hw = &pf->hw; @@ -841,7 +839,7 @@ void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index) /* TODO: Should we warn about missing Rx timestamp event? */ if (!(prttsyn_stat & BIT(index))) { spin_unlock_bh(&pf->ptp_rx_lock); - return; + return 0; } /* Clear the latched event since we're about to read its register */ @@ -854,7 +852,27 @@ void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index) ns = (((u64)hi) << 32) | lo; - i40e_ptp_convert_to_hwtstamp(skb_hwtstamps(skb), ns); + return ns; +} + +/** + * i40e_ptp_rx_hwtstamp - Utility function which checks for an Rx timestamp + * @pf: Board private structure + * @skb: Particular skb to send timestamp with + * @index: Index into the receive timestamp registers for the timestamp + * + * The XL710 receives a notification in the receive descriptor with an offset + * into the set of RXTIME registers where the timestamp is for that skb. This + * function goes and fetches the receive timestamp from that offset, if a valid + * one exists. The RXTIME registers are in ns, so we must convert the result + * first. + **/ +void i40e_ptp_rx_hwtstamp(struct i40e_pf *pf, struct sk_buff *skb, u8 index) +{ + u64 ns = i40e_ptp_rx_hwtstamp_raw(pf, index); + + if (ns) + i40e_ptp_convert_to_hwtstamp(skb_hwtstamps(skb), ns); } /** From patchwork Wed Sep 7 15:45:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969161 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86440C38145 for ; Wed, 7 Sep 2022 15:46:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229767AbiIGPqA (ORCPT ); Wed, 7 Sep 2022 11:46:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229672AbiIGPpt (ORCPT ); Wed, 7 Sep 2022 11:45:49 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0307B6AA0F for ; Wed, 7 Sep 2022 08:45:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565547; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q/RYCLcFxWXVg4sepSF6T1VgsBBx9ixyS2xZVokLGcw=; b=Z+NyGw3CaZzzBnCi3a1SlposuQ7934n+o8WgEuEWeiqK5+rlHgHfgQWlrkdRJ9rY0j+oeb 1ijNECQJu5Zv3KOpPCVMa8gamBV+Fg0FYrLP4KlXCKlwyNDwRBsHxLqPKCLlJSEyVY2mNZ zLLRzLtRR9ldjFHRlyyWMzpGUjFgNxM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-375-DEwJ5IVyMXKGQHQ4fK2TUA-1; Wed, 07 Sep 2022 11:45:44 -0400 X-MC-Unique: DEwJ5IVyMXKGQHQ4fK2TUA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 46F98101A58E; Wed, 7 Sep 2022 15:45:43 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id DBCC6403167; Wed, 7 Sep 2022 15:45:42 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 199C530721A6C; Wed, 7 Sep 2022 17:45:41 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 08/18] i40e: refactor i40e_rx_checksum with helper From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:41 +0200 Message-ID: <166256554106.1434226.11099369196515014657.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC No functional change, this is in preparation for later patches. The helper function does not depend on skb, which will be used in later patches. Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 66 ++++++++++++++++----------- 1 file changed, 40 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index f6ba97a0166e..a7a896321880 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -1751,45 +1751,38 @@ bool i40e_alloc_rx_buffers(struct i40e_ring *rx_ring, u16 cleaned_count) return true; } -/** - * i40e_rx_checksum - Indicate in skb if hw indicated a good cksum - * @vsi: the VSI we care about - * @skb: skb currently being received and modified - * @rx_desc: the receive descriptor - **/ -static inline void i40e_rx_checksum(struct i40e_vsi *vsi, - struct sk_buff *skb, - union i40e_rx_desc *rx_desc) +struct i40e_rx_checksum_ret { + u16 ip_summed; + u16 csum_level; +}; + +static inline struct i40e_rx_checksum_ret +_i40e_rx_checksum(struct i40e_vsi *vsi, + u64 qword, + struct i40e_rx_ptype_decoded decoded) { - struct i40e_rx_ptype_decoded decoded; + struct i40e_rx_checksum_ret ret = {}; u32 rx_error, rx_status; bool ipv4, ipv6; - u8 ptype; - u64 qword; - qword = le64_to_cpu(rx_desc->wb.qword1.status_error_len); - ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >> I40E_RXD_QW1_PTYPE_SHIFT; rx_error = (qword & I40E_RXD_QW1_ERROR_MASK) >> I40E_RXD_QW1_ERROR_SHIFT; rx_status = (qword & I40E_RXD_QW1_STATUS_MASK) >> I40E_RXD_QW1_STATUS_SHIFT; - decoded = decode_rx_desc_ptype(ptype); - skb->ip_summed = CHECKSUM_NONE; - - skb_checksum_none_assert(skb); + ret.ip_summed = CHECKSUM_NONE; /* Rx csum enabled and ip headers found? */ if (!(vsi->netdev->features & NETIF_F_RXCSUM)) - return; + return ret; /* did the hardware decode the packet and checksum? */ if (!(rx_status & BIT(I40E_RX_DESC_STATUS_L3L4P_SHIFT))) - return; + return ret; /* both known and outer_ip must be set for the below code to work */ if (!(decoded.known && decoded.outer_ip)) - return; + return ret; ipv4 = (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP) && (decoded.outer_ip_ver == I40E_RX_PTYPE_OUTER_IPV4); @@ -1805,7 +1798,7 @@ static inline void i40e_rx_checksum(struct i40e_vsi *vsi, if (ipv6 && rx_status & BIT(I40E_RX_DESC_STATUS_IPV6EXADD_SHIFT)) /* don't increment checksum err here, non-fatal err */ - return; + return ret; /* there was some L4 error, count error and punt packet to the stack */ if (rx_error & BIT(I40E_RX_DESC_ERROR_L4E_SHIFT)) @@ -1816,30 +1809,51 @@ static inline void i40e_rx_checksum(struct i40e_vsi *vsi, * the csum. */ if (rx_error & BIT(I40E_RX_DESC_ERROR_PPRS_SHIFT)) - return; + return ret; /* If there is an outer header present that might contain a checksum * we need to bump the checksum level by 1 to reflect the fact that * we are indicating we validated the inner checksum. */ if (decoded.tunnel_type >= I40E_RX_PTYPE_TUNNEL_IP_GRENAT) - skb->csum_level = 1; + ret.csum_level = 1; /* Only report checksum unnecessary for TCP, UDP, or SCTP */ switch (decoded.inner_prot) { case I40E_RX_PTYPE_INNER_PROT_TCP: case I40E_RX_PTYPE_INNER_PROT_UDP: case I40E_RX_PTYPE_INNER_PROT_SCTP: - skb->ip_summed = CHECKSUM_UNNECESSARY; + ret.ip_summed = CHECKSUM_UNNECESSARY; fallthrough; default: break; } - return; + return ret; checksum_fail: vsi->back->hw_csum_rx_error++; + return ret; +} + +/** + * i40e_rx_checksum - Indicate in skb if hw indicated a good cksum + * @vsi: the VSI we care about + * @skb: skb currently being received and modified + * @rx_desc: the receive descriptor + **/ +static inline void i40e_rx_checksum(struct i40e_vsi *vsi, + struct sk_buff *skb, + union i40e_rx_desc *rx_desc) +{ + struct i40e_rx_checksum_ret ret; + u64 qword = le64_to_cpu(rx_desc->wb.qword1.status_error_len); + u8 ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >> I40E_RXD_QW1_PTYPE_SHIFT; + struct i40e_rx_ptype_decoded decoded = decode_rx_desc_ptype(ptype); + + ret = _i40e_rx_checksum(vsi, qword, decoded); + skb->ip_summed = ret.ip_summed; + skb->csum_level = ret.csum_level; } /** From patchwork Wed Sep 7 15:45:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969162 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD679C38145 for ; Wed, 7 Sep 2022 15:46:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229817AbiIGPqC (ORCPT ); Wed, 7 Sep 2022 11:46:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229634AbiIGPp6 (ORCPT ); Wed, 7 Sep 2022 11:45:58 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 170706F268 for ; Wed, 7 Sep 2022 08:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565551; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e8qzYn67mc0YUjcpq2BNfwdPV/CrkwgKNBPhMy5km50=; b=PaWyiTfVzOc9mYAiiJpuwWU6LM8eCbLSFcEv9myzFe/alAB77gII39wt0UGq7uc0tDcGMn qG3L4FxV97BtzzkBckmLQwDp4dq+i4twU1IyL7LVk9uRB5yqj04Cx2TAf3FuASKiGOvUIq EM7wgwyTL2LeHRPJFh0iAcRBGfeJio4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-412-P7_TQDObNXik3CSFnSY3QQ-1; Wed, 07 Sep 2022 11:45:48 -0400 X-MC-Unique: P7_TQDObNXik3CSFnSY3QQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1BE0D811E90; Wed, 7 Sep 2022 15:45:47 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id D37234C816; Wed, 7 Sep 2022 15:45:46 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 27C7C30721A6C; Wed, 7 Sep 2022 17:45:46 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 09/18] bpf: export btf functions for modules From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:46 +0200 Message-ID: <166256554611.1434226.15342391322311314838.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC This allows modules to lookup their own module BTF info. These are get and set operations that bump the refcount. Thus, modules can use this to control the lifetime. Suggested-by: Kumar Kartikeya Dwivedi Signed-off-by: Jesper Dangaard Brouer --- include/linux/btf.h | 2 ++ kernel/bpf/btf.c | 13 ++++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/include/linux/btf.h b/include/linux/btf.h index ad93c2d9cc1c..a66266c00c04 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -148,6 +148,8 @@ u32 btf_obj_id(const struct btf *btf); bool btf_is_kernel(const struct btf *btf); bool btf_is_module(const struct btf *btf); struct module *btf_try_get_module(const struct btf *btf); +struct btf *btf_get_module_btf(const struct module *module); +void btf_put_module_btf(struct btf *btf); u32 btf_nr_types(const struct btf *btf); bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, const struct btf_member *m, diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 903719b89238..1e95391e0ca1 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -534,6 +534,7 @@ s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind) return -ENOENT; } +EXPORT_SYMBOL_GPL(btf_find_by_name_kind); static s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p) { @@ -1673,6 +1674,15 @@ void btf_put(struct btf *btf) } } +void btf_put_module_btf(struct btf *btf) +{ + if (!btf_is_module(btf)) + return; + + btf_put(btf); +} +EXPORT_SYMBOL_GPL(btf_put_module_btf); + static int env_resolve_init(struct btf_verifier_env *env) { struct btf *btf = env->btf; @@ -7051,7 +7061,7 @@ struct module *btf_try_get_module(const struct btf *btf) /* Returns struct btf corresponding to the struct module. * This function can return NULL or ERR_PTR. */ -static struct btf *btf_get_module_btf(const struct module *module) +struct btf *btf_get_module_btf(const struct module *module) { #ifdef CONFIG_DEBUG_INFO_BTF_MODULES struct btf_module *btf_mod, *tmp; @@ -7080,6 +7090,7 @@ static struct btf *btf_get_module_btf(const struct module *module) return btf; } +EXPORT_SYMBOL_GPL(btf_get_module_btf); BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int, flags) { From patchwork Wed Sep 7 15:45:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969163 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 877ACC54EE9 for ; Wed, 7 Sep 2022 15:46:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229436AbiIGPqS (ORCPT ); Wed, 7 Sep 2022 11:46:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229459AbiIGPqG (ORCPT ); Wed, 7 Sep 2022 11:46:06 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C037F6E8A9 for ; Wed, 7 Sep 2022 08:45:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565558; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nj5BlWEjHAbOxvMpguL3MAZX7qC3A5IN2hGEHZzAY40=; b=PpknVjmzvIYfRRsl3rWcJ7UItHyr3prjnzASkcwDS9hOT/xFgQqM8xJEkXtc9qdU2HLP3A 3KVxwNgpUko6bK5dKqpqdcYrfERjMQJyaqathsV34h+6LeubPgEm7crYVRDpFGjXJ8PWXr TzZCZNenQKnZ+260Vp2b6iPq/WfKhhI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-436-z2XTngRQPGGugyPz_uhMmg-1; Wed, 07 Sep 2022 11:45:53 -0400 X-MC-Unique: z2XTngRQPGGugyPz_uhMmg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7425D80A0B9; Wed, 7 Sep 2022 15:45:52 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3A29790A04; Wed, 7 Sep 2022 15:45:52 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 35B0330721A6C; Wed, 7 Sep 2022 17:45:51 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 10/18] btf: Add helper for kernel modules to lookup full BTF ID From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:51 +0200 Message-ID: <166256555117.1434226.17491394872817800182.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC NIC driver modules need to store the full BTF ID as the last member in metadata area usually written as xdp_hints_common->btf_full_id. This full BTF ID is a 64-bit value, encoding the modules own BTF object ID as the high 32-bit and specific struct BTF type ID as lower 32-bit. Drivers should invoke this once at init time and cache this BTF ID for runtime usage. Signed-off-by: Jesper Dangaard Brouer --- include/linux/btf.h | 1 + kernel/bpf/btf.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/btf.h b/include/linux/btf.h index a66266c00c04..b8f7c92b6767 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -150,6 +150,7 @@ bool btf_is_module(const struct btf *btf); struct module *btf_try_get_module(const struct btf *btf); struct btf *btf_get_module_btf(const struct module *module); void btf_put_module_btf(struct btf *btf); +u64 btf_get_module_btf_full_id(struct btf *btf, const char *name); u32 btf_nr_types(const struct btf *btf); bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, const struct btf_member *m, diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 1e95391e0ca1..10a859943a49 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7092,6 +7092,29 @@ struct btf *btf_get_module_btf(const struct module *module) } EXPORT_SYMBOL_GPL(btf_get_module_btf); +u64 btf_get_module_btf_full_id(struct btf *btf, const char *name) +{ + s32 type_id; + u64 obj_id; + + if (IS_ERR_OR_NULL(btf)) + return 0; + + obj_id = btf_obj_id(btf); + type_id = btf_find_by_name_kind(btf, name, BTF_KIND_STRUCT); + if (type_id < 0) { + pr_warn("Module %s(ID:%d): BTF cannot find struct %s", + btf->name, (u32)obj_id, name); + return 0; + } + + pr_info("Module %s(ID:%d): BTF type id %d for struct %s", + btf->name, (u32)obj_id, type_id, name); + + return type_id | (obj_id << 32); +} +EXPORT_SYMBOL_GPL(btf_get_module_btf_full_id); + BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int, flags) { struct btf *btf = NULL; From patchwork Wed Sep 7 15:45:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969164 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4FC1C6FA89 for ; Wed, 7 Sep 2022 15:46:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229692AbiIGPqT (ORCPT ); Wed, 7 Sep 2022 11:46:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229783AbiIGPqM (ORCPT ); Wed, 7 Sep 2022 11:46:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 814046FA0F for ; Wed, 7 Sep 2022 08:46:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RxXs7B8LQSH1QSFqxekwrMBinIO+X+jrbfvcwnHR9n0=; b=f+nMjHKAUkdJzZ0Jjkyt1c3bAK9FKqqXB1FPjDvnGNMLap12qzskQ68ZzReEb5nWN99SZz JsJ3z1j/6nKc7xrejqMyXTEirJjDjxgqGGscNOFU+7K3+KTVKhPFpv0ofEElEZTQDt3hNh gwW73RaWNECCg2uo3LyQ2NqAsU7kghs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-198-Hi2NMbUKP-unt3JTnucn4w-1; Wed, 07 Sep 2022 11:45:58 -0400 X-MC-Unique: Hi2NMbUKP-unt3JTnucn4w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ACCA03800C2E; Wed, 7 Sep 2022 15:45:57 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4A6E24010D2A; Wed, 7 Sep 2022 15:45:57 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 45F9930721A6C; Wed, 7 Sep 2022 17:45:56 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 11/18] i40e: add XDP-hints handling From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:45:56 +0200 Message-ID: <166256555623.1434226.18405044972840995177.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add two different XDP-hints named "xdp_hints_i40e" and "xdp_hints_i40e_timestamp". The "xdp_hints_i40e" struct is compatible with common struct, and extends with member i40e_hash_ptype (type struct i40e_rx_ptype_decoded) what contains more details on what protocol the packet contains. Info on IPv4 or IPv6, fragmented or not, L4 protocols UDP, TCP, SCTP, ICMP or timesync. The "xdp_hints_i40e_timestamp" struct is also compatible with common struct, and extends on top of "xdp_hints_i40e" by adding a 64-bit "rx_timestamp" provided by hardware. Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/i40e/i40e_main.c | 22 ++++ drivers/net/ethernet/intel/i40e/i40e_txrx.c | 155 ++++++++++++++++++++++++--- 2 files changed, 160 insertions(+), 17 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index b36bf9c3e1e4..50deaa25099e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -27,6 +28,10 @@ static const char i40e_driver_string[] = static const char i40e_copyright[] = "Copyright (c) 2013 - 2019 Intel Corporation."; +static struct btf *this_module_btf; +extern u64 btf_id_xdp_hints_i40e; +extern u64 btf_id_xdp_hints_i40e_timestamp; + /* a bit of forward declarations */ static void i40e_vsi_reinit_locked(struct i40e_vsi *vsi); static void i40e_handle_reset_warning(struct i40e_pf *pf, bool lock_acquired); @@ -13661,6 +13666,7 @@ static int i40e_config_netdev(struct i40e_vsi *vsi) NETIF_F_SCTP_CRC | NETIF_F_RXHASH | NETIF_F_RXCSUM | + NETIF_F_XDP_HINTS | 0; if (!(pf->hw_features & I40E_HW_OUTER_UDP_CSUM_CAPABLE)) @@ -13705,6 +13711,7 @@ static int i40e_config_netdev(struct i40e_vsi *vsi) netdev->hw_features |= hw_features; netdev->features |= hw_features | NETIF_F_HW_VLAN_CTAG_FILTER; + netdev->features |= NETIF_F_XDP_HINTS; netdev->hw_enc_features |= NETIF_F_TSO_MANGLEID; netdev->features &= ~NETIF_F_HW_TC; @@ -16617,6 +16624,15 @@ static struct pci_driver i40e_driver = { .sriov_configure = i40e_pci_sriov_configure, }; +static void i40e_this_module_btf_lookups(struct btf *btf) +{ + btf_id_xdp_hints_i40e = btf_get_module_btf_full_id(btf, + "xdp_hints_i40e"); + + btf_id_xdp_hints_i40e_timestamp = btf_get_module_btf_full_id(btf, + "xdp_hints_i40e_timestamp"); +} + /** * i40e_init_module - Driver registration routine * @@ -16628,6 +16644,10 @@ static int __init i40e_init_module(void) pr_info("%s: %s\n", i40e_driver_name, i40e_driver_string); pr_info("%s: %s\n", i40e_driver_name, i40e_copyright); + this_module_btf = btf_get_module_btf(THIS_MODULE); + if (this_module_btf) + i40e_this_module_btf_lookups(this_module_btf); + /* There is no need to throttle the number of active tasks because * each device limits its own task using a state bit for scheduling * the service task, and the device tasks do not interfere with each @@ -16658,5 +16678,7 @@ static void __exit i40e_exit_module(void) destroy_workqueue(i40e_wq); ida_destroy(&i40e_client_ida); i40e_dbg_exit(); + if (!IS_ERR_OR_NULL(this_module_btf)) + btf_put_module_btf(this_module_btf); } module_exit(i40e_exit_module); diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index a7a896321880..d945ac122d4c 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -1819,15 +1819,10 @@ _i40e_rx_checksum(struct i40e_vsi *vsi, ret.csum_level = 1; /* Only report checksum unnecessary for TCP, UDP, or SCTP */ - switch (decoded.inner_prot) { - case I40E_RX_PTYPE_INNER_PROT_TCP: - case I40E_RX_PTYPE_INNER_PROT_UDP: - case I40E_RX_PTYPE_INNER_PROT_SCTP: + if (likely(decoded.inner_prot == I40E_RX_PTYPE_INNER_PROT_TCP || + decoded.inner_prot == I40E_RX_PTYPE_INNER_PROT_UDP || + decoded.inner_prot == I40E_RX_PTYPE_INNER_PROT_SCTP)) ret.ip_summed = CHECKSUM_UNNECESSARY; - fallthrough; - default: - break; - } return ret; @@ -1858,19 +1853,17 @@ static inline void i40e_rx_checksum(struct i40e_vsi *vsi, /** * i40e_ptype_to_htype - get a hash type - * @ptype: the ptype value from the descriptor + * @ptype: the decoded ptype value from the descriptor * * Returns a hash type to be used by skb_set_hash **/ -static inline int i40e_ptype_to_htype(u8 ptype) +static inline int i40e_ptype_to_htype(struct i40e_rx_ptype_decoded decoded) { - struct i40e_rx_ptype_decoded decoded = decode_rx_desc_ptype(ptype); - - if (!decoded.known) + if (unlikely(!decoded.known)) return PKT_HASH_TYPE_NONE; - if (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP && - decoded.payload_layer == I40E_RX_PTYPE_PAYLOAD_LAYER_PAY4) + if (likely(decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP && + decoded.payload_layer == I40E_RX_PTYPE_PAYLOAD_LAYER_PAY4)) return PKT_HASH_TYPE_L4; else if (decoded.outer_ip == I40E_RX_PTYPE_OUTER_IP && decoded.payload_layer == I40E_RX_PTYPE_PAYLOAD_LAYER_PAY3) @@ -1900,8 +1893,11 @@ static inline void i40e_rx_hash(struct i40e_ring *ring, return; if ((rx_desc->wb.qword1.status_error_len & rss_mask) == rss_mask) { + struct i40e_rx_ptype_decoded ptype; + + ptype = decode_rx_desc_ptype(rx_ptype); hash = le32_to_cpu(rx_desc->wb.qword0.hi_dword.rss); - skb_set_hash(skb, hash, i40e_ptype_to_htype(rx_ptype)); + skb_set_hash(skb, hash, i40e_ptype_to_htype(ptype)); } } @@ -1947,6 +1943,129 @@ void i40e_process_skb_fields(struct i40e_ring *rx_ring, skb->protocol = eth_type_trans(skb, rx_ring->netdev); } +struct xdp_hints_i40e { + struct i40e_rx_ptype_decoded i40e_hash_ptype; + struct xdp_hints_common common; +}; + +struct xdp_hints_i40e_timestamp { + u64 rx_timestamp; + struct xdp_hints_i40e base; +}; + +/* Extending xdp_hints_flags */ +enum xdp_hints_flags_driver { + HINT_FLAG_RX_TIMESTAMP = BIT(16), +}; + +/* BTF full IDs gets looked up on driver i40e_init_module */ +u64 btf_id_xdp_hints_i40e; +u64 btf_id_xdp_hints_i40e_timestamp; + +static inline u32 i40e_rx_checksum_xdp(struct i40e_vsi *vsi, u64 qword1, + struct xdp_hints_i40e *xdp_hints, + struct i40e_rx_ptype_decoded ptype) +{ + struct i40e_rx_checksum_ret ret; + + ret = _i40e_rx_checksum(vsi, qword1, ptype); + return xdp_hints_set_rx_csum(&xdp_hints->common, ret.ip_summed, ret.csum_level); +} + +static inline u32 i40e_rx_hash_xdp(struct i40e_ring *ring, + union i40e_rx_desc *rx_desc, + struct xdp_buff *xdp, + u64 rx_desc_qword1, + struct xdp_hints_i40e *xdp_hints, + struct i40e_rx_ptype_decoded ptype + ) +{ + const u64 rss_mask = (u64)I40E_RX_DESC_FLTSTAT_RSS_HASH << + I40E_RX_DESC_STATUS_FLTSTAT_SHIFT; + u32 flags = 0; + + if (unlikely(!(ring->netdev->features & NETIF_F_RXHASH))) { + struct i40e_rx_ptype_decoded zero = {}; + + xdp_hints->i40e_hash_ptype = zero; + return 0; + } + + if (likely((rx_desc_qword1 & rss_mask) == rss_mask)) { + u32 hash = le32_to_cpu(rx_desc->wb.qword0.hi_dword.rss); + u32 htype; + + /* i40e provide extra information about protocol type */ + xdp_hints->i40e_hash_ptype = ptype; + htype = i40e_ptype_to_htype(ptype); + flags = xdp_hints_set_rx_hash(&xdp_hints->common, hash, htype); + } + return flags; +} + +static inline void i40e_process_xdp_hints(struct i40e_ring *rx_ring, + union i40e_rx_desc *rx_desc, + struct xdp_buff *xdp, + u64 qword) +{ + u32 rx_status = (qword & I40E_RXD_QW1_STATUS_MASK) >> + I40E_RXD_QW1_STATUS_SHIFT; + u32 tsynvalid = rx_status & I40E_RXD_QW1_STATUS_TSYNVALID_MASK; + u32 tsyn = (rx_status & I40E_RXD_QW1_STATUS_TSYNINDX_MASK) >> + I40E_RXD_QW1_STATUS_TSYNINDX_SHIFT; + u64 tsyn_ts; + + struct i40e_rx_ptype_decoded ptype; + struct xdp_hints_i40e *xdp_hints; + struct xdp_hints_common *common; + u32 btf_full_id = btf_id_xdp_hints_i40e; + u32 btf_sz = sizeof(*xdp_hints); + u32 f1 = 0, f2, f3, f4, f5 = 0; + u8 rx_ptype; + + if (!(rx_ring->netdev->features & NETIF_F_XDP_HINTS)) + return; + + /* Driver have xdp headroom when using build_skb */ + if (unlikely(!ring_uses_build_skb(rx_ring))) + return; + + xdp_hints = xdp->data - btf_sz; + common = &xdp_hints->common; + + if (unlikely(tsynvalid)) { + struct xdp_hints_i40e_timestamp *hints; + + tsyn_ts = i40e_ptp_rx_hwtstamp_raw(rx_ring->vsi->back, tsyn); + btf_full_id = btf_id_xdp_hints_i40e_timestamp; + btf_sz = sizeof(*hints); + hints = xdp->data - btf_sz; + hints->rx_timestamp = ns_to_ktime(tsyn_ts); + f1 = HINT_FLAG_RX_TIMESTAMP; + } + + /* ptype needed by both hash and checksum code */ + rx_ptype = (qword & I40E_RXD_QW1_PTYPE_MASK) >> I40E_RXD_QW1_PTYPE_SHIFT; + ptype = decode_rx_desc_ptype(rx_ptype); + + f2 = i40e_rx_hash_xdp(rx_ring, rx_desc, xdp, qword, xdp_hints, ptype); + f3 = i40e_rx_checksum_xdp(rx_ring->vsi, qword, xdp_hints, ptype); + f4 = xdp_hints_set_rxq(common, rx_ring->queue_index); + + if (unlikely(qword & BIT(I40E_RX_DESC_STATUS_L2TAG1P_SHIFT))) { + __le16 vlan_tag = rx_desc->wb.qword0.lo_dword.l2tag1; + + f5 = xdp_hints_set_vlan(common, le16_to_cpu(vlan_tag), + htons(ETH_P_8021Q)); + } + + xdp_hints_set_flags(common, (f1 | f2 | f3 | f4 | f5)); + common->btf_full_id = btf_full_id; + xdp->data_meta = xdp->data - btf_sz; + + xdp_buff_set_hints_flags(xdp, true); +} + /** * i40e_cleanup_headers - Correct empty headers * @rx_ring: rx descriptor ring packet is being transacted on @@ -2495,7 +2614,7 @@ static int i40e_clean_rx_irq(struct i40e_ring *rx_ring, int budget) */ dma_rmb(); - if (i40e_rx_is_programming_status(qword)) { + if (unlikely(i40e_rx_is_programming_status(qword))) { i40e_clean_programming_status(rx_ring, rx_desc->raw.qword[0], qword); @@ -2522,6 +2641,8 @@ static int i40e_clean_rx_irq(struct i40e_ring *rx_ring, int budget) rx_buffer->page_offset - offset; xdp_prepare_buff(&xdp, hard_start, offset, size, true); xdp_buff_clear_frags_flag(&xdp); + prefetchw(xdp.data - 8); /* xdp.data_meta cacheline */ + i40e_process_xdp_hints(rx_ring, rx_desc, &xdp, qword); #if (PAGE_SIZE > 4096) /* At larger PAGE_SIZE, frame_sz depend on len size */ xdp.frame_sz = i40e_rx_frame_truesize(rx_ring, size); From patchwork Wed Sep 7 15:46:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969165 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA291C6FA8A for ; Wed, 7 Sep 2022 15:46:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229731AbiIGPqU (ORCPT ); Wed, 7 Sep 2022 11:46:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229836AbiIGPqM (ORCPT ); Wed, 7 Sep 2022 11:46:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9149A82762 for ; Wed, 7 Sep 2022 08:46:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565566; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EQkFLDjoyFYBBnPe05h3hhYJQzwJrKql3Np1KjDu3/w=; b=XnjbzR/GMrf8EpWBF51QOw6wL2nv1GUn/JRvCpobCLtMUv2Z5c+E/Phmar/d7IRXgiyruY jl3dA/0OMRush1xbIjEf4pO00W6NBbr0sc79J1rJmXB4ktv+JheRH62qtyYQbG1n1pi0sb MPleh9A9SsDbOJt5i9jR4Dewpl5UYIc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-461-Fyr_K0d3Oyaoqn-ZWrtPjA-1; Wed, 07 Sep 2022 11:46:03 -0400 X-MC-Unique: Fyr_K0d3Oyaoqn-ZWrtPjA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B4B7B85A58D; Wed, 7 Sep 2022 15:46:02 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 556BD4010E4D; Wed, 7 Sep 2022 15:46:02 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 51EE930721A6C; Wed, 7 Sep 2022 17:46:01 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 12/18] net: use XDP-hints in xdp_frame to SKB conversion From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:01 +0200 Message-ID: <166256556130.1434226.10426110144984671774.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC This patch makes the net/core/xdp function __xdp_build_skb_from_frame() consume HW offloads provided via XDP-hints when creating an SKB based on an xdp_frame. This is an initial step towards SKB less drivers that moves SKB handing to net/core. Current users that already benefit from this are: Redirect into veth and cpumap. XDP_PASS action in bpf_test_run_xdp_live and driver ethernet/aquantia/atlantic/. Signed-off-by: Jesper Dangaard Brouer --- include/net/xdp.h | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++ net/core/xdp.c | 17 ++++++++----- 2 files changed, 83 insertions(+), 6 deletions(-) diff --git a/include/net/xdp.h b/include/net/xdp.h index c7cdcef83fa5..bdb497c7b296 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -153,6 +153,68 @@ static __always_inline u32 xdp_hints_set_vlan(struct xdp_hints_common *hints, return flags; } +/* XDP hints to SKB helper functions */ +static inline void xdp_hint2skb_record_rx_queue(struct sk_buff *skb, + struct xdp_hints_common *hints) +{ + if (hints->xdp_hints_flags & HINT_FLAG_RX_QUEUE) + skb_record_rx_queue(skb, hints->rx_queue); +} + +static inline void xdp_hint2skb_set_hash(struct sk_buff *skb, + struct xdp_hints_common *hints) +{ + u32 hash_type = hints->xdp_hints_flags & HINT_FLAG_RX_HASH_TYPE_MASK; + + if (hash_type) { + hash_type = hash_type >> HINT_FLAG_RX_HASH_TYPE_SHIFT; + skb_set_hash(skb, hints->rx_hash32, hash_type); + } +} + +static inline void xdp_hint2skb_checksum(struct sk_buff *skb, + struct xdp_hints_common *hints) +{ + u32 csum_type = hints->xdp_hints_flags & HINT_FLAG_CSUM_TYPE_MASK; + u32 csum_level = hints->xdp_hints_flags & HINT_FLAG_CSUM_LEVEL_MASK; + + if (csum_type == CHECKSUM_UNNECESSARY) + skb->ip_summed = CHECKSUM_UNNECESSARY; + + if (csum_level) + skb->csum_level = csum_level >> HINT_FLAG_CSUM_LEVEL_SHIFT; + + /* TODO: First driver implementing CHECKSUM_PARTIAL or CHECKSUM_COMPLETE + * need to implement handling here. + */ +} + +static inline void xdp_hint2skb_vlan_hw_tag(struct sk_buff *skb, + struct xdp_hints_common *hints) +{ + u32 flags = hints->xdp_hints_flags; + __be16 proto = htons(ETH_P_8021Q); + + if (flags & HINT_FLAG_VLAN_PROTO_ETH_P_8021AD) + proto = htons(ETH_P_8021AD); + + if (flags & HINT_FLAG_VLAN_PRESENT) { + /* like: __vlan_hwaccel_put_tag */ + skb->vlan_proto = proto; + skb->vlan_tci = hints->vlan_tci; + skb->vlan_present = 1; + } +} + +static inline void xdp_hint2skb(struct sk_buff *skb, + struct xdp_hints_common *hints) +{ + xdp_hint2skb_record_rx_queue(skb, hints); + xdp_hint2skb_set_hash(skb, hints); + xdp_hint2skb_checksum(skb, hints); + xdp_hint2skb_vlan_hw_tag(skb, hints); +} + /** * DOC: XDP RX-queue information * @@ -364,6 +426,16 @@ static __always_inline bool xdp_frame_is_frag_pfmemalloc(struct xdp_frame *frame return !!(frame->flags & XDP_FLAGS_FRAGS_PF_MEMALLOC); } +static __always_inline bool xdp_frame_has_hints_compat(struct xdp_frame *xdpf) +{ + u32 flags = xdpf->flags; + + if (!(flags & XDP_FLAGS_HINTS_COMPAT_COMMON)) + return false; + + return !!(flags & XDP_FLAGS_HINTS_MASK); +} + #define XDP_BULK_QUEUE_SIZE 16 struct xdp_frame_bulk { int count; diff --git a/net/core/xdp.c b/net/core/xdp.c index a57bd5278b47..ffa353367941 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -623,6 +623,7 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, struct net_device *dev) { struct skb_shared_info *sinfo = xdp_get_shared_info_from_frame(xdpf); + struct xdp_hints_common *xdp_hints = NULL; unsigned int headroom, frame_size; void *hard_start; u8 nr_frags; @@ -640,14 +641,17 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, frame_size = xdpf->frame_sz; hard_start = xdpf->data - headroom; + prefetch(xdpf->data); /* cache-line for eth_type_trans */ skb = build_skb_around(skb, hard_start, frame_size); if (unlikely(!skb)) return NULL; skb_reserve(skb, headroom); __skb_put(skb, xdpf->len); - if (xdpf->metasize) + if (xdpf->metasize) { skb_metadata_set(skb, xdpf->metasize); + prefetch(xdpf->data - sizeof(*xdp_hints)); + } if (unlikely(xdp_frame_has_frags(xdpf))) xdp_update_skb_shared_info(skb, nr_frags, @@ -658,11 +662,12 @@ struct sk_buff *__xdp_build_skb_from_frame(struct xdp_frame *xdpf, /* Essential SKB info: protocol and skb->dev */ skb->protocol = eth_type_trans(skb, dev); - /* Optional SKB info, currently missing: - * - HW checksum info (skb->ip_summed) - * - HW RX hash (skb_set_hash) - * - RX ring dev queue index (skb_record_rx_queue) - */ + /* Populate (optional) HW offload hints in SKB via XDP-hints */ + if (xdp_frame_has_hints_compat(xdpf) + && xdpf->metasize >= sizeof(*xdp_hints)) { + xdp_hints = xdpf->data - sizeof(*xdp_hints); + xdp_hint2skb(skb, xdp_hints); + } /* Until page_pool get SKB return path, release DMA here */ xdp_release_frame(xdpf); From patchwork Wed Sep 7 15:46:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969166 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FE93C38145 for ; Wed, 7 Sep 2022 15:46:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229785AbiIGPqX (ORCPT ); Wed, 7 Sep 2022 11:46:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229614AbiIGPqN (ORCPT ); Wed, 7 Sep 2022 11:46:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91FED6CF7C for ; Wed, 7 Sep 2022 08:46:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565571; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A559hgNhNxZLa/lcKgd8/J/9OLu6thHW1dOCvPXH/Cw=; b=TONa26D2EjRJX/xIeh1JnArf1O+1lXVKSXXp8QEOqW5Nobiz5xK2OsWpD92GQU9wzFTHSS ICGWEfvUtHRoO9qf6A51ikaptb8CejL/s+YPIEb6WN0fgqzxfjVaaiTFb0cOSPB+P0D5MI 2LksLa2bvyKwkI2bOnbjb+zZSulDUEI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-620-Y5EFfDaTN2euBjOXU9qFmA-1; Wed, 07 Sep 2022 11:46:08 -0400 X-MC-Unique: Y5EFfDaTN2euBjOXU9qFmA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CA63D811E87; Wed, 7 Sep 2022 15:46:07 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 66F4D945D2; Wed, 7 Sep 2022 15:46:07 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 5EA5530721A6C; Wed, 7 Sep 2022 17:46:06 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 13/18] mvneta: add XDP-hints support From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:06 +0200 Message-ID: <166256556634.1434226.5193053845415014583.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Lorenzo Bianconi In mvneta_rx_swbm() code path this driver already builds the SKB based on the xdp_buff. The natural next step is to use XDP-hints to populate the SKB fields, even when sending packets to normal netstack. The hardware/driver only support RX checksum offloading, which is stored as XDP-hints. Still the generic function xdp_hint2skb() that applies all common hints is called. This makes sense as an XDP bpf_prog have the opportunity to add some of these common hints prior to SKB creation. Signed-off-by: Lorenzo Bianconi Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/marvell/mvneta.c | 59 ++++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 0caa2df87c04..7d0055488a86 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -40,6 +40,7 @@ #include #include #include +#include /* Registers */ #define MVNETA_RXQ_CONFIG_REG(q) (0x1400 + ((q) << 2)) @@ -371,6 +372,9 @@ #define MVNETA_RX_GET_BM_POOL_ID(rxd) \ (((rxd)->status & MVNETA_RXD_BM_POOL_MASK) >> MVNETA_RXD_BM_POOL_SHIFT) +static struct btf *mvneta_btf; +static u64 btf_id_xdp_hints; + enum { ETHTOOL_STAT_EEE_WAKEUP, ETHTOOL_STAT_SKB_ALLOC_ERR, @@ -2308,12 +2312,15 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp, struct mvneta_rx_desc *rx_desc, struct mvneta_rx_queue *rxq, struct xdp_buff *xdp, int *size, - struct page *page) + struct page *page, u32 status) { unsigned char *data = page_address(page); int data_len = -MVNETA_MH_SIZE, len; + struct xdp_hints_common *xdp_hints; struct net_device *dev = pp->dev; enum dma_data_direction dma_dir; + u32 xdp_hints_flags; + u16 cksum; if (*size > MVNETA_MAX_RX_BUF_SIZE) { len = MVNETA_MAX_RX_BUF_SIZE; @@ -2336,6 +2343,20 @@ mvneta_swbm_rx_frame(struct mvneta_port *pp, xdp_buff_clear_frags_flag(xdp); xdp_prepare_buff(xdp, data, pp->rx_offset_correction + MVNETA_MH_SIZE, data_len, false); + + if (unlikely(!(pp->dev->features & NETIF_F_XDP_HINTS))) { + xdp_buff_clear_hints_flags(xdp); + return; + } + + xdp_hints = xdp->data - sizeof(*xdp_hints); + cksum = mvneta_rx_csum(pp, status); + xdp_hints_flags = xdp_hints_set_rx_csum(xdp_hints, cksum, 0); + xdp_hints_set_flags(xdp_hints, xdp_hints_flags); + xdp_hints->btf_full_id = btf_id_xdp_hints; + xdp->data_meta = xdp->data - sizeof(*xdp_hints); + + xdp_buff_set_hints_flags(xdp, true); } static void @@ -2385,9 +2406,25 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, *size -= len; } +static void +mvneta_set_skb_hints_from_xdp(struct xdp_buff *xdp, struct sk_buff *skb) +{ + struct xdp_hints_common *xdp_hints; + + if (!(xdp_buff_has_hints_compat(xdp))) + return; + + if (xdp->data - xdp->data_meta < sizeof(*xdp_hints)) + return; + + xdp_hints = xdp->data - sizeof(*xdp_hints); + xdp_hint2skb(skb, xdp_hints); +} + + static struct sk_buff * mvneta_swbm_build_skb(struct mvneta_port *pp, struct page_pool *pool, - struct xdp_buff *xdp, u32 desc_status) + struct xdp_buff *xdp) { struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); struct sk_buff *skb; @@ -2404,7 +2441,7 @@ mvneta_swbm_build_skb(struct mvneta_port *pp, struct page_pool *pool, skb_reserve(skb, xdp->data - xdp->data_hard_start); skb_put(skb, xdp->data_end - xdp->data); - skb->ip_summed = mvneta_rx_csum(pp, desc_status); + mvneta_set_skb_hints_from_xdp(xdp, skb); if (unlikely(xdp_buff_has_frags(xdp))) xdp_update_skb_shared_info(skb, num_frags, @@ -2424,8 +2461,8 @@ static int mvneta_rx_swbm(struct napi_struct *napi, struct net_device *dev = pp->dev; struct mvneta_stats ps = {}; struct bpf_prog *xdp_prog; - u32 desc_status, frame_sz; struct xdp_buff xdp_buf; + u32 frame_sz; xdp_init_buff(&xdp_buf, PAGE_SIZE, &rxq->xdp_rxq); xdp_buf.data_hard_start = NULL; @@ -2458,10 +2495,8 @@ static int mvneta_rx_swbm(struct napi_struct *napi, size = rx_desc->data_size; frame_sz = size - ETH_FCS_LEN; - desc_status = rx_status; - mvneta_swbm_rx_frame(pp, rx_desc, rxq, &xdp_buf, - &size, page); + &size, page, rx_status); } else { if (unlikely(!xdp_buf.data_hard_start)) { rx_desc->buf_phys_addr = 0; @@ -2487,7 +2522,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, mvneta_run_xdp(pp, rxq, xdp_prog, &xdp_buf, frame_sz, &ps)) goto next; - skb = mvneta_swbm_build_skb(pp, rxq->page_pool, &xdp_buf, desc_status); + skb = mvneta_swbm_build_skb(pp, rxq->page_pool, &xdp_buf); if (IS_ERR(skb)) { struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); @@ -5613,7 +5648,7 @@ static int mvneta_probe(struct platform_device *pdev) } dev->features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | - NETIF_F_TSO | NETIF_F_RXCSUM; + NETIF_F_TSO | NETIF_F_RXCSUM | NETIF_F_XDP_HINTS; dev->hw_features |= dev->features; dev->vlan_features |= dev->features; dev->priv_flags |= IFF_LIVE_ADDR_CHANGE; @@ -5817,6 +5852,11 @@ static int __init mvneta_driver_init(void) { int ret; + mvneta_btf = btf_get_module_btf(THIS_MODULE); + if (mvneta_btf) + btf_id_xdp_hints = btf_get_module_btf_full_id(mvneta_btf, + "xdp_hints_common"); + ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "net/mvneta:online", mvneta_cpu_online, mvneta_cpu_down_prepare); @@ -5844,6 +5884,7 @@ module_init(mvneta_driver_init); static void __exit mvneta_driver_exit(void) { + btf_put_module_btf(mvneta_btf); platform_driver_unregister(&mvneta_driver); cpuhp_remove_multi_state(CPUHP_NET_MVNETA_DEAD); cpuhp_remove_multi_state(online_hpstate); From patchwork Wed Sep 7 15:46:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969167 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C945C54EE9 for ; Wed, 7 Sep 2022 15:46:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229829AbiIGPqa (ORCPT ); Wed, 7 Sep 2022 11:46:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229715AbiIGPqU (ORCPT ); Wed, 7 Sep 2022 11:46:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 719696E2E8 for ; Wed, 7 Sep 2022 08:46:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M3j2ti94ulcMEx9Tac54YYCFvFTHx7LCXv/0luMg6Iw=; b=KPDM+pTvNd6ahMUkziYw4tBSiJoIwi4YUgHpgXGxI1z7dLDv+ox/MR+dR62Ol0Aq4lwamj WI5Wm4ui/pvz26sHzwz5AnGLOyV9TTRuSKETzcVSqOjjUi/03E4qypy9pPlHbXWo0g+0Tr LeB+y9cXm5jROIMEVRrwvmf0ImM1EEw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-669-UiJN6ZCfMv644hMeL_EwWA-1; Wed, 07 Sep 2022 11:46:13 -0400 X-MC-Unique: UiJN6ZCfMv644hMeL_EwWA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A9D4F101A54E; Wed, 7 Sep 2022 15:46:12 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6F08340B40CB; Wed, 7 Sep 2022 15:46:12 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 6AE6230721A6C; Wed, 7 Sep 2022 17:46:11 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 14/18] i40e: Add xdp_hints_union From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:11 +0200 Message-ID: <166256557140.1434226.13138556693050092410.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC The union named "xdp_hints_union" must contain all the xdp_hints_* struct's available in this driver. This is used when decoding the modules BTF to identify the available XDP-hints struct's. As metadata grows backwards padding are needed for proper alignment. This alignment is verified by compile time checks via BUILD_BUG_ON(). Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 31 +++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index d945ac122d4c..e21f3ff4c811 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -1953,6 +1953,36 @@ struct xdp_hints_i40e_timestamp { struct xdp_hints_i40e base; }; +/* xdp_hints_union defines xdp_hints_* structs available in this driver. + * As metadata grows backwards structure are padded to align. + */ +union xdp_hints_union { + struct xdp_hints_i40e_timestamp i40e_ts; + struct { + u64 pad1_ts; + struct xdp_hints_i40e i40e; + }; + struct { + u64 pad2_ts; + u32 pad3_i40e; + struct xdp_hints_common common; + }; +}; // __aligned(4) __attribute__((packed)); + +union xdp_hints_union define_union; + +#define OFFSET1 offsetof(union xdp_hints_union, common) +#define OFFSET2 offsetof(union xdp_hints_union, i40e.common) +#define OFFSET3 offsetof(union xdp_hints_union, i40e_ts.base.common) + +static void xdp_hints_compile_check(void) +{ + union xdp_hints_union my_union = {}; + + BUILD_BUG_ON(OFFSET1 != OFFSET2); + BUILD_BUG_ON(OFFSET1 != OFFSET3); +} + /* Extending xdp_hints_flags */ enum xdp_hints_flags_driver { HINT_FLAG_RX_TIMESTAMP = BIT(16), @@ -1968,6 +1998,7 @@ static inline u32 i40e_rx_checksum_xdp(struct i40e_vsi *vsi, u64 qword1, { struct i40e_rx_checksum_ret ret; + xdp_hints_compile_check(); ret = _i40e_rx_checksum(vsi, qword1, ptype); return xdp_hints_set_rx_csum(&xdp_hints->common, ret.ip_summed, ret.csum_level); } From patchwork Wed Sep 7 15:46:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969168 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21D49C6FA86 for ; Wed, 7 Sep 2022 15:46:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229833AbiIGPqc (ORCPT ); Wed, 7 Sep 2022 11:46:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229794AbiIGPqX (ORCPT ); Wed, 7 Sep 2022 11:46:23 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B1876FA0F for ; Wed, 7 Sep 2022 08:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565581; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/bHwryhcgQVyZoWewHxFzRoF7IbOXIWIyWvvNF2T2D4=; b=RDA2GP3g6WhGynJb96XpXvQqCl6++IsvRg7mDoOwqMxuOhPiR8t5IS8XOMzPib/iiHaCr8 vCb895sNeoRXiHrpTnW0MgujIM6BNChv0gxj58GnKxrPQPvVi+o24kPeuLWTviwPX4ltR0 w3WfNBU7qElMyh3d1pyxkEebINIBOuI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-460-YoCiClzfN6O4QuPx259pag-1; Wed, 07 Sep 2022 11:46:18 -0400 X-MC-Unique: YoCiClzfN6O4QuPx259pag-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EAF4885A585; Wed, 7 Sep 2022 15:46:17 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EC5D1121314; Wed, 7 Sep 2022 15:46:17 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 7913930721A6C; Wed, 7 Sep 2022 17:46:16 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 15/18] ixgbe: enable xdp-hints From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:16 +0200 Message-ID: <166256557645.1434226.13205220368236323814.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Maryam Tahhan Similar to i40e driver, add xdp hw-hints support for ixgbe driver in order to report rx csum offload for xdp_redirect. Signed-off-by: Maryam Tahhan Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 180 ++++++++++++++++++++++--- 1 file changed, 155 insertions(+), 25 deletions(-) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index d1e430b8c8aa..0c8ee19e6d44 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -60,6 +61,15 @@ static char ixgbe_default_device_descr[] = static const char ixgbe_copyright[] = "Copyright (c) 1999-2016 Intel Corporation."; +static struct btf *ixgbe_btf; + +struct xdp_hints_ixgbe { + u32 rss_type; + struct xdp_hints_common common; +}; + +u64 btf_id_xdp_hints_ixgbe; + static const char ixgbe_overheat_msg[] = "Network adapter has been stopped because it has over heated. Restart the computer. If the problem persists, power off the system and replace the adapter"; static const struct ixgbe_info *ixgbe_info_tbl[] = { @@ -1460,40 +1470,42 @@ static inline bool ixgbe_rx_is_fcoe(struct ixgbe_ring *ring, } #endif /* IXGBE_FCOE */ -/** - * ixgbe_rx_checksum - indicate in skb if hw indicated a good cksum - * @ring: structure containing ring specific data - * @rx_desc: current Rx descriptor being processed - * @skb: skb currently being received and modified - **/ -static inline void ixgbe_rx_checksum(struct ixgbe_ring *ring, - union ixgbe_adv_rx_desc *rx_desc, - struct sk_buff *skb) + +struct ixgbe_rx_checksum_ret { + u16 ip_summed; + u16 csum_level; + u8 encapsulation; +}; + +static inline struct ixgbe_rx_checksum_ret +_ixgbe_rx_checksum(struct ixgbe_ring *ring, + union ixgbe_adv_rx_desc *rx_desc, + __le16 pkt_info) { - __le16 pkt_info = rx_desc->wb.lower.lo_dword.hs_rss.pkt_info; bool encap_pkt = false; + struct ixgbe_rx_checksum_ret ret = {}; - skb_checksum_none_assert(skb); + ret.ip_summed = CHECKSUM_NONE; /* Rx csum disabled */ if (!(ring->netdev->features & NETIF_F_RXCSUM)) - return; + return ret; /* check for VXLAN and Geneve packets */ if (pkt_info & cpu_to_le16(IXGBE_RXDADV_PKTTYPE_VXLAN)) { encap_pkt = true; - skb->encapsulation = 1; + ret.encapsulation = 1; } /* if IP and error */ if (ixgbe_test_staterr(rx_desc, IXGBE_RXD_STAT_IPCS) && ixgbe_test_staterr(rx_desc, IXGBE_RXDADV_ERR_IPE)) { ring->rx_stats.csum_err++; - return; + return ret; } if (!ixgbe_test_staterr(rx_desc, IXGBE_RXD_STAT_L4CS)) - return; + return ret; if (ixgbe_test_staterr(rx_desc, IXGBE_RXDADV_ERR_TCPE)) { /* @@ -1501,26 +1513,49 @@ static inline void ixgbe_rx_checksum(struct ixgbe_ring *ring, * checksum errors. */ if ((pkt_info & cpu_to_le16(IXGBE_RXDADV_PKTTYPE_UDP)) && - test_bit(__IXGBE_RX_CSUM_UDP_ZERO_ERR, &ring->state)) - return; + test_bit(__IXGBE_RX_CSUM_UDP_ZERO_ERR, &ring->state)) + return ret; ring->rx_stats.csum_err++; - return; + return ret; } /* It must be a TCP or UDP packet with a valid checksum */ - skb->ip_summed = CHECKSUM_UNNECESSARY; + ret.ip_summed = CHECKSUM_UNNECESSARY; if (encap_pkt) { if (!ixgbe_test_staterr(rx_desc, IXGBE_RXD_STAT_OUTERIPCS)) - return; + return ret; if (ixgbe_test_staterr(rx_desc, IXGBE_RXDADV_ERR_OUTERIPER)) { - skb->ip_summed = CHECKSUM_NONE; - return; + ret.ip_summed = CHECKSUM_NONE; + return ret; } /* If we checked the outer header let the stack know */ - skb->csum_level = 1; + ret.csum_level = 1; } + + return ret; +} + +/** + * ixgbe_rx_checksum - indicate in skb if hw indicated a good cksum + * @ring: structure containing ring specific data + * @rx_desc: current Rx descriptor being processed + * @skb: skb currently being received and modified + **/ +static inline void ixgbe_rx_checksum(struct ixgbe_ring *ring, + union ixgbe_adv_rx_desc *rx_desc, + struct sk_buff *skb) +{ + struct ixgbe_rx_checksum_ret ret; + __le16 pkt_info = rx_desc->wb.lower.lo_dword.hs_rss.pkt_info; + + skb_checksum_none_assert(skb); + + ret = _ixgbe_rx_checksum(ring, rx_desc, pkt_info); + skb->ip_summed = ret.ip_summed; + skb->csum_level = ret.csum_level; + skb->encapsulation = ret.encapsulation; } static unsigned int ixgbe_rx_offset(struct ixgbe_ring *rx_ring) @@ -1714,6 +1749,85 @@ void ixgbe_process_skb_fields(struct ixgbe_ring *rx_ring, skb->protocol = eth_type_trans(skb, dev); } +static inline u32 +ixgbe_rx_checksum_xdp(struct ixgbe_ring *ring, + union ixgbe_adv_rx_desc *rx_desc, + struct xdp_hints_ixgbe *xdp_hints, + __le16 pkt_info) +{ + struct ixgbe_rx_checksum_ret ret = {}; + + ret = _ixgbe_rx_checksum(ring, rx_desc, pkt_info); + return xdp_hints_set_rx_csum(&xdp_hints->common, ret.ip_summed, ret.csum_level); +} + +static inline u32 ixgbe_rx_hash_xdp(struct ixgbe_ring *ring, + union ixgbe_adv_rx_desc *rx_desc, + struct xdp_hints_ixgbe *xdp_hints, + __le16 pkt_info) +{ + u32 flags = 0, hash, htype = PKT_HASH_TYPE_L2; + + xdp_hints->rss_type = 0; + + if (unlikely(!(ring->netdev->features & NETIF_F_RXHASH))) + return 0; + + xdp_hints->rss_type = le16_to_cpu(rx_desc->wb.lower.lo_dword.hs_rss.pkt_info) & + IXGBE_RXDADV_RSSTYPE_MASK; + + if (unlikely(!xdp_hints->rss_type)) + return 0; + + hash = le32_to_cpu(rx_desc->wb.lower.hi_dword.rss); + htype = (IXGBE_RSS_L4_TYPES_MASK & (1ul << xdp_hints->rss_type)) ? + PKT_HASH_TYPE_L4 : PKT_HASH_TYPE_L3; + flags = xdp_hints_set_rx_hash(&xdp_hints->common, hash, htype); + + return flags; +} + +static inline void ixgbe_process_xdp_hints(struct ixgbe_ring *ring, + union ixgbe_adv_rx_desc *rx_desc, + struct xdp_buff *xdp) +{ + __le16 pkt_info = rx_desc->wb.lower.lo_dword.hs_rss.pkt_info; + struct xdp_hints_ixgbe *xdp_hints; + struct xdp_hints_common *common; + u32 btf_id = btf_id_xdp_hints_ixgbe; + u32 btf_sz = sizeof(*xdp_hints); + u32 f1 = 0, f2, f3, f4, f5 = 0; + + if (!(ring->netdev->features & NETIF_F_XDP_HINTS)) { + xdp_buff_clear_hints_flags(xdp); + return; + } + + /* Driver have xdp headroom when using build_skb */ + if (unlikely(!ring_uses_build_skb(ring))) + return; + + xdp_hints = xdp->data - btf_sz; + common = &xdp_hints->common; + + f2 = ixgbe_rx_hash_xdp(ring, rx_desc, xdp_hints, pkt_info); + f3 = ixgbe_rx_checksum_xdp(ring, rx_desc, xdp_hints, pkt_info); + f4 = xdp_hints_set_rxq(common, ring->queue_index); + + if ((ring->netdev->features & NETIF_F_HW_VLAN_CTAG_RX) && + ixgbe_test_staterr(rx_desc, IXGBE_RXD_STAT_VP)) { + u16 vid = le16_to_cpu(rx_desc->wb.upper.vlan); + + f5 = xdp_hints_set_vlan(common, vid, htons(ETH_P_8021Q)); + } + + xdp_hints_set_flags(common, (f1 | f2 | f3 | f4 | f5)); + common->btf_full_id = btf_id; + xdp->data_meta = xdp->data - btf_sz; + + xdp_buff_set_hints_flags(xdp, true); +} + void ixgbe_rx_skb(struct ixgbe_q_vector *q_vector, struct sk_buff *skb) { @@ -2344,6 +2458,8 @@ static int ixgbe_clean_rx_irq(struct ixgbe_q_vector *q_vector, hard_start = page_address(rx_buffer->page) + rx_buffer->page_offset - offset; xdp_prepare_buff(&xdp, hard_start, offset, size, true); + prefetchw(xdp.data - 8); /* xdp.data_meta cacheline */ + ixgbe_process_xdp_hints(rx_ring, rx_desc, &xdp); xdp_buff_clear_frags_flag(&xdp); #if (PAGE_SIZE > 4096) /* At larger PAGE_SIZE, frame_sz depend on len size */ @@ -10963,7 +11079,8 @@ static int ixgbe_probe(struct pci_dev *pdev, const struct pci_device_id *ent) NETIF_F_TSO6 | NETIF_F_RXHASH | NETIF_F_RXCSUM | - NETIF_F_HW_CSUM; + NETIF_F_HW_CSUM | + NETIF_F_XDP_HINTS; #define IXGBE_GSO_PARTIAL_FEATURES (NETIF_F_GSO_GRE | \ NETIF_F_GSO_GRE_CSUM | \ @@ -11002,7 +11119,7 @@ static int ixgbe_probe(struct pci_dev *pdev, const struct pci_device_id *ent) netdev->features |= NETIF_F_HIGHDMA; netdev->vlan_features |= netdev->features | NETIF_F_TSO_MANGLEID; - netdev->hw_enc_features |= netdev->vlan_features; + netdev->hw_enc_features |= netdev->vlan_features | NETIF_F_XDP_HINTS; netdev->mpls_features |= NETIF_F_SG | NETIF_F_TSO | NETIF_F_TSO6 | @@ -11546,6 +11663,11 @@ static struct pci_driver ixgbe_driver = { .err_handler = &ixgbe_err_handler }; +static void ixgbe_this_module_btf_lookups(struct btf *btf) +{ + btf_id_xdp_hints_ixgbe = btf_get_module_btf_full_id(btf, "xdp_hints_ixgbe"); +} + /** * ixgbe_init_module - Driver Registration Routine * @@ -11555,6 +11677,7 @@ static struct pci_driver ixgbe_driver = { static int __init ixgbe_init_module(void) { int ret; + pr_info("%s\n", ixgbe_driver_string); pr_info("%s\n", ixgbe_copyright); @@ -11573,6 +11696,10 @@ static int __init ixgbe_init_module(void) return ret; } + ixgbe_btf = btf_get_module_btf(THIS_MODULE); + if (ixgbe_btf) + ixgbe_this_module_btf_lookups(ixgbe_btf); + #ifdef CONFIG_IXGBE_DCA dca_register_notify(&dca_notifier); #endif @@ -11600,6 +11727,9 @@ static void __exit ixgbe_exit_module(void) destroy_workqueue(ixgbe_wq); ixgbe_wq = NULL; } + + if (!IS_ERR_OR_NULL(ixgbe_btf)) + btf_put_module_btf(ixgbe_btf); } #ifdef CONFIG_IXGBE_DCA From patchwork Wed Sep 7 15:46:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969169 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 524B5C6FA86 for ; Wed, 7 Sep 2022 15:46:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230012AbiIGPqw (ORCPT ); Wed, 7 Sep 2022 11:46:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229796AbiIGPqb (ORCPT ); Wed, 7 Sep 2022 11:46:31 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01C736AA0F for ; Wed, 7 Sep 2022 08:46:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565590; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cS7W2e1Dl56yHRp9VGJuoJDiUQXh5Ga6c6qOZmdIdE0=; b=A33ygBHZnkK/zUbVVv2NOcrlGA7Ag3wIXBW6RM9ve+a2Iaor8opf+OWU+fMrEqBcgmf7hP cnQQWHj/ml4a70Iax6rdUsCXWfUYddjz7IDAXpJ0Kowhd6edGKukinRBAJMG+jViLdbP8b O03XTpVsx237zY+y78vW5NQFW9NvGvw= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-107-szCGiJBEP2eSDktWtJGiBw-1; Wed, 07 Sep 2022 11:46:23 -0400 X-MC-Unique: szCGiJBEP2eSDktWtJGiBw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 09AA12999B52; Wed, 7 Sep 2022 15:46:23 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8DB64945DB; Wed, 7 Sep 2022 15:46:22 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 8897730721A6C; Wed, 7 Sep 2022 17:46:21 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 16/18] ixgbe: add rx timestamp xdp hints support From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:21 +0200 Message-ID: <166256558150.1434226.3657678401785942330.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Maryam Tahhan Enable rx timestamp xdp-hints for ixgbe. Similar to i40e. Signed-off-by: Maryam Tahhan Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 2 + drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 37 +++++++++++ drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 82 ++++++++++++++++--------- 3 files changed, 90 insertions(+), 31 deletions(-) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h index 5369a97ff5ec..97b3fbd2de28 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h @@ -1023,6 +1023,8 @@ void ixgbe_ptp_rx_hang(struct ixgbe_adapter *adapter); void ixgbe_ptp_tx_hang(struct ixgbe_adapter *adapter); void ixgbe_ptp_rx_pktstamp(struct ixgbe_q_vector *, struct sk_buff *); void ixgbe_ptp_rx_rgtstamp(struct ixgbe_q_vector *, struct sk_buff *skb); +u64 ixgbe_ptp_convert_to_hwtstamp(struct ixgbe_adapter *adapter, u64 timestamp); +u64 ixgbe_ptp_rx_hwtstamp_raw(struct ixgbe_adapter *adapter); static inline void ixgbe_ptp_rx_hwtstamp(struct ixgbe_ring *rx_ring, union ixgbe_adv_rx_desc *rx_desc, struct sk_buff *skb) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index 0c8ee19e6d44..dc371b4c65bb 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -68,7 +68,18 @@ struct xdp_hints_ixgbe { struct xdp_hints_common common; }; +struct xdp_hints_ixgbe_timestamp { + u64 rx_timestamp; + struct xdp_hints_ixgbe base; +}; + +/* Extending xdp_hints_flags */ +enum xdp_hints_flags_driver { + HINT_FLAG_RX_TIMESTAMP = BIT(16), +}; + u64 btf_id_xdp_hints_ixgbe; +u64 btf_id_xdp_hints_ixgbe_timestamp; static const char ixgbe_overheat_msg[] = "Network adapter has been stopped because it has over heated. Restart the computer. If the problem persists, power off the system and replace the adapter"; @@ -1797,6 +1808,8 @@ static inline void ixgbe_process_xdp_hints(struct ixgbe_ring *ring, u32 btf_id = btf_id_xdp_hints_ixgbe; u32 btf_sz = sizeof(*xdp_hints); u32 f1 = 0, f2, f3, f4, f5 = 0; + u32 flags = ring->q_vector->adapter->flags; + struct ixgbe_q_vector *q_vector = ring->q_vector; if (!(ring->netdev->features & NETIF_F_XDP_HINTS)) { xdp_buff_clear_hints_flags(xdp); @@ -1810,6 +1823,25 @@ static inline void ixgbe_process_xdp_hints(struct ixgbe_ring *ring, xdp_hints = xdp->data - btf_sz; common = &xdp_hints->common; + if (q_vector && q_vector->adapter) { + if (unlikely(flags & IXGBE_FLAG_RX_HWTSTAMP_ENABLED)) { + u64 regval = 0, ns = 0; + struct xdp_hints_ixgbe_timestamp *hints; + + regval = ixgbe_ptp_rx_hwtstamp_raw(q_vector->adapter); + if (regval) { + ns = ixgbe_ptp_convert_to_hwtstamp(q_vector->adapter, regval); + if (ns) { + btf_id = btf_id_xdp_hints_ixgbe_timestamp; + btf_sz = sizeof(*hints); + hints = xdp->data - btf_sz; + hints->rx_timestamp = ns_to_ktime(ns); + f1 = HINT_FLAG_RX_TIMESTAMP; + } + } + } + } + f2 = ixgbe_rx_hash_xdp(ring, rx_desc, xdp_hints, pkt_info); f3 = ixgbe_rx_checksum_xdp(ring, rx_desc, xdp_hints, pkt_info); f4 = xdp_hints_set_rxq(common, ring->queue_index); @@ -11665,7 +11697,10 @@ static struct pci_driver ixgbe_driver = { static void ixgbe_this_module_btf_lookups(struct btf *btf) { - btf_id_xdp_hints_ixgbe = btf_get_module_btf_full_id(btf, "xdp_hints_ixgbe"); + btf_id_xdp_hints_ixgbe = btf_get_module_btf_full_id(btf, + "xdp_hints_ixgbe"); + btf_id_xdp_hints_ixgbe_timestamp = btf_get_module_btf_full_id(btf, + "xdp_hints_ixgbe_timestamp"); } /** diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c index 9f06896a049b..561265b2816e 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c @@ -379,11 +379,11 @@ static u64 ixgbe_ptp_read_82599(const struct cyclecounter *cc) /** * ixgbe_ptp_convert_to_hwtstamp - convert register value to hw timestamp * @adapter: private adapter structure - * @hwtstamp: stack timestamp structure * @timestamp: unsigned 64bit system time value * - * We need to convert the adapter's RX/TXSTMP registers into a hwtstamp value - * which can be used by the stack's ptp functions. + * We need to convert the adapter's RX/TXSTMP registers into a ns value + * which can be converted later to a hwtstamp to be used by the stack's + * ptp functions. * * The lock is used to protect consistency of the cyclecounter and the SYSTIME * registers. However, it does not need to protect against the Rx or Tx @@ -393,16 +393,13 @@ static u64 ixgbe_ptp_read_82599(const struct cyclecounter *cc) * In addition to the timestamp in hardware, some controllers need a software * overflow cyclecounter, and this function takes this into account as well. **/ -static void ixgbe_ptp_convert_to_hwtstamp(struct ixgbe_adapter *adapter, - struct skb_shared_hwtstamps *hwtstamp, - u64 timestamp) +u64 ixgbe_ptp_convert_to_hwtstamp(struct ixgbe_adapter *adapter, + u64 timestamp) { unsigned long flags; struct timespec64 systime; u64 ns; - memset(hwtstamp, 0, sizeof(*hwtstamp)); - switch (adapter->hw.mac.type) { /* X550 and later hardware supposedly represent time using a seconds * and nanoseconds counter, instead of raw 64bits nanoseconds. We need @@ -433,7 +430,7 @@ static void ixgbe_ptp_convert_to_hwtstamp(struct ixgbe_adapter *adapter, ns = timecounter_cyc2time(&adapter->hw_tc, timestamp); spin_unlock_irqrestore(&adapter->tmreg_lock, flags); - hwtstamp->hwtstamp = ns_to_ktime(ns); + return ns; } /** @@ -820,11 +817,13 @@ static void ixgbe_ptp_tx_hwtstamp(struct ixgbe_adapter *adapter) struct sk_buff *skb = adapter->ptp_tx_skb; struct ixgbe_hw *hw = &adapter->hw; struct skb_shared_hwtstamps shhwtstamps; - u64 regval = 0; + u64 regval = 0, ns = 0; regval |= (u64)IXGBE_READ_REG(hw, IXGBE_TXSTMPL); regval |= (u64)IXGBE_READ_REG(hw, IXGBE_TXSTMPH) << 32; - ixgbe_ptp_convert_to_hwtstamp(adapter, &shhwtstamps, regval); + ns = ixgbe_ptp_convert_to_hwtstamp(adapter, regval); + if (ns) + shhwtstamps.hwtstamp = ns_to_ktime(ns); /* Handle cleanup of the ptp_tx_skb ourselves, and unlock the state * bit prior to notifying the stack via skb_tstamp_tx(). This prevents @@ -892,6 +891,10 @@ void ixgbe_ptp_rx_pktstamp(struct ixgbe_q_vector *q_vector, struct sk_buff *skb) { __le64 regval; + u64 ns = 0; + struct skb_shared_hwtstamps *hwtstamp = skb_hwtstamps(skb); + + memset(hwtstamp, 0, sizeof(*hwtstamp)); /* copy the bits out of the skb, and then trim the skb length */ skb_copy_bits(skb, skb->len - IXGBE_TS_HDR_LEN, ®val, @@ -904,8 +907,35 @@ void ixgbe_ptp_rx_pktstamp(struct ixgbe_q_vector *q_vector, * DWORD: N N + 1 N + 2 * Field: End of Packet SYSTIMH SYSTIML */ - ixgbe_ptp_convert_to_hwtstamp(q_vector->adapter, skb_hwtstamps(skb), - le64_to_cpu(regval)); + ns = ixgbe_ptp_convert_to_hwtstamp(q_vector->adapter, le64_to_cpu(regval)); + if (ns) + hwtstamp->hwtstamp = ns_to_ktime(ns); +} + +/** + * ixgbe_ptp_rx_hwtstamp_raw - utility function which returns the RX time stamp + * @adapter: the private adapter struct + * + * If the timestamp is valid, we return the raw value, else return 0; + */ +u64 ixgbe_ptp_rx_hwtstamp_raw(struct ixgbe_adapter *adapter) +{ + struct ixgbe_hw *hw = &adapter->hw; + u32 tsyncrxctl; + u64 regval = 0; + + /* Read the tsyncrxctl register afterwards in order to prevent taking an + * I/O hit on every packet. + */ + + tsyncrxctl = IXGBE_READ_REG(hw, IXGBE_TSYNCRXCTL); + if (!(tsyncrxctl & IXGBE_TSYNCRXCTL_VALID)) + return 0; + + regval |= (u64)IXGBE_READ_REG(hw, IXGBE_RXSTMPL); + regval |= (u64)IXGBE_READ_REG(hw, IXGBE_RXSTMPH) << 32; + + return regval; } /** @@ -921,29 +951,21 @@ void ixgbe_ptp_rx_rgtstamp(struct ixgbe_q_vector *q_vector, struct sk_buff *skb) { struct ixgbe_adapter *adapter; - struct ixgbe_hw *hw; - u64 regval = 0; - u32 tsyncrxctl; + u64 regval = 0, ns = 0; + struct skb_shared_hwtstamps *hwtstamp = skb_hwtstamps(skb); /* we cannot process timestamps on a ring without a q_vector */ if (!q_vector || !q_vector->adapter) return; + memset(hwtstamp, 0, sizeof(*hwtstamp)); adapter = q_vector->adapter; - hw = &adapter->hw; - - /* Read the tsyncrxctl register afterwards in order to prevent taking an - * I/O hit on every packet. - */ - - tsyncrxctl = IXGBE_READ_REG(hw, IXGBE_TSYNCRXCTL); - if (!(tsyncrxctl & IXGBE_TSYNCRXCTL_VALID)) - return; - - regval |= (u64)IXGBE_READ_REG(hw, IXGBE_RXSTMPL); - regval |= (u64)IXGBE_READ_REG(hw, IXGBE_RXSTMPH) << 32; - - ixgbe_ptp_convert_to_hwtstamp(adapter, skb_hwtstamps(skb), regval); + regval = ixgbe_ptp_rx_hwtstamp_raw(adapter); + if (regval) { + ns = ixgbe_ptp_convert_to_hwtstamp(adapter, regval); + if (ns) + hwtstamp->hwtstamp = ns_to_ktime(ns); + } } /** From patchwork Wed Sep 7 15:46:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969170 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9E24C6FA86 for ; Wed, 7 Sep 2022 15:46:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229863AbiIGPq4 (ORCPT ); Wed, 7 Sep 2022 11:46:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229799AbiIGPqf (ORCPT ); Wed, 7 Sep 2022 11:46:35 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3A177674A for ; Wed, 7 Sep 2022 08:46:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565591; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z1uHBXgN3LQWGJMOETV0xAA89SFw6+HHNLzEdVhd4bc=; b=Wmqw4l1niQoOHaLwqpRuGtDmAi49XEIKak1kR6JVM9xu5pQNm0dPqxgplLqMQtiDBI3xOv Ts2vUh8cXdBhLLHTcaUCBDsjJI0axKswtIu4L3H76eeeDN5cydOt57VR4pa1qr2PrKr59I jTaJjbt5lvSM7FswDWVL5gvZFqGimqY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-599-fkIafPXaMWCR--5Do-l_ZQ-1; Wed, 07 Sep 2022 11:46:28 -0400 X-MC-Unique: fkIafPXaMWCR--5Do-l_ZQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D4DFB3C01D8B; Wed, 7 Sep 2022 15:46:27 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id 97DC290A04; Wed, 7 Sep 2022 15:46:27 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 9479D30721A6C; Wed, 7 Sep 2022 17:46:26 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 17/18] xsk: AF_XDP xdp-hints support in desc options From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:26 +0200 Message-ID: <166256558657.1434226.7390735974413846384.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Maryam Tahhan Simply set AF_XDP descriptor options to XDP flags. Jesper: Will this really be acceptable by AF_XDP maintainers? Signed-off-by: Maryam Tahhan --- include/uapi/linux/if_xdp.h | 2 +- net/xdp/xsk.c | 2 +- net/xdp/xsk_queue.h | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h index a78a8096f4ce..9335b56474e7 100644 --- a/include/uapi/linux/if_xdp.h +++ b/include/uapi/linux/if_xdp.h @@ -103,7 +103,7 @@ struct xdp_options { struct xdp_desc { __u64 addr; __u32 len; - __u32 options; + __u32 options; /* set to the values of xdp_hints_flags*/ }; /* UMEM descriptor is __u64 */ diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 5b4ce6ba1bc7..32095d78f06b 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -141,7 +141,7 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len) int err; addr = xp_get_handle(xskb); - err = xskq_prod_reserve_desc(xs->rx, addr, len); + err = xskq_prod_reserve_desc(xs->rx, addr, len, xdp->flags); if (err) { xs->rx_queue_full++; return err; diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index fb20bf7207cf..7a66f082f97e 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -368,7 +368,7 @@ static inline u32 xskq_prod_reserve_addr_batch(struct xsk_queue *q, struct xdp_d } static inline int xskq_prod_reserve_desc(struct xsk_queue *q, - u64 addr, u32 len) + u64 addr, u32 len, u32 flags) { struct xdp_rxtx_ring *ring = (struct xdp_rxtx_ring *)q->ring; u32 idx; @@ -380,6 +380,7 @@ static inline int xskq_prod_reserve_desc(struct xsk_queue *q, idx = q->cached_prod++ & q->ring_mask; ring->desc[idx].addr = addr; ring->desc[idx].len = len; + ring->desc[idx].options = flags; return 0; } From patchwork Wed Sep 7 15:46:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 12969171 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10D13C54EE9 for ; Wed, 7 Sep 2022 15:47:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229979AbiIGPrM (ORCPT ); Wed, 7 Sep 2022 11:47:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230013AbiIGPqw (ORCPT ); Wed, 7 Sep 2022 11:46:52 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0E3AB95A0 for ; Wed, 7 Sep 2022 08:46:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662565596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RpZB9X0DET31F7pJclkQNUipKM0FnFOUPz4JB+ooQnY=; b=WBjf/fOrIRRZbWK8IXtGipUjEIhFd7vCGWrF64FDKYuoQA6xH31PpU4HbR2uOXRJo8c2hg 5PEMIgDYBKs9D1m9TujwuYgbwx8IADGR88SJb5ZiJoS04CTdP/XlWtF2OsjaIYLq135yzJ bPE7DBjbW1ImfS1vr3am61BUxmqPGCA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-554-pSXxsqiNPUWLKJjMVin8Pg-1; Wed, 07 Sep 2022 11:46:33 -0400 X-MC-Unique: pSXxsqiNPUWLKJjMVin8Pg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E15878039B2; Wed, 7 Sep 2022 15:46:32 +0000 (UTC) Received: from firesoul.localdomain (unknown [10.40.208.22]) by smtp.corp.redhat.com (Postfix) with ESMTP id A63C6C15BB3; Wed, 7 Sep 2022 15:46:32 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id A37D430721A6C; Wed, 7 Sep 2022 17:46:31 +0200 (CEST) Subject: [PATCH RFCv2 bpf-next 18/18] ixgbe: AF_XDP xdp-hints processing in ixgbe_clean_rx_irq_zc From: Jesper Dangaard Brouer To: bpf@vger.kernel.org Cc: Jesper Dangaard Brouer , netdev@vger.kernel.org, xdp-hints@xdp-project.net, larysa.zaremba@intel.com, memxor@gmail.com, Lorenzo Bianconi , mtahhan@redhat.com, Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , dave@dtucker.co.uk, Magnus Karlsson , bjorn@kernel.org Date: Wed, 07 Sep 2022 17:46:31 +0200 Message-ID: <166256559162.1434226.13443800671075647862.stgit@firesoul> In-Reply-To: <166256538687.1434226.15760041133601409770.stgit@firesoul> References: <166256538687.1434226.15760041133601409770.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Maryam Tahhan Add XDP-hints processing to the AF_XDP zero-copy code path. Signed-off-by: Maryam Tahhan Signed-off-by: Jesper Dangaard Brouer --- drivers/net/ethernet/intel/ixgbe/ixgbe.h | 3 +++ drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 4 ++-- drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 2 ++ 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe.h b/drivers/net/ethernet/intel/ixgbe/ixgbe.h index 97b3fbd2de28..22eddadb3f7c 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe.h +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe.h @@ -1025,6 +1025,9 @@ void ixgbe_ptp_rx_pktstamp(struct ixgbe_q_vector *, struct sk_buff *); void ixgbe_ptp_rx_rgtstamp(struct ixgbe_q_vector *, struct sk_buff *skb); u64 ixgbe_ptp_convert_to_hwtstamp(struct ixgbe_adapter *adapter, u64 timestamp); u64 ixgbe_ptp_rx_hwtstamp_raw(struct ixgbe_adapter *adapter); +inline void ixgbe_process_xdp_hints(struct ixgbe_ring *ring, + union ixgbe_adv_rx_desc *rx_desc, + struct xdp_buff *xdp); static inline void ixgbe_ptp_rx_hwtstamp(struct ixgbe_ring *rx_ring, union ixgbe_adv_rx_desc *rx_desc, struct sk_buff *skb) diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index dc371b4c65bb..18f00f2bacaf 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -1798,7 +1798,7 @@ static inline u32 ixgbe_rx_hash_xdp(struct ixgbe_ring *ring, return flags; } -static inline void ixgbe_process_xdp_hints(struct ixgbe_ring *ring, +inline void ixgbe_process_xdp_hints(struct ixgbe_ring *ring, union ixgbe_adv_rx_desc *rx_desc, struct xdp_buff *xdp) { @@ -2395,7 +2395,7 @@ static struct sk_buff *ixgbe_run_xdp(struct ixgbe_adapter *adapter, return ERR_PTR(-result); } -static unsigned int ixgbe_rx_frame_truesize(struct ixgbe_ring *rx_ring, +static inline unsigned int ixgbe_rx_frame_truesize(struct ixgbe_ring *rx_ring, unsigned int size) { unsigned int truesize; diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c index 1703c640a434..c3fb8f7660df 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c @@ -304,7 +304,9 @@ int ixgbe_clean_rx_irq_zc(struct ixgbe_q_vector *q_vector, } bi->xdp->data_end = bi->xdp->data + size; + ixgbe_process_xdp_hints(rx_ring, rx_desc, bi->xdp); xsk_buff_dma_sync_for_cpu(bi->xdp, rx_ring->xsk_pool); + xdp_res = ixgbe_run_xdp_zc(adapter, rx_ring, bi->xdp); if (likely(xdp_res & (IXGBE_XDP_TX | IXGBE_XDP_REDIR))) {