From patchwork Fri Nov 1 02:38:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: mrpre X-Patchwork-Id: 13858615 X-Patchwork-Delegate: bpf@iogearbox.net Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.3]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B259A2B9A9; Fri, 1 Nov 2024 02:39:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730428807; cv=none; b=SoSUmqkWsUonDF4wZqU+ByVQTnnIkxl5FFRMsSskEHFurPLCYENXnI+l4Og0V1B4SM9Ck4mCmo6d2l5iehG8/wCXcmiuAOxJldyMjli+qkogp+xhGDQWyNRSghqjjgs4dVwGJ4tOkLQ4S9VrVOT/jkAeYlNfjItl8RmFpk2B5hI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730428807; c=relaxed/simple; bh=Z+ZkIB0RdM/+CPfXmsVZSiThc3L3iacF1dcFI3HhQo0=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=Gm7lQ8CTYUzuvPGHeL5N2gHb9Mx2bNDjJEYaI6U8SH00JrbbFt8CxgZYzRfZoGHBhO0PQfg1UFBeQN/geSEK8JbRih4GQ1xxBeEmFsktTkFAoz3q2s4tWkpUtYUZgjLNuOXE4SdbxrB1VVMoHPXUvEmWOuhjOFwdXi9ExuB0T4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=RMN/0boe; arc=none smtp.client-ip=117.135.210.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="RMN/0boe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:Subject:Date:Message-ID:MIME-Version; bh=H7+4E g8iqwnqiUSNdNxG+5d23N0V4b7WKti1AwEQikQ=; b=RMN/0boe6zwcC6oawUHuu 9CwUlKyl7e/WmxFrfdewsU/Ymi0UAD30DXPoRIQ8fwY658AyjG6dsfW2YAe12Hld FcsZzwVHSUIQJFapW35cfNBuq3V2TZWJKWatY1QTDGXxHuIphWVeT3iMW38pR4RW Qj6iG+DGHvHtU2eEf2t91Y= Received: from localhost.localdomain (unknown [47.252.33.72]) by gzga-smtp-mtada-g1-3 (Coremail) with SMTP id _____wCXf_dEPyRnhWzQCQ--.1082S2; Fri, 01 Nov 2024 10:39:06 +0800 (CST) From: mrpre To: yonghong.song@linux.dev, john.fastabend@gmail.com, martin.lau@kernel.org, edumazet@google.com, jakub@cloudflare.com, davem@davemloft.net, dsahern@kernel.org, kuba@kernel.org, pabeni@redhat.com, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org Cc: mrpre Subject: [PATCH 1/2] bpf: Introduce cpu affinity for sockmap Date: Fri, 1 Nov 2024 10:38:31 +0800 Message-ID: <20241101023832.32404-1-mrpre@163.com> X-Mailer: git-send-email 2.43.5 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: _____wCXf_dEPyRnhWzQCQ--.1082S2 X-Coremail-Antispam: 1Uf129KBjvJXoWfJF1rtFyrJFyUXw43JrWxtFb_yoWkXFWkpF Z0ka17CF4UJFW0qws0qaykZr4a9w18Kw1jkFZ3Ka4Syr92gr1FgFWrKF9ayF1Ykr4DCr4x ArsFgrWIy3yxZ3DanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0piX4S7UUUUU= X-CM-SenderInfo: xpus2vi6rwjhhfrp/1tbiWx6Kp2ckPncXwgAAs3 X-Patchwork-Delegate: bpf@iogearbox.net Why we need cpu affinity: Mainstream data planes, like Nginx and HAProxy, utilize CPU affinity by binding user processes to specific CPUs. This avoids interference between processes and prevents impact from other processes. Sockmap, as an optimization to accelerate such proxy programs, currently lacks the ability to specify CPU affinity. The current implementation of sockmap handling backlog is based on workqueue, which operates by calling 'schedule_delayed_work()'. It's current implementation prefers to schedule on the local CPU, i.e., the CPU that handled the packet under softirq. For extremely high traffic with large numbers of packets, 'sk_psock_backlog' becomes a large loop. For multi-threaded programs with only one map, we expect different sockets to run on different CPUs. It is important to note that this feature is not a general performance optimization. Instead, it provides users with the ability to bind to specific CPU, allowing them to enhance overall operating system utilization based on their own system environments. Implementation: 1.When updating the sockmap, support passing a CPU parameter and save it to the psock. 2.When scheduling psock, determine which CPU to run on using the psock's CPU information. 3.For thoes sockmap without CPU affinity, keep original logic by using 'schedule_delayed_work()'. Performance Testing: 'client <-> sockmap proxy <-> server' Using 'iperf3' tests, with the iperf server bound to CPU5 and the iperf client bound to CPU6, performance without using CPU affinity is around 34 Gbits/s, and CPU usage is concentrated on CPU5 and CPU6. ''' [ 5] local 127.0.0.1 port 57144 connected to 127.0.0.1 port 10000 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 3.95 GBytes 33.9 Gbits/sec [ 5] 1.00-2.00 sec 3.95 GBytes 34.0 Gbits/sec ...... ''' With using CPU affinity, the performnce is close to direct connection (without any proxy). ''' [ 5] local 127.0.0.1 port 56518 connected to 127.0.0.1 port 10000 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 7.76 GBytes 66.6 Gbits/sec [ 5] 1.00-2.00 sec 7.76 GBytes 66.7 Gbits/sec ...... ''' Signed-off-by: Jiayuan Chen --- include/linux/bpf.h | 3 ++- include/linux/skmsg.h | 8 ++++++++ include/uapi/linux/bpf.h | 4 ++++ kernel/bpf/syscall.c | 23 +++++++++++++++++------ net/core/skmsg.c | 11 +++++++---- net/core/sock_map.c | 12 +++++++----- 6 files changed, 45 insertions(+), 16 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c3ba4d475174..a56028c389e7 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3080,7 +3080,8 @@ int bpf_prog_test_run_syscall(struct bpf_prog *prog, int sock_map_get_from_fd(const union bpf_attr *attr, struct bpf_prog *prog); int sock_map_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype); -int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, u64 flags); +int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, u64 flags, + s32 target_cpu); int sock_map_bpf_prog_query(const union bpf_attr *attr, union bpf_attr __user *uattr); int sock_map_link_create(const union bpf_attr *attr, struct bpf_prog *prog); diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h index d9b03e0746e7..919425a92adf 100644 --- a/include/linux/skmsg.h +++ b/include/linux/skmsg.h @@ -117,6 +117,7 @@ struct sk_psock { struct delayed_work work; struct sock *sk_pair; struct rcu_work rwork; + s32 target_cpu; }; int sk_msg_alloc(struct sock *sk, struct sk_msg *msg, int len, @@ -514,6 +515,13 @@ static inline bool sk_psock_strp_enabled(struct sk_psock *psock) return !!psock->saved_data_ready; } +static inline int sk_psock_strp_get_cpu(struct sk_psock *psock) +{ + if (psock->target_cpu != -1) + return psock->target_cpu; + return WORK_CPU_UNBOUND; +} + #if IS_ENABLED(CONFIG_NET_SOCK_MSG) #define BPF_F_STRPARSER (1UL << 1) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index f28b6527e815..2019a87b5d4a 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1509,6 +1509,10 @@ union bpf_attr { __aligned_u64 next_key; }; __u64 flags; + union { + /* specify the CPU where the sockmap job run on */ + __aligned_u64 target_cpu; + }; }; struct { /* struct used by BPF_MAP_*_BATCH commands */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8254b2973157..95f719b9c3f3 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -239,10 +239,9 @@ static int bpf_obj_pin_uptrs(struct btf_record *rec, void *obj) } static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, - void *key, void *value, __u64 flags) + void *key, void *value, __u64 flags, s32 target_cpu) { int err; - /* Need to create a kthread, thus must support schedule */ if (bpf_map_is_offloaded(map)) { return bpf_map_offload_update_elem(map, key, value, flags); @@ -252,7 +251,7 @@ static int bpf_map_update_value(struct bpf_map *map, struct file *map_file, return map->ops->map_update_elem(map, key, value, flags); } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH || map->map_type == BPF_MAP_TYPE_SOCKMAP) { - return sock_map_update_elem_sys(map, key, value, flags); + return sock_map_update_elem_sys(map, key, value, flags, target_cpu); } else if (IS_FD_PROG_ARRAY(map)) { return bpf_fd_array_map_update_elem(map, map_file, key, value, flags); @@ -1680,12 +1679,14 @@ static int map_lookup_elem(union bpf_attr *attr) } -#define BPF_MAP_UPDATE_ELEM_LAST_FIELD flags +#define BPF_MAP_UPDATE_ELEM_LAST_FIELD target_cpu static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) { bpfptr_t ukey = make_bpfptr(attr->key, uattr.is_kernel); bpfptr_t uvalue = make_bpfptr(attr->value, uattr.is_kernel); + bpfptr_t utarget_cpu = make_bpfptr(attr->target_cpu, uattr.is_kernel); + s64 target_cpu = 0; struct bpf_map *map; void *key, *value; u32 value_size; @@ -1723,7 +1724,17 @@ static int map_update_elem(union bpf_attr *attr, bpfptr_t uattr) goto free_key; } - err = bpf_map_update_value(map, fd_file(f), key, value, attr->flags); + if (map->map_type == BPF_MAP_TYPE_SOCKMAP && + !bpfptr_is_null(utarget_cpu)) { + if (copy_from_bpfptr(&target_cpu, utarget_cpu, sizeof(target_cpu)) || + target_cpu > nr_cpu_ids) { + err = -EINVAL; + goto err_put; + } + } else { + target_cpu = -1; + } + err = bpf_map_update_value(map, fd_file(f), key, value, attr->flags, (s32)target_cpu); if (!err) maybe_wait_bpf_programs(map); @@ -1947,7 +1958,7 @@ int generic_map_update_batch(struct bpf_map *map, struct file *map_file, break; err = bpf_map_update_value(map, map_file, key, value, - attr->batch.elem_flags); + attr->batch.elem_flags, -1); if (err) break; diff --git a/net/core/skmsg.c b/net/core/skmsg.c index b1dcbd3be89e..d3b6f2468dab 100644 --- a/net/core/skmsg.c +++ b/net/core/skmsg.c @@ -679,7 +679,8 @@ static void sk_psock_backlog(struct work_struct *work) * other work that might be here. */ if (sk_psock_test_state(psock, SK_PSOCK_TX_ENABLED)) - schedule_delayed_work(&psock->work, 1); + schedule_delayed_work_on(sk_psock_strp_get_cpu(psock), + &psock->work, 1); goto end; } /* Hard errors break pipe and stop xmit. */ @@ -729,6 +730,7 @@ struct sk_psock *sk_psock_init(struct sock *sk, int node) psock->saved_destroy = prot->destroy; psock->saved_close = prot->close; psock->saved_write_space = sk->sk_write_space; + psock->target_cpu = -1; INIT_LIST_HEAD(&psock->link); spin_lock_init(&psock->link_lock); @@ -934,7 +936,7 @@ static int sk_psock_skb_redirect(struct sk_psock *from, struct sk_buff *skb) } skb_queue_tail(&psock_other->ingress_skb, skb); - schedule_delayed_work(&psock_other->work, 0); + schedule_delayed_work_on(sk_psock_strp_get_cpu(psock_other), &psock_other->work, 0); spin_unlock_bh(&psock_other->ingress_lock); return 0; } @@ -1012,7 +1014,8 @@ static int sk_psock_verdict_apply(struct sk_psock *psock, struct sk_buff *skb, spin_lock_bh(&psock->ingress_lock); if (sk_psock_test_state(psock, SK_PSOCK_TX_ENABLED)) { skb_queue_tail(&psock->ingress_skb, skb); - schedule_delayed_work(&psock->work, 0); + schedule_delayed_work_on(sk_psock_strp_get_cpu(psock), + &psock->work, 0); err = 0; } spin_unlock_bh(&psock->ingress_lock); @@ -1044,7 +1047,7 @@ static void sk_psock_write_space(struct sock *sk) psock = sk_psock(sk); if (likely(psock)) { if (sk_psock_test_state(psock, SK_PSOCK_TX_ENABLED)) - schedule_delayed_work(&psock->work, 0); + schedule_delayed_work_on(sk_psock_strp_get_cpu(psock), &psock->work, 0); write_space = psock->saved_write_space; } rcu_read_unlock(); diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 07d6aa4e39ef..36e9787c60de 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -465,7 +465,7 @@ static int sock_map_get_next_key(struct bpf_map *map, void *key, void *next) } static int sock_map_update_common(struct bpf_map *map, u32 idx, - struct sock *sk, u64 flags) + struct sock *sk, u64 flags, s32 target_cpu) { struct bpf_stab *stab = container_of(map, struct bpf_stab, map); struct sk_psock_link *link; @@ -490,6 +490,8 @@ static int sock_map_update_common(struct bpf_map *map, u32 idx, psock = sk_psock(sk); WARN_ON_ONCE(!psock); + psock->target_cpu = target_cpu; + spin_lock_bh(&stab->lock); osk = stab->sks[idx]; if (osk && flags == BPF_NOEXIST) { @@ -548,7 +550,7 @@ static int sock_hash_update_common(struct bpf_map *map, void *key, struct sock *sk, u64 flags); int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, - u64 flags) + u64 flags, s32 target_cpu) { struct socket *sock; struct sock *sk; @@ -579,7 +581,7 @@ int sock_map_update_elem_sys(struct bpf_map *map, void *key, void *value, if (!sock_map_sk_state_allowed(sk)) ret = -EOPNOTSUPP; else if (map->map_type == BPF_MAP_TYPE_SOCKMAP) - ret = sock_map_update_common(map, *(u32 *)key, sk, flags); + ret = sock_map_update_common(map, *(u32 *)key, sk, flags, target_cpu); else ret = sock_hash_update_common(map, key, sk, flags); sock_map_sk_release(sk); @@ -605,7 +607,7 @@ static long sock_map_update_elem(struct bpf_map *map, void *key, if (!sock_map_sk_state_allowed(sk)) ret = -EOPNOTSUPP; else if (map->map_type == BPF_MAP_TYPE_SOCKMAP) - ret = sock_map_update_common(map, *(u32 *)key, sk, flags); + ret = sock_map_update_common(map, *(u32 *)key, sk, flags, -1); else ret = sock_hash_update_common(map, key, sk, flags); bh_unlock_sock(sk); @@ -621,7 +623,7 @@ BPF_CALL_4(bpf_sock_map_update, struct bpf_sock_ops_kern *, sops, if (likely(sock_map_sk_is_suitable(sops->sk) && sock_map_op_okay(sops))) return sock_map_update_common(map, *(u32 *)key, sops->sk, - flags); + flags, -1); return -EOPNOTSUPP; } From patchwork Fri Nov 1 02:38:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: mrpre X-Patchwork-Id: 13858616 X-Patchwork-Delegate: bpf@iogearbox.net Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.3]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1CBCA73451; Fri, 1 Nov 2024 02:40:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730428829; cv=none; b=fpAviWnJCDC5kimbpmsgqktBEcDZ0ktCh6Ck9aNXv78Ye1395Tz0tVS0uhondKs4ecctpCs8mjNKmQnoSRTCXqnua6YkU6gtyFPOiKwLr63hb4gOXlIqn3Vl3ad/BY41qkAHOKwHbPW66q51aGDv2WlZxErILio8Jyb/1hj97iY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730428829; c=relaxed/simple; bh=HamhB/9kVT6nC1ZqKi2kZElwvGTbkV/1CeQUqskKmSs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kw6LMnfxLARA9Ct/95RyTdX05HlngAK7jo9XXgf04SOMeaHX0mB4g7rYz27x4q/YZ8t8ba9YJmKFRdrZOa4pzEtRiG7Ba69wDRJPKn1R4OwCSH1gVnfJ+b3rTraCfUbzAlTJ4Al9Vje6EwCSPqH9uQQP0+8T+0EKX29VJvyg2ho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=hRnK2B2H; arc=none smtp.client-ip=220.197.31.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="hRnK2B2H" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:Subject:Date:Message-ID:MIME-Version; bh=7I3LX 8cExmDiPWhipK+414Z78BH39l2/Zm0UHVryeA0=; b=hRnK2B2HpGKfAJ2U57jbO eyUELpf/1JC+ZawhUz6nOyTjaMfYxX7fQ2zYcRidaGxrfvGRLzpvoHUCEvQO3cHk Ku8kGjTUXz7jFb9zmht3V1KRWXi6AGMR2ns/RJ8PNenmSzeD7/sSspUICuQxKnX2 y1KjTZDU+V/wnxCv+bDnd8= Received: from localhost.localdomain (unknown [47.252.33.72]) by gzga-smtp-mtada-g1-3 (Coremail) with SMTP id _____wCXf_dEPyRnhWzQCQ--.1082S3; Fri, 01 Nov 2024 10:39:12 +0800 (CST) From: mrpre To: yonghong.song@linux.dev, john.fastabend@gmail.com, martin.lau@kernel.org, edumazet@google.com, jakub@cloudflare.com, davem@davemloft.net, dsahern@kernel.org, kuba@kernel.org, pabeni@redhat.com, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org Cc: mrpre Subject: [PATCH 2/2] bpf: implement libbpf sockmap cpu affinity Date: Fri, 1 Nov 2024 10:38:32 +0800 Message-ID: <20241101023832.32404-2-mrpre@163.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241101023832.32404-1-mrpre@163.com> References: <20241101023832.32404-1-mrpre@163.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CM-TRANSID: _____wCXf_dEPyRnhWzQCQ--.1082S3 X-Coremail-Antispam: 1Uf129KBjvJXoW3WrWkAr15tw1xuF1fGw1kuFg_yoW7ArWDpF 9YkF1SkF4Sqay5XrWYqa1IgrW5CF4Igw1jyrsrJ3W5ArnFgw1Iqr1Iyan3Ar13Wrs5Zw48 A34a9r4rJ348ZF7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0pEZ2-7UUUUU= X-CM-SenderInfo: xpus2vi6rwjhhfrp/xtbBDxKKp2ckMsj0iwACs0 X-Patchwork-Delegate: bpf@iogearbox.net implement libbpf sockmap cpu affinity Signed-off-by: Jiayuan Chen --- tools/include/uapi/linux/bpf.h | 4 ++++ tools/lib/bpf/bpf.c | 22 +++++++++++++++++++ tools/lib/bpf/bpf.h | 9 ++++++++ tools/lib/bpf/libbpf.map | 1 + .../selftests/bpf/prog_tests/sockmap_basic.c | 19 ++++++++++++---- 5 files changed, 51 insertions(+), 4 deletions(-) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index f28b6527e815..2019a87b5d4a 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1509,6 +1509,10 @@ union bpf_attr { __aligned_u64 next_key; }; __u64 flags; + union { + /* specify the CPU where the sockmap job run on */ + __aligned_u64 target_cpu; + }; }; struct { /* struct used by BPF_MAP_*_BATCH commands */ diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 2a4c71501a17..13c3f3cfe889 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -401,6 +401,28 @@ int bpf_map_update_elem(int fd, const void *key, const void *value, return libbpf_err_errno(ret); } +int bpf_map_update_elem_opts(int fd, const void *key, const void *value, + __u64 flags, const struct bpf_map_update_opts *opts) +{ + union bpf_attr attr; + int ret; + __u64 *target_cpu; + + if (!OPTS_VALID(opts, bpf_map_update_opts)) + return libbpf_err(-EINVAL); + + target_cpu = OPTS_GET(opts, target_cpu, NULL); + memset(&attr, 0, sizeof(attr)); + attr.map_fd = fd; + attr.key = ptr_to_u64(key); + attr.value = ptr_to_u64(value); + attr.flags = flags; + attr.target_cpu = ptr_to_u64(target_cpu); + + ret = sys_bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr)); + return libbpf_err_errno(ret); +} + int bpf_map_lookup_elem(int fd, const void *key, void *value) { const size_t attr_sz = offsetofend(union bpf_attr, flags); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index a4a7b1ad1b63..aec6dfddf697 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -147,6 +147,15 @@ LIBBPF_API int bpf_btf_load(const void *btf_data, size_t btf_size, LIBBPF_API int bpf_map_update_elem(int fd, const void *key, const void *value, __u64 flags); +struct bpf_map_update_opts { + size_t sz; /* size of this struct for forward/backward compatibility */ + /* specify the CPU where the sockmap job run on */ + __u64 *target_cpu; + size_t :0; +}; +#define bpf_map_update_opts__last_field target_cpu +LIBBPF_API int bpf_map_update_elem_opts(int fd, const void *key, const void *value, + __u64 flags, const struct bpf_map_update_opts *opts); LIBBPF_API int bpf_map_lookup_elem(int fd, const void *key, void *value); LIBBPF_API int bpf_map_lookup_elem_flags(int fd, const void *key, void *value, diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 54b6f312cfa8..ab5ec29f542d 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -17,6 +17,7 @@ LIBBPF_0.0.1 { bpf_map_lookup_and_delete_elem; bpf_map_lookup_elem; bpf_map_update_elem; + bpf_map_update_elem_opts; bpf_obj_get; bpf_obj_get_info_by_fd; bpf_obj_pin; diff --git a/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c b/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c index 82bfb266741c..84a35cb4b9fe 100644 --- a/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c +++ b/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c @@ -190,13 +190,18 @@ static void test_skmsg_helpers_with_link(enum bpf_map_type map_type) test_skmsg_load_helpers__destroy(skel); } -static void test_sockmap_update(enum bpf_map_type map_type) +static void test_sockmap_update(enum bpf_map_type map_type, bool cpu_affinity) { int err, prog, src; struct test_sockmap_update *skel; struct bpf_map *dst_map; const __u32 zero = 0; char dummy[14] = {0}; + __u64 target_cpu = 0; + + LIBBPF_OPTS(bpf_map_update_opts, update_opts, + .target_cpu = &target_cpu, + ); LIBBPF_OPTS(bpf_test_run_opts, topts, .data_in = dummy, .data_size_in = sizeof(dummy), @@ -219,7 +224,11 @@ static void test_sockmap_update(enum bpf_map_type map_type) else dst_map = skel->maps.dst_sock_hash; - err = bpf_map_update_elem(src, &zero, &sk, BPF_NOEXIST); + if (cpu_affinity) + err = bpf_map_update_elem_opts(src, &zero, &sk, BPF_NOEXIST, &update_opts); + else + err = bpf_map_update_elem(src, &zero, &sk, BPF_NOEXIST); + if (!ASSERT_OK(err, "update_elem(src)")) goto out; @@ -896,9 +905,11 @@ void test_sockmap_basic(void) if (test__start_subtest("sockhash sk_msg load helpers")) test_skmsg_helpers(BPF_MAP_TYPE_SOCKHASH); if (test__start_subtest("sockmap update")) - test_sockmap_update(BPF_MAP_TYPE_SOCKMAP); + test_sockmap_update(BPF_MAP_TYPE_SOCKMAP, false); + if (test__start_subtest("sockmap update cpu affinity")) + test_sockmap_update(BPF_MAP_TYPE_SOCKMAP, true); if (test__start_subtest("sockhash update")) - test_sockmap_update(BPF_MAP_TYPE_SOCKHASH); + test_sockmap_update(BPF_MAP_TYPE_SOCKHASH, false); if (test__start_subtest("sockmap update in unsafe context")) test_sockmap_invalid_update(); if (test__start_subtest("sockmap copy"))