From patchwork Thu Feb 23 01:12:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13149752 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7350C61DA4 for ; Thu, 23 Feb 2023 01:12:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232787AbjBWBMz (ORCPT ); Wed, 22 Feb 2023 20:12:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232785AbjBWBMy (ORCPT ); Wed, 22 Feb 2023 20:12:54 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA0DD44AA for ; Wed, 22 Feb 2023 17:12:51 -0800 (PST) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 31N0L8Rg028753 for ; Wed, 22 Feb 2023 17:12:51 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=Zbd/te8usnZP0uNPZmt1i2+H89k3iv3PDQU7aFpxgkc=; b=e+BwYBIAolrdnOWe3cQhLd3NHPBXycC1B57JE4uTt1xoEzTnzZNcxBrVddJou/Im7qEC NjTOst93jLZCdnjc9DMFUW7ENaXFS9AVJZsTSRS1qLs+wULHnpRkvwx69anCE2xOniha kUHEr+cv1a6GDhXK+O9JBo3ZXnCq2VEY/t/IVyzH7iC/gCJCVwnndMTIv9eRiREjU9De Nv1+lZ3gmx/7uPRlHAtAtl1UrfzkZr2lPT05zXDnwYPPHm/d7tTPvzFCbInsuIpqIFsc 0yK/jaP1ITk8qTOXSxw5RuMoN8suq9t0QTBCM3VPj1XNMpD5T5OZLN6d5blbp7iPLFmb jQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3nwdy9xs6n-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 22 Feb 2023 17:12:50 -0800 Received: from twshared16996.15.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 22 Feb 2023 17:12:48 -0800 Received: by devbig931.frc1.facebook.com (Postfix, from userid 460691) id 7A3935BD8CBD; Wed, 22 Feb 2023 17:12:40 -0800 (PST) From: Kui-Feng Lee To: , , , , , , CC: Kui-Feng Lee Subject: [PATCH bpf-next v2 1/6] bpf: Create links for BPF struct_ops maps. Date: Wed, 22 Feb 2023 17:12:33 -0800 Message-ID: <20230223011238.12313-2-kuifeng@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223011238.12313-1-kuifeng@meta.com> References: <20230223011238.12313-1-kuifeng@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: dGx3afYLazaIbLlxrdG9TAyox3sYT_hI X-Proofpoint-ORIG-GUID: dGx3afYLazaIbLlxrdG9TAyox3sYT_hI X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-22_12,2023-02-22_02,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net BPF struct_ops maps are employed directly to register TCP Congestion Control algorithms. Unlike other BPF programs that terminate when their links gone, the struct_ops program reduces its refcount solely upon death of its FD. The link of a BPF struct_ops map provides a uniform experience akin to other types of BPF programs. bpf_links are responsible for registering their associated struct_ops. You can only use a struct_ops that has the BPF_F_LINK flag set to create a bpf_link, while a structs without this flag behaves in the same manner as before and is registered upon updating its value. Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 11 + include/uapi/linux/bpf.h | 12 +- kernel/bpf/bpf_struct_ops.c | 376 ++++++++++++++++++++++++++++++--- kernel/bpf/syscall.c | 26 ++- tools/include/uapi/linux/bpf.h | 12 +- 5 files changed, 402 insertions(+), 35 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 8b5d0b4c4ada..9d6fd874e5ee 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1395,6 +1395,11 @@ struct bpf_link { struct work_struct work; }; +struct bpf_struct_ops_link { + struct bpf_link link; + struct bpf_map __rcu *map; +}; + struct bpf_link_ops { void (*release)(struct bpf_link *link); void (*dealloc)(struct bpf_link *link); @@ -1961,6 +1966,7 @@ int bpf_link_new_fd(struct bpf_link *link); struct file *bpf_link_new_file(struct bpf_link *link, int *reserved_fd); struct bpf_link *bpf_link_get_from_fd(u32 ufd); struct bpf_link *bpf_link_get_curr_or_next(u32 *id); +int bpf_struct_ops_link_create(union bpf_attr *attr); int bpf_obj_pin_user(u32 ufd, const char __user *pathname); int bpf_obj_get_user(const char __user *pathname, int flags); @@ -2305,6 +2311,11 @@ static inline void bpf_link_put(struct bpf_link *link) { } +static inline int bpf_struct_ops_link_create(union bpf_attr *attr) +{ + return -EOPNOTSUPP; +} + static inline int bpf_obj_get_user(const char __user *pathname, int flags) { return -EOPNOTSUPP; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 17afd2b35ee5..cd0ff39981e8 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1033,6 +1033,7 @@ enum bpf_attach_type { BPF_PERF_EVENT, BPF_TRACE_KPROBE_MULTI, BPF_LSM_CGROUP, + BPF_STRUCT_OPS, __MAX_BPF_ATTACH_TYPE }; @@ -1266,6 +1267,9 @@ enum { /* Create a map that is suitable to be an inner map with dynamic max entries */ BPF_F_INNER_MAP = (1U << 12), + +/* Create a map that will be registered/unregesitered by the backed bpf_link */ + BPF_F_LINK = (1U << 13), }; /* Flags for BPF_PROG_QUERY. */ @@ -1507,7 +1511,10 @@ union bpf_attr { } task_fd_query; struct { /* struct used by BPF_LINK_CREATE command */ - __u32 prog_fd; /* eBPF program to attach */ + union { + __u32 prog_fd; /* eBPF program to attach */ + __u32 map_fd; /* eBPF struct_ops to attach */ + }; union { __u32 target_fd; /* object to attach to */ __u32 target_ifindex; /* target ifindex */ @@ -6354,6 +6361,9 @@ struct bpf_link_info { struct { __u32 ifindex; } xdp; + struct { + __u32 map_id; + } struct_ops; }; } __attribute__((aligned(8))); diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index ece9870cab68..cfc69033c1b8 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -14,8 +14,10 @@ enum bpf_struct_ops_state { BPF_STRUCT_OPS_STATE_INIT, + BPF_STRUCT_OPS_STATE_UNREG, BPF_STRUCT_OPS_STATE_INUSE, BPF_STRUCT_OPS_STATE_TOBEFREE, + BPF_STRUCT_OPS_STATE_TOBEUNREG, }; #define BPF_STRUCT_OPS_COMMON_VALUE \ @@ -58,6 +60,8 @@ struct bpf_struct_ops_map { struct bpf_struct_ops_value kvalue; }; +static DEFINE_MUTEX(update_mutex); + #define VALUE_PREFIX "bpf_struct_ops_" #define VALUE_PREFIX_LEN (sizeof(VALUE_PREFIX) - 1) @@ -253,22 +257,23 @@ int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, if (unlikely(*(u32 *)key != 0)) return -ENOENT; + mutex_lock(&st_map->lock); + kvalue = &st_map->kvalue; - /* Pair with smp_store_release() during map_update */ state = smp_load_acquire(&kvalue->state); if (state == BPF_STRUCT_OPS_STATE_INIT) { memset(value, 0, map->value_size); + mutex_unlock(&st_map->lock); return 0; } - /* No lock is needed. state and refcnt do not need - * to be updated together under atomic context. - */ uvalue = value; memcpy(uvalue, st_map->uvalue, map->value_size); uvalue->state = state; refcount_set(&uvalue->refcnt, refcount_read(&kvalue->refcnt)); + mutex_unlock(&st_map->lock); + return 0; } @@ -349,6 +354,150 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, model, flags, tlinks, NULL); } +/* + * Maintain the state of kvalue. + * + * For a struct_ops that has no link, its state diagram is + * + * INIT ----> INUSE --> TOBEFREE + * ^ | + * | (refcnt == 0) | + * +---------------------+ + * + * For a struct_ops that has a link (BPF_F_LINK), its state diagram is + * + * (refcnt == 0) + * +-----------------------+ + * | | + * V | + * INIT ---> UNREG -+--> INUSE --> TOBEUNREG + * ^ | + * | V + * +---------- TOBEFREE + * (refcnt == 0) + * + * After transiting to the INUSE state of a struct_ops, the refcnt of + * its kvalue is set to 1. + * + * After transiting from the INUSE state of a struct_ops, the caller + * should decrease the refcnt of its kvalue by 1 by calling + * bpf_struct_ops_put(). + * + * TOBEFREE and TOBEUNREG are in a grace period, waiting for other + * tasks holding references of the struct_ops. When the refcnt drops + * from 1 to 0, TOBEFREE and TOBEUNREG are transited to INIT and UNREG + * respectively. + * + * It is safe to assume that there will be no registration race + * conditions after a task transits the same struct_ops to INUSE, + * TOBEFREE and TOBEUNREG states. The task is able to register or + * unregister the struct_ops without the need for any additional + * synchronization. + */ +static int bpf_struct_ops_transit_state(struct bpf_struct_ops_map *st_map, + enum bpf_struct_ops_state src, + enum bpf_struct_ops_state dst) +{ + int old_state; + + switch (src) { + case BPF_STRUCT_OPS_STATE_INIT: + if (dst != BPF_STRUCT_OPS_STATE_INUSE && + dst != BPF_STRUCT_OPS_STATE_UNREG) + return -EINVAL; + + old_state = cmpxchg(&st_map->kvalue.state, src, dst); + if (old_state != src) + break; + + if (dst == BPF_STRUCT_OPS_STATE_INUSE) + refcount_set(&st_map->kvalue.refcnt, 1); + break; + + case BPF_STRUCT_OPS_STATE_UNREG: + if (dst != BPF_STRUCT_OPS_STATE_INUSE && + dst != BPF_STRUCT_OPS_STATE_TOBEFREE) + return -EINVAL; + + old_state = cmpxchg(&st_map->kvalue.state, src, dst); + if (old_state != src) + break; + + if (dst == BPF_STRUCT_OPS_STATE_INUSE) + refcount_set(&st_map->kvalue.refcnt, 1); + else if (dst == BPF_STRUCT_OPS_STATE_TOBEFREE) + cmpxchg(&st_map->kvalue.state, dst, BPF_STRUCT_OPS_STATE_INIT); + break; + + case BPF_STRUCT_OPS_STATE_INUSE: + if (dst != BPF_STRUCT_OPS_STATE_TOBEFREE && + dst != BPF_STRUCT_OPS_STATE_TOBEUNREG) + return -EINVAL; + + old_state = cmpxchg(&st_map->kvalue.state, src, dst); + break; + + case BPF_STRUCT_OPS_STATE_TOBEFREE: + /* + * This transition should only be performed when the + * refcnt drops to 0 from 1. + */ + if (dst != BPF_STRUCT_OPS_STATE_INIT) + return -EINVAL; + old_state = cmpxchg(&st_map->kvalue.state, src, dst); + if (old_state != src) + break; + break; + + case BPF_STRUCT_OPS_STATE_TOBEUNREG: + /* + * This transition should only be performed when the + * refcnt drops to 0 from 1. + */ + if (dst != BPF_STRUCT_OPS_STATE_UNREG) + return -EINVAL; + old_state = cmpxchg(&st_map->kvalue.state, src, dst); + if (old_state != src) + return old_state; + break; + + default: + return -EOPNOTSUPP; + } + + return old_state; +} + +static int bpf_struct_ops_transit_state_check(struct bpf_struct_ops_map *st_map, + enum bpf_struct_ops_state src, + enum bpf_struct_ops_state dst) +{ + int err; + + err = bpf_struct_ops_transit_state(st_map, src, dst); + if (err < 0) + return err; + if (err != src) + return -EINVAL; + return 0; +} + +/* + * Restore the state of a struct_ops to UNREG from INUSE. + * + * It handles the case which a struct_ops transited to INUSE from + * UNREG successfully; somehow, need to rollback the struct_ops state. + */ +static void bpf_struct_ops_restore_unreg(struct bpf_struct_ops_map *st_map) +{ + struct bpf_struct_ops_value *kvalue; + + kvalue = &st_map->kvalue; + refcount_set(&kvalue->refcnt, 0); + /* Make sure the above change is seen before the state change. */ + smp_store_release(&kvalue->state, BPF_STRUCT_OPS_STATE_UNREG); +} + static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, void *value, u64 flags) { @@ -390,7 +539,11 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, mutex_lock(&st_map->lock); - if (kvalue->state != BPF_STRUCT_OPS_STATE_INIT) { + /* Make sure that all following changes are seen after the + * state value here. + */ + if (smp_load_acquire(&kvalue->state) >= BPF_STRUCT_OPS_STATE_INUSE || + refcount_read(&kvalue->refcnt)) { err = -EBUSY; goto unlock; } @@ -491,17 +644,21 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, *(unsigned long *)(udata + moff) = prog->aux->id; } - refcount_set(&kvalue->refcnt, 1); + if (st_map->map.map_flags & BPF_F_LINK) { + /* Let bpf_link handle registration & unregistration. */ + err = bpf_struct_ops_transit_state_check(st_map, BPF_STRUCT_OPS_STATE_INIT, + BPF_STRUCT_OPS_STATE_UNREG); + goto unlock; + } + bpf_map_inc(map); set_memory_rox((long)st_map->image, 1); err = st_ops->reg(kdata); if (likely(!err)) { - /* Pair with smp_load_acquire() during lookup_elem(). - * It ensures the above udata updates (e.g. prog->aux->id) - * can be seen once BPF_STRUCT_OPS_STATE_INUSE is set. - */ - smp_store_release(&kvalue->state, BPF_STRUCT_OPS_STATE_INUSE); + /* Infallible */ + bpf_struct_ops_transit_state(st_map, BPF_STRUCT_OPS_STATE_INIT, + BPF_STRUCT_OPS_STATE_INUSE); goto unlock; } @@ -526,28 +683,49 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, static int bpf_struct_ops_map_delete_elem(struct bpf_map *map, void *key) { - enum bpf_struct_ops_state prev_state; struct bpf_struct_ops_map *st_map; + int old_state; + int err = 0; st_map = (struct bpf_struct_ops_map *)map; - prev_state = cmpxchg(&st_map->kvalue.state, - BPF_STRUCT_OPS_STATE_INUSE, - BPF_STRUCT_OPS_STATE_TOBEFREE); - switch (prev_state) { + + old_state = bpf_struct_ops_transit_state(st_map, + (st_map->map.map_flags & BPF_F_LINK ? + BPF_STRUCT_OPS_STATE_UNREG : + BPF_STRUCT_OPS_STATE_INUSE), + BPF_STRUCT_OPS_STATE_TOBEFREE); + + if (old_state < 0) + return old_state; + + switch (old_state) { + case BPF_STRUCT_OPS_STATE_UNREG: + break; case BPF_STRUCT_OPS_STATE_INUSE: - st_map->st_ops->unreg(&st_map->kvalue.data); - if (refcount_dec_and_test(&st_map->kvalue.refcnt)) - bpf_map_put(map); - return 0; + if (st_map->map.map_flags & BPF_F_LINK) + err = -EBUSY; + else { + st_map->st_ops->unreg(&st_map->kvalue.data); + bpf_struct_ops_put(&st_map->kvalue.data); + } + break; case BPF_STRUCT_OPS_STATE_TOBEFREE: - return -EINPROGRESS; + err = -EINPROGRESS; + break; + case BPF_STRUCT_OPS_STATE_TOBEUNREG: + err = -EBUSY; + break; case BPF_STRUCT_OPS_STATE_INIT: - return -ENOENT; + err = -ENOENT; + break; default: WARN_ON_ONCE(1); /* Should never happen. Treat it as not found. */ - return -ENOENT; + err = -ENOENT; + break; } + + return err; } static void bpf_struct_ops_map_seq_show_elem(struct bpf_map *map, void *key, @@ -585,7 +763,7 @@ static void bpf_struct_ops_map_free(struct bpf_map *map) static int bpf_struct_ops_map_alloc_check(union bpf_attr *attr) { if (attr->key_size != sizeof(unsigned int) || attr->max_entries != 1 || - attr->map_flags || !attr->btf_vmlinux_value_type_id) + (attr->map_flags & ~BPF_F_LINK) || !attr->btf_vmlinux_value_type_id) return -EINVAL; return 0; } @@ -671,6 +849,15 @@ static void bpf_struct_ops_put_rcu(struct rcu_head *head) struct bpf_struct_ops_map *st_map; st_map = container_of(head, struct bpf_struct_ops_map, rcu); + + /* The struct_ops can be reused after a rcu grace period. */ + if (st_map->kvalue.state == BPF_STRUCT_OPS_STATE_TOBEFREE) + bpf_struct_ops_transit_state(st_map, BPF_STRUCT_OPS_STATE_TOBEFREE, + BPF_STRUCT_OPS_STATE_INIT); + else if (st_map->kvalue.state == BPF_STRUCT_OPS_STATE_TOBEUNREG) + bpf_struct_ops_transit_state(st_map, BPF_STRUCT_OPS_STATE_TOBEUNREG, + BPF_STRUCT_OPS_STATE_UNREG); + bpf_map_put(&st_map->map); } @@ -684,6 +871,7 @@ void bpf_struct_ops_put(const void *kdata) st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue); + /* The struct_ops's function may switch to another struct_ops. * * For example, bpf_tcp_cc_x->init() may switch to @@ -698,3 +886,143 @@ void bpf_struct_ops_put(const void *kdata) call_rcu(&st_map->rcu, bpf_struct_ops_put_rcu); } } + +static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link) +{ + struct bpf_struct_ops_link *st_link; + struct bpf_struct_ops_map *st_map; + + st_link = container_of(link, struct bpf_struct_ops_link, link); + if (st_link->map) { + st_map = (struct bpf_struct_ops_map *)st_link->map; + bpf_struct_ops_transit_state(st_map, BPF_STRUCT_OPS_STATE_INUSE, + (st_map->map.map_flags & BPF_F_LINK ? + BPF_STRUCT_OPS_STATE_TOBEUNREG : + BPF_STRUCT_OPS_STATE_TOBEFREE)); + st_map->st_ops->unreg(&st_map->kvalue.data); + bpf_struct_ops_put(&st_map->kvalue.data); + } + kfree(st_link); +} + +static int bpf_struct_ops_map_link_detach(struct bpf_link *link) +{ + struct bpf_struct_ops_link *st_link; + struct bpf_struct_ops_map *st_map; + + mutex_lock(&update_mutex); + st_link = container_of(link, struct bpf_struct_ops_link, link); + st_map = container_of(st_link->map, struct bpf_struct_ops_map, map); + if (st_map) { + /* + * All chaning on st_link->map are protected by + * update_mutex. This ensures that the struct_ops is + * INUSE, and the state transition always success. + */ + rcu_assign_pointer(st_link->map, NULL); + bpf_struct_ops_transit_state(st_map, BPF_STRUCT_OPS_STATE_INUSE, + (st_map->map.map_flags & BPF_F_LINK ? + BPF_STRUCT_OPS_STATE_TOBEUNREG : + BPF_STRUCT_OPS_STATE_TOBEFREE)); + st_map->st_ops->unreg(&st_map->kvalue.data); + bpf_struct_ops_put(&st_map->kvalue.data); + } + mutex_unlock(&update_mutex); + + return 0; +} + +static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_struct_ops_link *st_link; + struct bpf_map *map; + + st_link = container_of(link, struct bpf_struct_ops_link, link); + rcu_read_lock_trace(); + map = rcu_dereference(st_link->map); + if (map) + seq_printf(seq, "map_id:\t%d\n", map->id); + rcu_read_unlock_trace(); +} + +static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link, + struct bpf_link_info *info) +{ + struct bpf_struct_ops_link *st_link; + struct bpf_map *map; + + st_link = container_of(link, struct bpf_struct_ops_link, link); + rcu_read_lock_trace(); + map = rcu_dereference(st_link->map); + if (map) + info->struct_ops.map_id = map->id; + rcu_read_unlock_trace(); + return 0; +} + +static const struct bpf_link_ops bpf_struct_ops_map_lops = { + .dealloc = bpf_struct_ops_map_link_dealloc, + .detach = bpf_struct_ops_map_link_detach, + .show_fdinfo = bpf_struct_ops_map_link_show_fdinfo, + .fill_link_info = bpf_struct_ops_map_link_fill_link_info, +}; + +int bpf_struct_ops_link_create(union bpf_attr *attr) +{ + struct bpf_struct_ops_link *link = NULL; + struct bpf_link_primer link_primer; + struct bpf_struct_ops_map *st_map; + struct bpf_map *map; + int err; + + map = bpf_map_get(attr->link_create.map_fd); + if (!map) + return -EINVAL; + + if (map->map_type != BPF_MAP_TYPE_STRUCT_OPS || !(map->map_flags & BPF_F_LINK)) { + err = -EINVAL; + goto err_out; + } + + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) { + err = -ENOMEM; + goto err_out; + } + bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_map_lops, NULL); + link->map = map; + + st_map = (struct bpf_struct_ops_map *)map; + + err = bpf_struct_ops_transit_state_check(st_map, BPF_STRUCT_OPS_STATE_UNREG, + BPF_STRUCT_OPS_STATE_INUSE); + if (err) + goto err_out; + + err = bpf_link_prime(&link->link, &link_primer); + if (err) { + bpf_struct_ops_restore_unreg(st_map); + goto err_out; + } + + set_memory_rox((long)st_map->image, 1); + err = st_map->st_ops->reg(st_map->kvalue.data); + if (err) { + bpf_struct_ops_restore_unreg(st_map); + bpf_link_cleanup(&link_primer); + + set_memory_nx((long)st_map->image, 1); + set_memory_rw((long)st_map->image, 1); + goto err_out; + } + + + return bpf_link_settle(&link_primer); + +err_out: + bpf_map_put(map); + kfree(link); + return err; +} + diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index cda8d00f3762..2670de8dd0d4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2735,10 +2735,11 @@ void bpf_link_inc(struct bpf_link *link) static void bpf_link_free(struct bpf_link *link) { bpf_link_free_id(link->id); + /* detach BPF program, clean up used resources */ if (link->prog) { - /* detach BPF program, clean up used resources */ link->ops->release(link); bpf_prog_put(link->prog); + /* The struct_ops links clean up map by them-selves. */ } /* free bpf_link and its containing memory */ link->ops->dealloc(link); @@ -2794,16 +2795,19 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) const struct bpf_prog *prog = link->prog; char prog_tag[sizeof(prog->tag) * 2 + 1] = { }; - bin2hex(prog_tag, prog->tag, sizeof(prog->tag)); seq_printf(m, "link_type:\t%s\n" - "link_id:\t%u\n" - "prog_tag:\t%s\n" - "prog_id:\t%u\n", + "link_id:\t%u\n", bpf_link_type_strs[link->type], - link->id, - prog_tag, - prog->aux->id); + link->id); + if (prog) { + bin2hex(prog_tag, prog->tag, sizeof(prog->tag)); + seq_printf(m, + "prog_tag:\t%s\n" + "prog_id:\t%u\n", + prog_tag, + prog->aux->id); + } if (link->ops->show_fdinfo) link->ops->show_fdinfo(link, m); } @@ -4278,7 +4282,8 @@ static int bpf_link_get_info_by_fd(struct file *file, info.type = link->type; info.id = link->id; - info.prog_id = link->prog->aux->id; + if (link->prog) + info.prog_id = link->prog->aux->id; if (link->ops->fill_link_info) { err = link->ops->fill_link_info(link, &info); @@ -4541,6 +4546,9 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) if (CHECK_ATTR(BPF_LINK_CREATE)) return -EINVAL; + if (attr->link_create.attach_type == BPF_STRUCT_OPS) + return bpf_struct_ops_link_create(attr); + prog = bpf_prog_get(attr->link_create.prog_fd); if (IS_ERR(prog)) return PTR_ERR(prog); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 17afd2b35ee5..cd0ff39981e8 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1033,6 +1033,7 @@ enum bpf_attach_type { BPF_PERF_EVENT, BPF_TRACE_KPROBE_MULTI, BPF_LSM_CGROUP, + BPF_STRUCT_OPS, __MAX_BPF_ATTACH_TYPE }; @@ -1266,6 +1267,9 @@ enum { /* Create a map that is suitable to be an inner map with dynamic max entries */ BPF_F_INNER_MAP = (1U << 12), + +/* Create a map that will be registered/unregesitered by the backed bpf_link */ + BPF_F_LINK = (1U << 13), }; /* Flags for BPF_PROG_QUERY. */ @@ -1507,7 +1511,10 @@ union bpf_attr { } task_fd_query; struct { /* struct used by BPF_LINK_CREATE command */ - __u32 prog_fd; /* eBPF program to attach */ + union { + __u32 prog_fd; /* eBPF program to attach */ + __u32 map_fd; /* eBPF struct_ops to attach */ + }; union { __u32 target_fd; /* object to attach to */ __u32 target_ifindex; /* target ifindex */ @@ -6354,6 +6361,9 @@ struct bpf_link_info { struct { __u32 ifindex; } xdp; + struct { + __u32 map_id; + } struct_ops; }; } __attribute__((aligned(8))); From patchwork Thu Feb 23 01:12:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13149753 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63163C61DA4 for ; Thu, 23 Feb 2023 01:13:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232869AbjBWBNC (ORCPT ); Wed, 22 Feb 2023 20:13:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232645AbjBWBNB (ORCPT ); Wed, 22 Feb 2023 20:13:01 -0500 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76199DB for ; Wed, 22 Feb 2023 17:13:00 -0800 (PST) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31MNCTXF005575 for ; Wed, 22 Feb 2023 17:12:59 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=yrIycV+V2vWw4n9Nr+I18UnBoEkn0Vy76vFaqSEY3ck=; b=dmAmD8xWYo3nRZYCXqT1a9hdUla7FyDO/o0/vwe4afrQdrfL4ebRKN6b6f7sWZ5gJWX0 kNtvjunve0tmtFt+MkQEYsbpvfqmgw3Bi+lCvHx+pHwdpjoEpuuJ0TRUki7SB1D2eGw+ ORrAVPHRlapy/7GhJJfYqI7Ok+KEOa1RJoS2SE5iVlWgSAoHt4Nkdy15P497cTs2De0Y jOW4qE1sCFGCZxZF3x5AhSMsqXQijMMC+jb4iiMdHyCuS1F0RRxl2Vyzx6WHEGVOqLcU jg9ZrjZAdWdeYX/8EvR+DqfFu9s0yDb4RnnzFYYJhiqQwe2qYA8eKzi6Jne9XWmpdfIh nw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3nw7um0h10-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 22 Feb 2023 17:12:59 -0800 Received: from twshared1992.22.frc3.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 22 Feb 2023 17:12:58 -0800 Received: by devbig931.frc1.facebook.com (Postfix, from userid 460691) id A90995BD8CC0; Wed, 22 Feb 2023 17:12:40 -0800 (PST) From: Kui-Feng Lee To: , , , , , , CC: Kui-Feng Lee Subject: [PATCH bpf-next v2 2/6] net: Update an existing TCP congestion control algorithm. Date: Wed, 22 Feb 2023 17:12:34 -0800 Message-ID: <20230223011238.12313-3-kuifeng@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223011238.12313-1-kuifeng@meta.com> References: <20230223011238.12313-1-kuifeng@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: lOA6qxppvvVafnABPPMoh_2TDD4ItwrK X-Proofpoint-ORIG-GUID: lOA6qxppvvVafnABPPMoh_2TDD4ItwrK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-22_12,2023-02-22_02,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net This feature lets you immediately transition to another congestion control algorithm or implementation with the same name. Once a name is updated, new connections will apply this new algorithm. Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 1 + include/net/tcp.h | 2 ++ net/bpf/bpf_dummy_struct_ops.c | 6 ++++ net/ipv4/bpf_tcp_ca.c | 8 +++-- net/ipv4/tcp_cong.c | 58 ++++++++++++++++++++++++++++++---- 5 files changed, 66 insertions(+), 9 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 9d6fd874e5ee..7508ca89e814 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1451,6 +1451,7 @@ struct bpf_struct_ops { void *kdata, const void *udata); int (*reg)(void *kdata); void (*unreg)(void *kdata); + int (*update)(void *kdata, void *old_kdata); const struct btf_type *type; const struct btf_type *value_type; const char *name; diff --git a/include/net/tcp.h b/include/net/tcp.h index db9f828e9d1e..239cc0e2639c 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1117,6 +1117,8 @@ struct tcp_congestion_ops { int tcp_register_congestion_control(struct tcp_congestion_ops *type); void tcp_unregister_congestion_control(struct tcp_congestion_ops *type); +int tcp_update_congestion_control(struct tcp_congestion_ops *type, + struct tcp_congestion_ops *old_type); void tcp_assign_congestion_control(struct sock *sk); void tcp_init_congestion_control(struct sock *sk); diff --git a/net/bpf/bpf_dummy_struct_ops.c b/net/bpf/bpf_dummy_struct_ops.c index ff4f89a2b02a..158f14e240d0 100644 --- a/net/bpf/bpf_dummy_struct_ops.c +++ b/net/bpf/bpf_dummy_struct_ops.c @@ -222,12 +222,18 @@ static void bpf_dummy_unreg(void *kdata) { } +static int bpf_dummy_update(void *kdata, void *old_kdata) +{ + return -EOPNOTSUPP; +} + struct bpf_struct_ops bpf_bpf_dummy_ops = { .verifier_ops = &bpf_dummy_verifier_ops, .init = bpf_dummy_init, .check_member = bpf_dummy_ops_check_member, .init_member = bpf_dummy_init_member, .reg = bpf_dummy_reg, + .update = bpf_dummy_update, .unreg = bpf_dummy_unreg, .name = "bpf_dummy_ops", }; diff --git a/net/ipv4/bpf_tcp_ca.c b/net/ipv4/bpf_tcp_ca.c index 13fc0c185cd9..558b01d5250f 100644 --- a/net/ipv4/bpf_tcp_ca.c +++ b/net/ipv4/bpf_tcp_ca.c @@ -239,8 +239,6 @@ static int bpf_tcp_ca_init_member(const struct btf_type *t, if (bpf_obj_name_cpy(tcp_ca->name, utcp_ca->name, sizeof(tcp_ca->name)) <= 0) return -EINVAL; - if (tcp_ca_find(utcp_ca->name)) - return -EEXIST; return 1; } @@ -266,10 +264,16 @@ static void bpf_tcp_ca_unreg(void *kdata) tcp_unregister_congestion_control(kdata); } +static int bpf_tcp_ca_update(void *kdata, void *old_kdata) +{ + return tcp_update_congestion_control(kdata, old_kdata); +} + struct bpf_struct_ops bpf_tcp_congestion_ops = { .verifier_ops = &bpf_tcp_ca_verifier_ops, .reg = bpf_tcp_ca_reg, .unreg = bpf_tcp_ca_unreg, + .update = bpf_tcp_ca_update, .check_member = bpf_tcp_ca_check_member, .init_member = bpf_tcp_ca_init_member, .init = bpf_tcp_ca_init, diff --git a/net/ipv4/tcp_cong.c b/net/ipv4/tcp_cong.c index db8b4b488c31..98d33dad9062 100644 --- a/net/ipv4/tcp_cong.c +++ b/net/ipv4/tcp_cong.c @@ -75,14 +75,8 @@ struct tcp_congestion_ops *tcp_ca_find_key(u32 key) return NULL; } -/* - * Attach new congestion control algorithm to the list - * of available options. - */ -int tcp_register_congestion_control(struct tcp_congestion_ops *ca) +int tcp_ca_validate(struct tcp_congestion_ops *ca) { - int ret = 0; - /* all algorithms must implement these */ if (!ca->ssthresh || !ca->undo_cwnd || !(ca->cong_avoid || ca->cong_control)) { @@ -90,6 +84,20 @@ int tcp_register_congestion_control(struct tcp_congestion_ops *ca) return -EINVAL; } + return 0; +} + +/* Attach new congestion control algorithm to the list + * of available options. + */ +int tcp_register_congestion_control(struct tcp_congestion_ops *ca) +{ + int ret; + + ret = tcp_ca_validate(ca); + if (ret) + return ret; + ca->key = jhash(ca->name, sizeof(ca->name), strlen(ca->name)); spin_lock(&tcp_cong_list_lock); @@ -130,6 +138,42 @@ void tcp_unregister_congestion_control(struct tcp_congestion_ops *ca) } EXPORT_SYMBOL_GPL(tcp_unregister_congestion_control); +/* Replace a registered old ca with a new one. + * + * The new ca must have the same name as the old one, that has been + * registered. + */ +int tcp_update_congestion_control(struct tcp_congestion_ops *ca, struct tcp_congestion_ops *old_ca) +{ + struct tcp_congestion_ops *existing; + int ret; + + ret = tcp_ca_validate(ca); + if (ret) + return ret; + + ca->key = jhash(ca->name, sizeof(ca->name), strlen(ca->name)); + + spin_lock(&tcp_cong_list_lock); + existing = tcp_ca_find_key(ca->key); + if (ca->key == TCP_CA_UNSPEC || !existing || strcmp(existing->name, ca->name)) { + pr_notice("%s not registered or non-unique key\n", + ca->name); + ret = -EINVAL; + } else if (existing != old_ca) { + pr_notice("invalid old congestion control algorithm to replace\n"); + ret = -EINVAL; + } else { + list_del_rcu(&existing->list); + list_add_tail_rcu(&ca->list, &tcp_cong_list); + pr_debug("%s updated\n", ca->name); + } + spin_unlock(&tcp_cong_list_lock); + + return ret; +} +EXPORT_SYMBOL_GPL(tcp_update_congestion_control); + u32 tcp_ca_get_key_by_name(struct net *net, const char *name, bool *ecn_ca) { const struct tcp_congestion_ops *ca; From patchwork Thu Feb 23 01:12:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13149751 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BE3CC636D6 for ; Thu, 23 Feb 2023 01:12:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231820AbjBWBMx (ORCPT ); Wed, 22 Feb 2023 20:12:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232785AbjBWBMw (ORCPT ); Wed, 22 Feb 2023 20:12:52 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F20902D59 for ; Wed, 22 Feb 2023 17:12:49 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 31MNBsPi019909 for ; Wed, 22 Feb 2023 17:12:49 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=voRTy+0tce4KtlN7cCQNP3rHX9Y4fzbG9XrBeQXs0ek=; b=QwmNdkbtvY/DJV+e18eeJFaMp9Nc3KfM7Qh8uzKocAxPCmD0Ha/j+suxqa7tLJmIHuuv RKkrcL7MA6Elt754ldN1zLBcVuW3QFonwzOHaQEKMuGdWkKyvBc9P+oLmQ51/1rA1e7s Wte3yrzzaTz5EydpF5rPCpq5btHVeDBYACk/np2XApmAB3DgQmoIiUAMDOxt3bphZlAM +0JqK8djOqg7vLIlr7Nlu+doKXOCqzDh5+xYq9L+N4i9d1Nc3yyLEw915Biac6Jadf5r RHpT+55MMIlMoX7xyroaUbjRCpzjvD51W9hHIWJmo/1tj6bLMAnC2leb4Kiz771I7TkL 3A== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net (PPS) with ESMTPS id 3nw5n4sd5n-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 22 Feb 2023 17:12:48 -0800 Received: from twshared52565.14.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 22 Feb 2023 17:12:48 -0800 Received: by devbig931.frc1.facebook.com (Postfix, from userid 460691) id 6A0275BD8CFA; Wed, 22 Feb 2023 17:12:41 -0800 (PST) From: Kui-Feng Lee To: , , , , , , CC: Kui-Feng Lee Subject: [PATCH bpf-next v2 3/6] libbpf: Create a bpf_link in bpf_map__attach_struct_ops(). Date: Wed, 22 Feb 2023 17:12:35 -0800 Message-ID: <20230223011238.12313-4-kuifeng@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223011238.12313-1-kuifeng@meta.com> References: <20230223011238.12313-1-kuifeng@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: iaCQdnCJVSQMA4g37SCEh5b9h1nGM7XX X-Proofpoint-GUID: iaCQdnCJVSQMA4g37SCEh5b9h1nGM7XX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-22_12,2023-02-22_02,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net bpf_map__attach_struct_ops() was creating a dummy bpf_link as a placeholder, but now it is constructing an authentic one by calling bpf_link_create() if the map has the BPF_F_LINK flag. You can flag a struct_ops map with BPF_F_LINK by calling bpf_map__set_map_flags(). Signed-off-by: Kui-Feng Lee --- tools/lib/bpf/bpf.c | 2 + tools/lib/bpf/libbpf.c | 84 ++++++++++++++++++++++++++++++++---------- 2 files changed, 66 insertions(+), 20 deletions(-) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 9aff98f42a3d..6e270bb95ab7 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -731,6 +731,8 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, tracing)) return libbpf_err(-EINVAL); break; + case BPF_STRUCT_OPS: + break; default: if (!OPTS_ZEROED(opts, flags)) return libbpf_err(-EINVAL); diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 35a698eb825d..40f239cd1150 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -115,6 +115,7 @@ static const char * const attach_type_name[] = { [BPF_SK_REUSEPORT_SELECT_OR_MIGRATE] = "sk_reuseport_select_or_migrate", [BPF_PERF_EVENT] = "perf_event", [BPF_TRACE_KPROBE_MULTI] = "trace_kprobe_multi", + [BPF_STRUCT_OPS] = "struct_ops", }; static const char * const link_type_name[] = { @@ -7677,6 +7678,26 @@ static int bpf_object__resolve_externs(struct bpf_object *obj, return 0; } +static void bpf_map_prepare_vdata(const struct bpf_map *map) +{ + struct bpf_struct_ops *st_ops; + __u32 i; + + st_ops = map->st_ops; + for (i = 0; i < btf_vlen(st_ops->type); i++) { + struct bpf_program *prog = st_ops->progs[i]; + void *kern_data; + int prog_fd; + + if (!prog) + continue; + + prog_fd = bpf_program__fd(prog); + kern_data = st_ops->kern_vdata + st_ops->kern_func_off[i]; + *(unsigned long *)kern_data = prog_fd; + } +} + static int bpf_object_load(struct bpf_object *obj, int extra_log_level, const char *target_btf_path) { int err, i; @@ -7728,6 +7749,10 @@ static int bpf_object_load(struct bpf_object *obj, int extra_log_level, const ch btf__free(obj->btf_vmlinux); obj->btf_vmlinux = NULL; + for (i = 0; i < obj->nr_maps; i++) + if (bpf_map__is_struct_ops(&obj->maps[i])) + bpf_map_prepare_vdata(&obj->maps[i]); + obj->loaded = true; /* doesn't matter if successfully or not */ if (err) @@ -11429,11 +11454,29 @@ struct bpf_link *bpf_program__attach(const struct bpf_program *prog) return link; } +struct bpf_link_struct_ops { + struct bpf_link link; + int map_fd; +}; + static int bpf_link__detach_struct_ops(struct bpf_link *link) { + struct bpf_link_struct_ops *st_link; __u32 zero = 0; - if (bpf_map_delete_elem(link->fd, &zero)) + st_link = container_of(link, struct bpf_link_struct_ops, link); + + if (st_link->map_fd < 0) { + /* Fake bpf_link */ + if (bpf_map_delete_elem(link->fd, &zero)) + return -errno; + return 0; + } + + if (bpf_map_delete_elem(st_link->map_fd, &zero)) + return -errno; + + if (close(link->fd)) return -errno; return 0; @@ -11441,10 +11484,9 @@ static int bpf_link__detach_struct_ops(struct bpf_link *link) struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map) { - struct bpf_struct_ops *st_ops; - struct bpf_link *link; - __u32 i, zero = 0; - int err; + struct bpf_link_struct_ops *link; + __u32 zero = 0; + int err, fd; if (!bpf_map__is_struct_ops(map) || map->fd == -1) return libbpf_err_ptr(-EINVAL); @@ -11453,31 +11495,33 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map) if (!link) return libbpf_err_ptr(-EINVAL); - st_ops = map->st_ops; - for (i = 0; i < btf_vlen(st_ops->type); i++) { - struct bpf_program *prog = st_ops->progs[i]; - void *kern_data; - int prog_fd; + err = bpf_map_update_elem(map->fd, &zero, map->st_ops->kern_vdata, 0); + if (err) { + err = -errno; + free(link); + return libbpf_err_ptr(err); + } - if (!prog) - continue; + link->link.detach = bpf_link__detach_struct_ops; - prog_fd = bpf_program__fd(prog); - kern_data = st_ops->kern_vdata + st_ops->kern_func_off[i]; - *(unsigned long *)kern_data = prog_fd; + if (!(map->def.map_flags & BPF_F_LINK)) { + /* Fake bpf_link */ + link->link.fd = map->fd; + link->map_fd = -1; + return &link->link; } - err = bpf_map_update_elem(map->fd, &zero, st_ops->kern_vdata, 0); - if (err) { + fd = bpf_link_create(map->fd, -1, BPF_STRUCT_OPS, NULL); + if (fd < 0) { err = -errno; free(link); return libbpf_err_ptr(err); } - link->detach = bpf_link__detach_struct_ops; - link->fd = map->fd; + link->link.fd = fd; + link->map_fd = map->fd; - return link; + return &link->link; } typedef enum bpf_perf_event_ret (*bpf_perf_event_print_t)(struct perf_event_header *hdr, From patchwork Thu Feb 23 01:12:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13149754 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E84CC64ED6 for ; Thu, 23 Feb 2023 01:13:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232645AbjBWBND (ORCPT ); Wed, 22 Feb 2023 20:13:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231388AbjBWBNC (ORCPT ); Wed, 22 Feb 2023 20:13:02 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79A19769B for ; Wed, 22 Feb 2023 17:13:01 -0800 (PST) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 31MNC7ea020466 for ; Wed, 22 Feb 2023 17:13:00 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=DsA0yP4DYVtrC7j2kQbeoPGw5gxnkaT42Cu/MPsMZBM=; b=d7AofC5qJjx2T4Ej5o7hZIGJXKtitSq5MwOLU6CdyOl+rfq6qo6XN3dIe6Y/J8Rqxz+C 3JU5HSsMz6XrEZ07pzH4+1fcLIPIgNPcLLbHis+tzDQEa/qT36QcIjqMxq/fpjJlNE9D 9OMlaVes5TYslrgvozAw9Ft2J6D6SirvsP0pU+Spe9gaRIONn2oCxvVMdfty9mScW7FD fNVhCwscA4sR296yvmK8Nli5gSZUk+M56SglSjl9Baaz4rEAau5InUW3cKQtjqLAno+B kWfa6q19mbWeomy1shHi2MGUTPJIpOcv+20NlHfJTx4Rtg6G4b0ALM8Ro9LJEOVKuz8Z Pg== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0001303.ppops.net (PPS) with ESMTPS id 3nw5n4sd6w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 22 Feb 2023 17:13:00 -0800 Received: from twshared16996.15.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::e) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 22 Feb 2023 17:12:59 -0800 Received: by devbig931.frc1.facebook.com (Postfix, from userid 460691) id 113C85BD8CFF; Wed, 22 Feb 2023 17:12:41 -0800 (PST) From: Kui-Feng Lee To: , , , , , , CC: Kui-Feng Lee Subject: [PATCH bpf-next v2 4/6] bpf: Update the struct_ops of a bpf_link. Date: Wed, 22 Feb 2023 17:12:36 -0800 Message-ID: <20230223011238.12313-5-kuifeng@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223011238.12313-1-kuifeng@meta.com> References: <20230223011238.12313-1-kuifeng@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: e0gXe6xXgc5Wl8_oaP3bMN5WzwIN5tax X-Proofpoint-GUID: e0gXe6xXgc5Wl8_oaP3bMN5WzwIN5tax X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-22_12,2023-02-22_02,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net By improving the BPF_LINK_UPDATE command of bpf(), it should allow you to conveniently switch between different struct_ops on a single bpf_link. This would enable smoother transitions from one struct_ops to another. The struct_ops maps passing along with BPF_LINK_UPDATE should have the BPF_F_LINK flag. Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 1 + include/uapi/linux/bpf.h | 8 +++-- kernel/bpf/bpf_struct_ops.c | 69 +++++++++++++++++++++++++++++++++++++ kernel/bpf/syscall.c | 32 +++++++++++++++++ 4 files changed, 108 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 7508ca89e814..9797d9d87a3e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1409,6 +1409,7 @@ struct bpf_link_ops { void (*show_fdinfo)(const struct bpf_link *link, struct seq_file *seq); int (*fill_link_info)(const struct bpf_link *link, struct bpf_link_info *info); + int (*update_struct_ops)(struct bpf_link *link, struct bpf_map *new_map); }; struct bpf_tramp_link { diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index cd0ff39981e8..0702e88f7c08 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1555,8 +1555,12 @@ union bpf_attr { struct { /* struct used by BPF_LINK_UPDATE command */ __u32 link_fd; /* link fd */ - /* new program fd to update link with */ - __u32 new_prog_fd; + union { + /* new program fd to update link with */ + __u32 new_prog_fd; + /* new struct_ops map fd to update link with */ + __u32 new_map_fd; + }; __u32 flags; /* extra flags */ /* expected link's program fd; is specified only if * BPF_F_REPLACE flag is set in flags */ diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index cfc69033c1b8..700dc95a6daa 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -961,11 +961,80 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link, return 0; } +static int bpf_struct_ops_map_link_update(struct bpf_link *link, struct bpf_map *new_map) +{ + struct bpf_struct_ops_value *kvalue; + struct bpf_struct_ops_map *st_map, *old_st_map; + struct bpf_struct_ops_link *st_link; + struct bpf_map *old_map; + int err = 0; + + if (new_map->map_type != BPF_MAP_TYPE_STRUCT_OPS) + return -EINVAL; + + /* Ensure that the registration of the struct_ops matches the + * value of the pointer within the link. + */ + mutex_lock(&update_mutex); + + st_link = container_of(link, struct bpf_struct_ops_link, link); + + old_map = st_link->map; + if (!old_map) { + err = -EINVAL; + goto err_out; + } + + /* The new and old struct_ops must be the same type. */ + st_map = container_of(new_map, struct bpf_struct_ops_map, map); + + if (!(new_map->map_flags & BPF_F_LINK)) { + err = -EINVAL; + goto err_out; + } + + old_st_map = container_of(old_map, struct bpf_struct_ops_map, map); + if (st_map->st_ops != old_st_map->st_ops) { + err = -EINVAL; + goto err_out; + } + + err = bpf_struct_ops_transit_state_check(st_map, BPF_STRUCT_OPS_STATE_UNREG, + BPF_STRUCT_OPS_STATE_INUSE); + if (err) + goto err_out; + + kvalue = &st_map->kvalue; + + set_memory_rox((long)st_map->image, 1); + err = st_map->st_ops->update(kvalue->data, old_st_map->kvalue.data); + if (err) { + bpf_struct_ops_restore_unreg(st_map); + + set_memory_nx((long)st_map->image, 1); + set_memory_rw((long)st_map->image, 1); + goto err_out; + } + + bpf_map_inc(new_map); + rcu_assign_pointer(st_link->map, new_map); + + bpf_struct_ops_transit_state(old_st_map, BPF_STRUCT_OPS_STATE_INUSE, + BPF_STRUCT_OPS_STATE_TOBEUNREG); + bpf_struct_ops_put(&old_st_map->kvalue.data); + +err_out: + mutex_unlock(&update_mutex); + + return err; +} + static const struct bpf_link_ops bpf_struct_ops_map_lops = { .dealloc = bpf_struct_ops_map_link_dealloc, .detach = bpf_struct_ops_map_link_detach, .show_fdinfo = bpf_struct_ops_map_link_show_fdinfo, .fill_link_info = bpf_struct_ops_map_link_fill_link_info, + .update_struct_ops = bpf_struct_ops_map_link_update, }; int bpf_struct_ops_link_create(union bpf_attr *attr) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 2670de8dd0d4..423e6b7a6b41 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -4647,6 +4647,30 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) return ret; } +static int link_update_struct_ops(struct bpf_link *link, union bpf_attr *attr) +{ + struct bpf_map *new_map; + int ret = 0; + + new_map = bpf_map_get(attr->link_update.new_map_fd); + if (IS_ERR(new_map)) + return -EINVAL; + + if (new_map->map_type != BPF_MAP_TYPE_STRUCT_OPS) { + ret = -EINVAL; + goto out_put_map; + } + + if (link->ops->update_struct_ops) + ret = link->ops->update_struct_ops(link, new_map); + else + ret = -EINVAL; + +out_put_map: + bpf_map_put(new_map); + return ret; +} + #define BPF_LINK_UPDATE_LAST_FIELD link_update.old_prog_fd static int link_update(union bpf_attr *attr) @@ -4667,6 +4691,14 @@ static int link_update(union bpf_attr *attr) if (IS_ERR(link)) return PTR_ERR(link); + if (link->ops->update_struct_ops) { + if (flags) /* always replace the existing one */ + ret = -EINVAL; + else + ret = link_update_struct_ops(link, attr); + goto out_put_link; + } + new_prog = bpf_prog_get(attr->link_update.new_prog_fd); if (IS_ERR(new_prog)) { ret = PTR_ERR(new_prog); From patchwork Thu Feb 23 01:12:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13149755 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E914BC64EC7 for ; Thu, 23 Feb 2023 01:13:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233109AbjBWBNI (ORCPT ); Wed, 22 Feb 2023 20:13:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233088AbjBWBNE (ORCPT ); Wed, 22 Feb 2023 20:13:04 -0500 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2D7118B19 for ; Wed, 22 Feb 2023 17:13:03 -0800 (PST) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 31MMQq66029906 for ; Wed, 22 Feb 2023 17:13:03 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=FYS6F2jPmfg490T40yKUwjY2CM1x1BH1IHsHRF5A9ko=; b=kbTMT5UG8HZ3ZFjZ6ebkz9pJoJd+rcJH/c5ZyHMSHpOQy/WLBr3tTy1aQ/EJjL87V+27 J0p65xTNJ/mD0OhCeFrmylOCrEqu6zAnC+LAISrxzmu8lXcwxatPDfhVX5WUiliIyT3C 2Gdm8yUBvlzJTSVtcX+n0FzRKepOHz/e06vCQ9i1IgBeKoHJlxdDlRPFPDz/VoVpuTj7 PvL02Dn9uyvCS6E8QBO6NCf4V7b4TCHoJuIkVneY/4pi6euzlouUtrTOObgJxkdq/fEn JC3fbHcYZ0jDffbz8Fh7k630itl2UvqHgFuMGGwsiiPIGcZLbh5rCKc7QHbQy7a20BHv eQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3nw4rn0tke-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 22 Feb 2023 17:13:03 -0800 Received: from twshared16996.15.frc2.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 22 Feb 2023 17:12:59 -0800 Received: by devbig931.frc1.facebook.com (Postfix, from userid 460691) id 30FD15BD8D0B; Wed, 22 Feb 2023 17:12:42 -0800 (PST) From: Kui-Feng Lee To: , , , , , , CC: Kui-Feng Lee Subject: [PATCH bpf-next v2 5/6] libbpf: Update a bpf_link with another struct_ops. Date: Wed, 22 Feb 2023 17:12:37 -0800 Message-ID: <20230223011238.12313-6-kuifeng@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223011238.12313-1-kuifeng@meta.com> References: <20230223011238.12313-1-kuifeng@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: hsCxhCQZn84QPRoomJ8GQziMbEOLPkdW X-Proofpoint-ORIG-GUID: hsCxhCQZn84QPRoomJ8GQziMbEOLPkdW X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-22_12,2023-02-22_02,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Introduce bpf_link__update_struct_ops(), which will allow you to effortlessly transition the struct_ops map of any given bpf_link into an alternative. Signed-off-by: Kui-Feng Lee --- tools/lib/bpf/libbpf.c | 36 ++++++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf.h | 1 + tools/lib/bpf/libbpf.map | 2 ++ 3 files changed, 39 insertions(+) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 40f239cd1150..0cb3e5ed4d18 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -11524,6 +11524,42 @@ struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map) return &link->link; } +/* + * Swap the back struct_ops of a link with a new struct_ops map. + */ +int bpf_link__update_map(struct bpf_link *link, const struct bpf_map *map) +{ + struct bpf_link_struct_ops *st_ops_link; + __u32 zero = 0; + int err, fd; + + if (!bpf_map__is_struct_ops(map) || map->fd == -1) + return -EINVAL; + + /* Ensure the type of a link is correct */ + if (link->detach != bpf_link__detach_struct_ops) + return -EINVAL; + + err = bpf_map_update_elem(map->fd, &zero, map->st_ops->kern_vdata, 0); + if (err) { + err = -errno; + free(link); + return err; + } + + fd = bpf_link_update(link->fd, map->fd, NULL); + if (fd < 0) { + err = -errno; + free(link); + return err; + } + + st_ops_link = container_of(link, struct bpf_link_struct_ops, link); + st_ops_link->map_fd = map->fd; + + return 0; +} + typedef enum bpf_perf_event_ret (*bpf_perf_event_print_t)(struct perf_event_header *hdr, void *private_data); diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index 2efd80f6f7b9..5e62878d184c 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -695,6 +695,7 @@ bpf_program__attach_freplace(const struct bpf_program *prog, struct bpf_map; LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(const struct bpf_map *map); +LIBBPF_API int bpf_link__update_map(struct bpf_link *link, const struct bpf_map *map); struct bpf_iter_attach_opts { size_t sz; /* size of this struct for forward/backward compatibility */ diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 11c36a3c1a9f..e83571b04c19 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -384,4 +384,6 @@ LIBBPF_1.1.0 { } LIBBPF_1.0.0; LIBBPF_1.2.0 { + global: + bpf_link__update_map; } LIBBPF_1.1.0; From patchwork Thu Feb 23 01:12:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13149756 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46F08C61DA4 for ; Thu, 23 Feb 2023 01:15:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231388AbjBWBPw (ORCPT ); Wed, 22 Feb 2023 20:15:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229999AbjBWBPv (ORCPT ); Wed, 22 Feb 2023 20:15:51 -0500 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8330E769B for ; Wed, 22 Feb 2023 17:15:50 -0800 (PST) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 31MNCRl2006592 for ; Wed, 22 Feb 2023 17:15:49 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=rL1wm5tWNaKzwi393h9If7vkuXCooYbne/3VwBBzDpE=; b=jzYdS5g4fK+iL3SsminRW7ACxNh5bVKdh/nN6dSGf1n/39XK6zc3mjoto2LJ/n5V7E/4 kXZgM5JgstzTzyfOWIsIPSOq3R1PFS7vylqpAIgDFyUKuQ/VHkKsR2osdc5HtOb5KXLz p1UsXe0zAfp30RYBtwlrP74rb7sPBYA72CxV9s/euw6uMfdVyGkJlChkgxI7ImC7kxVJ 5K8kesVhUwgb45vEmmdjOmj7OWNQ7bLRPl2WqDPjlLJCInfqLEyFQtn3YbkUyRAFk54V KTTKEmVYEbeWKVX25eoJxrnFcHbeDO4EaCl/s4sTdjvu5PqGkTr9w8+D5sgDR9tbKYpN 5Q== Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3nwdy9xsj6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 22 Feb 2023 17:15:49 -0800 Received: from twshared18553.27.frc3.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 22 Feb 2023 17:15:48 -0800 Received: by devbig931.frc1.facebook.com (Postfix, from userid 460691) id 3F3DC5BD8D0D; Wed, 22 Feb 2023 17:12:42 -0800 (PST) From: Kui-Feng Lee To: , , , , , , CC: Kui-Feng Lee Subject: [PATCH bpf-next v2 6/6] selftests/bpf: Test switching TCP Congestion Control algorithms. Date: Wed, 22 Feb 2023 17:12:38 -0800 Message-ID: <20230223011238.12313-7-kuifeng@meta.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223011238.12313-1-kuifeng@meta.com> References: <20230223011238.12313-1-kuifeng@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: x2ZJOiyJSiapwMRumqXmHfX1bLbvoyr2 X-Proofpoint-ORIG-GUID: x2ZJOiyJSiapwMRumqXmHfX1bLbvoyr2 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22 definitions=2023-02-22_12,2023-02-22_02,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Create a pair of sockets that utilize the congestion control algorithm under a particular name. Then switch up this congestion control algorithm to another implementation and check whether newly created connections using the same cc name now run the new implementation. Signed-off-by: Kui-Feng Lee --- .../selftests/bpf/prog_tests/bpf_tcp_ca.c | 48 ++++++++++++++ .../selftests/bpf/progs/tcp_ca_update.c | 62 +++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/tcp_ca_update.c diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c b/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c index e980188d4124..a93956425bff 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c @@ -8,6 +8,7 @@ #include "bpf_dctcp.skel.h" #include "bpf_cubic.skel.h" #include "bpf_tcp_nogpl.skel.h" +#include "tcp_ca_update.skel.h" #include "bpf_dctcp_release.skel.h" #include "tcp_ca_write_sk_pacing.skel.h" #include "tcp_ca_incompl_cong_ops.skel.h" @@ -381,6 +382,51 @@ static void test_unsupp_cong_op(void) libbpf_set_print(old_print_fn); } +static void test_update_ca(void) +{ + struct tcp_ca_update *skel; + struct bpf_link *link; + int saved_ca1_cnt; + int err; + + skel = tcp_ca_update__open(); + if (!ASSERT_OK_PTR(skel, "open")) + return; + + err = bpf_map__set_map_flags(skel->maps.ca_update_1, + bpf_map__map_flags(skel->maps.ca_update_1) | BPF_F_LINK); + if (!ASSERT_OK(err, "set_map_flags_1")) + return; + + err = bpf_map__set_map_flags(skel->maps.ca_update_2, + bpf_map__map_flags(skel->maps.ca_update_2) | BPF_F_LINK); + if (!ASSERT_OK(err, "set_map_flags_2")) + return; + + err = tcp_ca_update__load(skel); + if (!ASSERT_OK(err, "load")) { + tcp_ca_update__destroy(skel); + return; + } + + link = bpf_map__attach_struct_ops(skel->maps.ca_update_1); + ASSERT_OK_PTR(link, "attach_struct_ops"); + + do_test("tcp_ca_update", NULL); + saved_ca1_cnt = skel->bss->ca1_cnt; + ASSERT_GT(saved_ca1_cnt, 0, "ca1_ca1_cnt"); + + err = bpf_link__update_map(link, skel->maps.ca_update_2); + ASSERT_OK(err, "update_struct_ops"); + + do_test("tcp_ca_update", NULL); + ASSERT_EQ(skel->bss->ca1_cnt, saved_ca1_cnt, "ca2_ca1_cnt"); + ASSERT_GT(skel->bss->ca2_cnt, 0, "ca2_ca2_cnt"); + + bpf_link__destroy(link); + tcp_ca_update__destroy(skel); +} + void test_bpf_tcp_ca(void) { if (test__start_subtest("dctcp")) @@ -399,4 +445,6 @@ void test_bpf_tcp_ca(void) test_incompl_cong_ops(); if (test__start_subtest("unsupp_cong_op")) test_unsupp_cong_op(); + if (test__start_subtest("update_ca")) + test_update_ca(); } diff --git a/tools/testing/selftests/bpf/progs/tcp_ca_update.c b/tools/testing/selftests/bpf/progs/tcp_ca_update.c new file mode 100644 index 000000000000..7aad4afe895b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/tcp_ca_update.c @@ -0,0 +1,62 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "vmlinux.h" + +#include +#include + +char _license[] SEC("license") = "GPL"; + +int ca1_cnt = 0; +int ca2_cnt = 0; + +#define USEC_PER_SEC 1000000UL + +#define min(a, b) ((a) < (b) ? (a) : (b)) + +static inline struct tcp_sock *tcp_sk(const struct sock *sk) +{ + return (struct tcp_sock *)sk; +} + +SEC("struct_ops/ca_update_1_cong_control") +void BPF_PROG(ca_update_1_cong_control, struct sock *sk, + const struct rate_sample *rs) +{ + ca1_cnt++; +} + +SEC("struct_ops/ca_update_2_cong_control") +void BPF_PROG(ca_update_2_cong_control, struct sock *sk, + const struct rate_sample *rs) +{ + ca2_cnt++; +} + +SEC("struct_ops/ca_update_ssthresh") +__u32 BPF_PROG(ca_update_ssthresh, struct sock *sk) +{ + return tcp_sk(sk)->snd_ssthresh; +} + +SEC("struct_ops/ca_update_undo_cwnd") +__u32 BPF_PROG(ca_update_undo_cwnd, struct sock *sk) +{ + return tcp_sk(sk)->snd_cwnd; +} + +SEC(".struct_ops") +struct tcp_congestion_ops ca_update_1 = { + .cong_control = (void *)ca_update_1_cong_control, + .ssthresh = (void *)ca_update_ssthresh, + .undo_cwnd = (void *)ca_update_undo_cwnd, + .name = "tcp_ca_update", +}; + +SEC(".struct_ops") +struct tcp_congestion_ops ca_update_2 = { + .cong_control = (void *)ca_update_2_cong_control, + .ssthresh = (void *)ca_update_ssthresh, + .undo_cwnd = (void *)ca_update_undo_cwnd, + .name = "tcp_ca_update", +};