| Message ID | cover.1692748902.git.dxu@dxuuu.xyz (mailing list archive) |
|---|---|
| Headers | show |
| Series | Improve prog array uref semantics | expand |
Hey Daniel, On Wed, Aug 23, 2023, at 9:08 AM, Daniel Xu wrote: > This patchset changes the behavior of TC and XDP hooks during attachment > such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra > uref taken. > > The goal behind this change is to try and prevent confusion for the > majority of use cases. The current behavior where when the last uref is > dropped the prog array map is emptied is quite confusing. Confusing > enough for there to be multiple references to it in ebpf-go [0][1]. > > Completely solving the problem is difficult. As stated in c9da161c6517 > ("bpf: fix clearing on persistent program array maps"), it is > difficult-to-impossible to walk the full dependency graph b/c it is too > dynamic. > > However in practice, I've found that all progs in a tailcall chain > share the same prog array map. Knowing that, if we take a uref on any > used prog array map when the program is attached, we can simplify the > majority use case and make it more ergonomic. > > I'll be the first to admit this is not a very clean solution. It does > not fully solve the problem. Nor does it make overall logic any simpler. > But I do think it makes a pretty big usability hole slightly smaller. > > I've done some basic testing using a repro program [3] I wrote to debug > the original issue that eventually led me to this patchset. If we wanna > move forward with this approach, I'll resend with selftests. > > [0]: > https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24 > [1]: > https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321 > [2]: https://github.com/danobi/tc_tailcall_repro I recently remembered about this again. Was suggested I poke you in case you're interested. I looked again and I think this is kinda a neat hack. I probably won't have time to pick this back up either way.
On Tue, Oct 29, 2024 at 11:36 PM Daniel Xu <dxu@dxuuu.xyz> wrote: > > Hey Daniel, > > On Wed, Aug 23, 2023, at 9:08 AM, Daniel Xu wrote: > > This patchset changes the behavior of TC and XDP hooks during attachment > > such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra > > uref taken. > > > > The goal behind this change is to try and prevent confusion for the > > majority of use cases. The current behavior where when the last uref is > > dropped the prog array map is emptied is quite confusing. Confusing > > enough for there to be multiple references to it in ebpf-go [0][1]. > > > > Completely solving the problem is difficult. As stated in c9da161c6517 > > ("bpf: fix clearing on persistent program array maps"), it is > > difficult-to-impossible to walk the full dependency graph b/c it is too > > dynamic. > > > > However in practice, I've found that all progs in a tailcall chain > > share the same prog array map. Knowing that, if we take a uref on any > > used prog array map when the program is attached, we can simplify the > > majority use case and make it more ergonomic. Are you proposing to inc map uref when prog is attached? But that re-adds the circular dependency that uref concept is solving. When prog is inserted into prog array prog refcnt is incremented. So if prog also incremented uref. The user space can exit but prog array and progs will stay there though nothing is using them. I guess I'm missing the idea. > > > > I'll be the first to admit this is not a very clean solution. It does > > not fully solve the problem. Nor does it make overall logic any simpler. > > But I do think it makes a pretty big usability hole slightly smaller. > > > > I've done some basic testing using a repro program [3] I wrote to debug > > the original issue that eventually led me to this patchset. If we wanna > > move forward with this approach, I'll resend with selftests. > > > > [0]: > > https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24 > > [1]: > > https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321 > > [2]: https://github.com/danobi/tc_tailcall_repro > > I recently remembered about this again. Was suggested I poke you in case you're interested. > I looked again and I think this is kinda a neat hack. I probably won't have time to pick this back > up either way. >
On Sat, Nov 16, 2024, at 2:17 PM, Alexei Starovoitov wrote: > On Tue, Oct 29, 2024 at 11:36 PM Daniel Xu <dxu@dxuuu.xyz> wrote: >> >> Hey Daniel, >> >> On Wed, Aug 23, 2023, at 9:08 AM, Daniel Xu wrote: >> > This patchset changes the behavior of TC and XDP hooks during attachment >> > such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra >> > uref taken. >> > >> > The goal behind this change is to try and prevent confusion for the >> > majority of use cases. The current behavior where when the last uref is >> > dropped the prog array map is emptied is quite confusing. Confusing >> > enough for there to be multiple references to it in ebpf-go [0][1]. >> > >> > Completely solving the problem is difficult. As stated in c9da161c6517 >> > ("bpf: fix clearing on persistent program array maps"), it is >> > difficult-to-impossible to walk the full dependency graph b/c it is too >> > dynamic. >> > >> > However in practice, I've found that all progs in a tailcall chain >> > share the same prog array map. Knowing that, if we take a uref on any >> > used prog array map when the program is attached, we can simplify the >> > majority use case and make it more ergonomic. > > Are you proposing to inc map uref when prog is attached? > > But that re-adds the circular dependency that uref concept is solving. > When prog is inserted into prog array prog refcnt is incremented. > So if prog also incremented uref. The user space can exit > but prog array and progs will stay there though nothing is using them. > I guess I'm missing the idea. IIRC the old-style tc/xdp attachment is the one incrementing the uref. Once whatever program there is detached the uref is dropped. So I don't think any circular refs can happen unless a prog can somehow prevent its own detachment. Could be mis-remembering though. Thanks, Daniel
On Wed, Nov 20, 2024 at 7:55 AM Daniel Xu <dxu@dxuuu.xyz> wrote: > > > > On Sat, Nov 16, 2024, at 2:17 PM, Alexei Starovoitov wrote: > > On Tue, Oct 29, 2024 at 11:36 PM Daniel Xu <dxu@dxuuu.xyz> wrote: > >> > >> Hey Daniel, > >> > >> On Wed, Aug 23, 2023, at 9:08 AM, Daniel Xu wrote: > >> > This patchset changes the behavior of TC and XDP hooks during attachment > >> > such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra > >> > uref taken. > >> > > >> > The goal behind this change is to try and prevent confusion for the > >> > majority of use cases. The current behavior where when the last uref is > >> > dropped the prog array map is emptied is quite confusing. Confusing > >> > enough for there to be multiple references to it in ebpf-go [0][1]. > >> > > >> > Completely solving the problem is difficult. As stated in c9da161c6517 > >> > ("bpf: fix clearing on persistent program array maps"), it is > >> > difficult-to-impossible to walk the full dependency graph b/c it is too > >> > dynamic. > >> > > >> > However in practice, I've found that all progs in a tailcall chain > >> > share the same prog array map. Knowing that, if we take a uref on any > >> > used prog array map when the program is attached, we can simplify the > >> > majority use case and make it more ergonomic. > > > > Are you proposing to inc map uref when prog is attached? > > > > But that re-adds the circular dependency that uref concept is solving. > > When prog is inserted into prog array prog refcnt is incremented. > > So if prog also incremented uref. The user space can exit > > but prog array and progs will stay there though nothing is using them. > > I guess I'm missing the idea. > > IIRC the old-style tc/xdp attachment is the one incrementing the uref. uref is incremented when FD is given to user space and file->release() callback decrements uref. I don't think any of the attach operations mess with uref. At least they shouldn't. > Once > whatever program there is detached the uref is dropped. So I don't think > any circular refs can happen unless a prog can somehow prevent its own > detachment. > > Could be mis-remembering though. > > Thanks, > Daniel
On Wed, Nov 20, 2024, at 8:07 AM, Alexei Starovoitov wrote: > On Wed, Nov 20, 2024 at 7:55 AM Daniel Xu <dxu@dxuuu.xyz> wrote: >> >> >> >> On Sat, Nov 16, 2024, at 2:17 PM, Alexei Starovoitov wrote: >> > On Tue, Oct 29, 2024 at 11:36 PM Daniel Xu <dxu@dxuuu.xyz> wrote: >> >> >> >> Hey Daniel, >> >> >> >> On Wed, Aug 23, 2023, at 9:08 AM, Daniel Xu wrote: >> >> > This patchset changes the behavior of TC and XDP hooks during attachment >> >> > such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra >> >> > uref taken. >> >> > >> >> > The goal behind this change is to try and prevent confusion for the >> >> > majority of use cases. The current behavior where when the last uref is >> >> > dropped the prog array map is emptied is quite confusing. Confusing >> >> > enough for there to be multiple references to it in ebpf-go [0][1]. >> >> > >> >> > Completely solving the problem is difficult. As stated in c9da161c6517 >> >> > ("bpf: fix clearing on persistent program array maps"), it is >> >> > difficult-to-impossible to walk the full dependency graph b/c it is too >> >> > dynamic. >> >> > >> >> > However in practice, I've found that all progs in a tailcall chain >> >> > share the same prog array map. Knowing that, if we take a uref on any >> >> > used prog array map when the program is attached, we can simplify the >> >> > majority use case and make it more ergonomic. >> > >> > Are you proposing to inc map uref when prog is attached? >> > >> > But that re-adds the circular dependency that uref concept is solving. >> > When prog is inserted into prog array prog refcnt is incremented. >> > So if prog also incremented uref. The user space can exit >> > but prog array and progs will stay there though nothing is using them. >> > I guess I'm missing the idea. >> >> IIRC the old-style tc/xdp attachment is the one incrementing the uref. > > uref is incremented when FD is given to user space and > file->release() callback decrements uref. > > I don't think any of the attach operations mess with uref. > At least they shouldn't. None yet. My patch was adding it. It's fine if it's too much of a hack - was just an idea.
This patchset changes the behavior of TC and XDP hooks during attachment such that any BPF_MAP_TYPE_PROG_ARRAY that the prog uses has an extra uref taken. The goal behind this change is to try and prevent confusion for the majority of use cases. The current behavior where when the last uref is dropped the prog array map is emptied is quite confusing. Confusing enough for there to be multiple references to it in ebpf-go [0][1]. Completely solving the problem is difficult. As stated in c9da161c6517 ("bpf: fix clearing on persistent program array maps"), it is difficult-to-impossible to walk the full dependency graph b/c it is too dynamic. However in practice, I've found that all progs in a tailcall chain share the same prog array map. Knowing that, if we take a uref on any used prog array map when the program is attached, we can simplify the majority use case and make it more ergonomic. I'll be the first to admit this is not a very clean solution. It does not fully solve the problem. Nor does it make overall logic any simpler. But I do think it makes a pretty big usability hole slightly smaller. I've done some basic testing using a repro program [3] I wrote to debug the original issue that eventually led me to this patchset. If we wanna move forward with this approach, I'll resend with selftests. [0]: https://github.com/cilium/ebpf/blob/01ebd4c1e2b9f8b3dd4fd2382aa1092c3c9bfc9d/doc.go#L22-L24 [1]: https://github.com/cilium/ebpf/blob/d1a52333f2c0fed085f8d742a5a3c164795d8492/collection.go#L320-L321 [2]: https://github.com/danobi/tc_tailcall_repro Daniel Xu (2): net: bpf: Make xdp and cls_bpf use bpf_prog_put_dev() bpf: Take a uref on BPF_MAP_TYPE_PROG_ARRAY maps during dev attachment include/linux/bpf.h | 1 + kernel/bpf/devmap.c | 8 ++++---- kernel/bpf/syscall.c | 46 +++++++++++++++++++++++++++++++++++++++++++- net/core/dev.c | 16 +++++++-------- net/sched/cls_bpf.c | 4 ++-- 5 files changed, 60 insertions(+), 15 deletions(-)