| Message ID | 20230116225724.377099-1-toke@redhat.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | BPF |
| Headers | show |
| Series | [RFC,bpf-next] Documentation/bpf: Add a description of "stable kfuncs" | expand |
On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: > Following up on the discussion at the BPF office hours, this patch adds a > description of the (new) concept of "stable kfuncs", which are kfuncs that > offer a "more stable" interface than what we have now, but is still not > part of UAPI. > > This is mostly meant as a straw man proposal to focus discussions around > stability guarantees. From the discussion, it seemed clear that there were > at least some people (myself included) who felt that there needs to be some > way to export functionality that we consider "stable" (in the sense of > "applications can rely on its continuing existence"). > > One option is to keep BPF helpers as the stable interface and implement > some technical solution for moving functionality from kfuncs to helpers > once it has stood the test of time and we're comfortable committing to it > as a stable API. Another is to freeze the helper definitions, and instead > use kfuncs for this purpose as well, by marking a subset of them as > "stable" in some way. Or we can do both and have multiple levels of "stable", > I suppose. > > This patch is an attempt to describe what the "stable kfuncs" idea might look > like, as well as to formulate some criteria for what we mean by "stable", and > describe an explicit deprecation procedure. Feel free to critique any part > of this (including rejecting the notion entirely). > > Some people mentioned (in the office hours) that should we decide to go in > this direction, there's some work that needs to be done in libbpf (and > probably the kernel too?) to bring the kfunc developer experience up to par > with helpers. Things like exporting kfunc definitions to vmlinux.h (to make > them discoverable), and having CO-RE support for using them, etc. I kinda > consider that orthogonal to what's described here, but I added a > placeholder reference indicating that this (TBD) functionality exists. Thanks for the writeup.. I did some edits to your sections to make some parts more clear and to leave out other parts (e.g. libbpf-related bits which are not relevant in here and it's one of many libs). I also edited some parts to leave us more flexibility. Here would be my take mixed in: 3. API (in)stability of kfuncs ============================== By default, kfuncs exported to BPF programs are considered a kernel-internal interface that can change between kernel versions. In the extreme case that could also include removal of a kfunc. This means that BPF programs using kfuncs might need to adapt to changes between kernel versions. In other words, kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought of as being similar to internal kernel API functions exported using the ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must initially start out as kfuncs. 3.1 Promotion to "stable" ------------------------- While kfuncs are by default considered unstable as described above, some kfuncs may warrant a stronger stability guarantee and could be marked as *stable*. The decision to move a kfunc to *stable* is taken on a case-by-case basis and has a high barrier, taking into account its usefulness under longer-term production deployment without any unforeseen API issues or limitations. In general, it is not expected that every kfunc will turn into a stable one - think of it as an exception rather than the norm. kfuncs which have been promoted to stable are then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to a BPF helper addition is up to the maintainers to decide. 1. Stable kfuncs will not change their function signature or functionality in a way that may cause incompatibilities for BPF programs calling the function. 2. The BPF community will make every reasonable effort to keep stable kfuncs around as long as they continue to be useful to real-world BPF applications. 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure might be implemented for them as outlined below. 3.2 Deprecation of kfuncs ------------------------- As described above, the community will make every reasonable effort to keep kfuncs available through future kernel versions once they are marked as stable. However, there may be the unforeseen case that BPF development moves in a direction where even a stable kfunc is no longer useful for program development. In this case, stable kfuncs can be marked as *deprecated* using the ``KF_DEPRECATED`` tag. Such deprecation request cannot be arbitrary and must explain why a given stable kfunc should be deprecated. 1. A deprecated stable kfunc will be kept in the kernel for a conservatively chosen period of time after it got first marked as deprecated (usually corresponding to a span of multiple years). 2. Deprecated functions will be documented in the kernel docs along with their remaining lifespan and including a recommendation for new functionality that can replace the usage of the deprecated function (or an explanation for why no such replacement exists). 3. After the deprecation period, the kfunc will be removed and the function name will be marked as invalid inside the kernel (to ensure that no new kfunc is accidentally introduced with the same name in the future). After this happens, BPF programs calling the kfunc will be refused by the verifier. Thanks, Daniel
Daniel Borkmann <daniel@iogearbox.net> writes: > On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >> Following up on the discussion at the BPF office hours, this patch adds a >> description of the (new) concept of "stable kfuncs", which are kfuncs that >> offer a "more stable" interface than what we have now, but is still not >> part of UAPI. >> >> This is mostly meant as a straw man proposal to focus discussions around >> stability guarantees. From the discussion, it seemed clear that there were >> at least some people (myself included) who felt that there needs to be some >> way to export functionality that we consider "stable" (in the sense of >> "applications can rely on its continuing existence"). >> >> One option is to keep BPF helpers as the stable interface and implement >> some technical solution for moving functionality from kfuncs to helpers >> once it has stood the test of time and we're comfortable committing to it >> as a stable API. Another is to freeze the helper definitions, and instead >> use kfuncs for this purpose as well, by marking a subset of them as >> "stable" in some way. Or we can do both and have multiple levels of "stable", >> I suppose. >> >> This patch is an attempt to describe what the "stable kfuncs" idea might look >> like, as well as to formulate some criteria for what we mean by "stable", and >> describe an explicit deprecation procedure. Feel free to critique any part >> of this (including rejecting the notion entirely). >> >> Some people mentioned (in the office hours) that should we decide to go in >> this direction, there's some work that needs to be done in libbpf (and >> probably the kernel too?) to bring the kfunc developer experience up to par >> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >> them discoverable), and having CO-RE support for using them, etc. I kinda >> consider that orthogonal to what's described here, but I added a >> placeholder reference indicating that this (TBD) functionality exists. > > Thanks for the writeup.. I did some edits to your sections to make some parts > more clear and to leave out other parts (e.g. libbpf-related bits which are not > relevant in here and it's one of many libs). I also edited some parts to leave > us more flexibility. Here would be my take mixed in: Edits LGTM, with just one nit, below: > 3. API (in)stability of kfuncs > ============================== > > By default, kfuncs exported to BPF programs are considered a kernel-internal > interface that can change between kernel versions. In the extreme case that > could also include removal of a kfunc. This means that BPF programs using > kfuncs might need to adapt to changes between kernel versions. In other words, > kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought > of as being similar to internal kernel API functions exported using the > ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must > initially start out as kfuncs. > > 3.1 Promotion to "stable" > ------------------------- > > While kfuncs are by default considered unstable as described above, some kfuncs > may warrant a stronger stability guarantee and could be marked as *stable*. The > decision to move a kfunc to *stable* is taken on a case-by-case basis and has > a high barrier, taking into account its usefulness under longer-term production > deployment without any unforeseen API issues or limitations. In general, it is > not expected that every kfunc will turn into a stable one - think of it as an > exception rather than the norm. kfuncs which have been promoted to stable are > then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to > a BPF helper addition is up to the maintainers to decide. > > 1. Stable kfuncs will not change their function signature or functionality in > a way that may cause incompatibilities for BPF programs calling the function. > > 2. The BPF community will make every reasonable effort to keep stable kfuncs > around as long as they continue to be useful to real-world BPF applications. > > 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure > might be implemented for them as outlined below. "deprecation procedure might be implemented" could be interpreted as "we may implement a deprecation procedure, or we may just remove it without one". Which is presumably not what you meant? So maybe: 3. Should a stable kfunc turn out to be no longer useful, the BPF community may decide to eventually remove it. In this case, before being removed that kfunc will go through a deprecation procedure as outlined below. > 3.2 Deprecation of kfuncs > ------------------------- > > As described above, the community will make every reasonable effort to keep > kfuncs available through future kernel versions once they are marked as stable. > However, there may be the unforeseen case that BPF development moves in a > direction where even a stable kfunc is no longer useful for program development. > In this case, stable kfuncs can be marked as *deprecated* using the > ``KF_DEPRECATED`` tag. Such deprecation request cannot be arbitrary and must > explain why a given stable kfunc should be deprecated. > > 1. A deprecated stable kfunc will be kept in the kernel for a conservatively > chosen period of time after it got first marked as deprecated (usually > corresponding to a span of multiple years). > 2. Deprecated functions will be documented in the kernel docs along with their > remaining lifespan and including a recommendation for new functionality that > can replace the usage of the deprecated function (or an explanation for why > no such replacement exists). > > 3. After the deprecation period, the kfunc will be removed and the function name > will be marked as invalid inside the kernel (to ensure that no new kfunc is > accidentally introduced with the same name in the future). After this > happens, BPF programs calling the kfunc will be refused by the verifier.
On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote: > Daniel Borkmann <daniel@iogearbox.net> writes: > >> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >>> Following up on the discussion at the BPF office hours, this patch adds a >>> description of the (new) concept of "stable kfuncs", which are kfuncs that >>> offer a "more stable" interface than what we have now, but is still not >>> part of UAPI. >>> >>> This is mostly meant as a straw man proposal to focus discussions around >>> stability guarantees. From the discussion, it seemed clear that there were >>> at least some people (myself included) who felt that there needs to be some >>> way to export functionality that we consider "stable" (in the sense of >>> "applications can rely on its continuing existence"). >>> >>> One option is to keep BPF helpers as the stable interface and implement >>> some technical solution for moving functionality from kfuncs to helpers >>> once it has stood the test of time and we're comfortable committing to it >>> as a stable API. Another is to freeze the helper definitions, and instead >>> use kfuncs for this purpose as well, by marking a subset of them as >>> "stable" in some way. Or we can do both and have multiple levels of "stable", >>> I suppose. >>> >>> This patch is an attempt to describe what the "stable kfuncs" idea might look >>> like, as well as to formulate some criteria for what we mean by "stable", and >>> describe an explicit deprecation procedure. Feel free to critique any part >>> of this (including rejecting the notion entirely). >>> >>> Some people mentioned (in the office hours) that should we decide to go in >>> this direction, there's some work that needs to be done in libbpf (and >>> probably the kernel too?) to bring the kfunc developer experience up to par >>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >>> them discoverable), and having CO-RE support for using them, etc. I kinda >>> consider that orthogonal to what's described here, but I added a >>> placeholder reference indicating that this (TBD) functionality exists. >> >> Thanks for the writeup.. I did some edits to your sections to make some parts >> more clear and to leave out other parts (e.g. libbpf-related bits which are not >> relevant in here and it's one of many libs). I also edited some parts to leave >> us more flexibility. Here would be my take mixed in: > > Edits LGTM, with just one nit, below: > >> 3. API (in)stability of kfuncs >> ============================== >> >> By default, kfuncs exported to BPF programs are considered a kernel-internal >> interface that can change between kernel versions. In the extreme case that >> could also include removal of a kfunc. This means that BPF programs using >> kfuncs might need to adapt to changes between kernel versions. In other words, >> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought >> of as being similar to internal kernel API functions exported using the >> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must >> initially start out as kfuncs. >> >> 3.1 Promotion to "stable" >> ------------------------- >> >> While kfuncs are by default considered unstable as described above, some kfuncs >> may warrant a stronger stability guarantee and could be marked as *stable*. The >> decision to move a kfunc to *stable* is taken on a case-by-case basis and has >> a high barrier, taking into account its usefulness under longer-term production >> deployment without any unforeseen API issues or limitations. In general, it is Forgot, we should probably also add after "[...] or limitations.": Such promotion request along with aforementioned argumentation on why a kfunc is ready to be stabilized must be driven from developer-side. >> not expected that every kfunc will turn into a stable one - think of it as an >> exception rather than the norm. kfuncs which have been promoted to stable are >> then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to >> a BPF helper addition is up to the maintainers to decide. >> >> 1. Stable kfuncs will not change their function signature or functionality in >> a way that may cause incompatibilities for BPF programs calling the function. >> >> 2. The BPF community will make every reasonable effort to keep stable kfuncs >> around as long as they continue to be useful to real-world BPF applications. >> >> 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure >> might be implemented for them as outlined below. > > "deprecation procedure might be implemented" could be interpreted as "we > may implement a deprecation procedure, or we may just remove it without > one". Which is presumably not what you meant? So maybe: > > 3. Should a stable kfunc turn out to be no longer useful, the BPF > community may decide to eventually remove it. In this case, before > being removed that kfunc will go through a deprecation procedure as > outlined below. Yes, that sounds good to me. >> 3.2 Deprecation of kfuncs >> ------------------------- >> >> As described above, the community will make every reasonable effort to keep >> kfuncs available through future kernel versions once they are marked as stable. >> However, there may be the unforeseen case that BPF development moves in a >> direction where even a stable kfunc is no longer useful for program development. >> In this case, stable kfuncs can be marked as *deprecated* using the >> ``KF_DEPRECATED`` tag. Such deprecation request cannot be arbitrary and must >> explain why a given stable kfunc should be deprecated. >> >> 1. A deprecated stable kfunc will be kept in the kernel for a conservatively >> chosen period of time after it got first marked as deprecated (usually >> corresponding to a span of multiple years). >> 2. Deprecated functions will be documented in the kernel docs along with their >> remaining lifespan and including a recommendation for new functionality that >> can replace the usage of the deprecated function (or an explanation for why >> no such replacement exists). >> >> 3. After the deprecation period, the kfunc will be removed and the function name >> will be marked as invalid inside the kernel (to ensure that no new kfunc is >> accidentally introduced with the same name in the future). After this >> happens, BPF programs calling the kfunc will be refused by the verifier. >
Daniel Borkmann <daniel@iogearbox.net> writes: > On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote: >> Daniel Borkmann <daniel@iogearbox.net> writes: >> >>> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >>>> Following up on the discussion at the BPF office hours, this patch adds a >>>> description of the (new) concept of "stable kfuncs", which are kfuncs that >>>> offer a "more stable" interface than what we have now, but is still not >>>> part of UAPI. >>>> >>>> This is mostly meant as a straw man proposal to focus discussions around >>>> stability guarantees. From the discussion, it seemed clear that there were >>>> at least some people (myself included) who felt that there needs to be some >>>> way to export functionality that we consider "stable" (in the sense of >>>> "applications can rely on its continuing existence"). >>>> >>>> One option is to keep BPF helpers as the stable interface and implement >>>> some technical solution for moving functionality from kfuncs to helpers >>>> once it has stood the test of time and we're comfortable committing to it >>>> as a stable API. Another is to freeze the helper definitions, and instead >>>> use kfuncs for this purpose as well, by marking a subset of them as >>>> "stable" in some way. Or we can do both and have multiple levels of "stable", >>>> I suppose. >>>> >>>> This patch is an attempt to describe what the "stable kfuncs" idea might look >>>> like, as well as to formulate some criteria for what we mean by "stable", and >>>> describe an explicit deprecation procedure. Feel free to critique any part >>>> of this (including rejecting the notion entirely). >>>> >>>> Some people mentioned (in the office hours) that should we decide to go in >>>> this direction, there's some work that needs to be done in libbpf (and >>>> probably the kernel too?) to bring the kfunc developer experience up to par >>>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >>>> them discoverable), and having CO-RE support for using them, etc. I kinda >>>> consider that orthogonal to what's described here, but I added a >>>> placeholder reference indicating that this (TBD) functionality exists. >>> >>> Thanks for the writeup.. I did some edits to your sections to make some parts >>> more clear and to leave out other parts (e.g. libbpf-related bits which are not >>> relevant in here and it's one of many libs). I also edited some parts to leave >>> us more flexibility. Here would be my take mixed in: >> >> Edits LGTM, with just one nit, below: >> >>> 3. API (in)stability of kfuncs >>> ============================== >>> >>> By default, kfuncs exported to BPF programs are considered a kernel-internal >>> interface that can change between kernel versions. In the extreme case that >>> could also include removal of a kfunc. This means that BPF programs using >>> kfuncs might need to adapt to changes between kernel versions. In other words, >>> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought >>> of as being similar to internal kernel API functions exported using the >>> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must >>> initially start out as kfuncs. >>> >>> 3.1 Promotion to "stable" >>> ------------------------- >>> >>> While kfuncs are by default considered unstable as described above, some kfuncs >>> may warrant a stronger stability guarantee and could be marked as *stable*. The >>> decision to move a kfunc to *stable* is taken on a case-by-case basis and has >>> a high barrier, taking into account its usefulness under longer-term production >>> deployment without any unforeseen API issues or limitations. In general, it is > > Forgot, we should probably also add after "[...] or limitations.": > > Such promotion request along with aforementioned argumentation on why a kfunc > is ready to be stabilized must be driven from developer-side. What does "driven from developer-side" mean, exactly? And what kind of developers (BPF app developers, or kernel devs)? >>> not expected that every kfunc will turn into a stable one - think of it as an >>> exception rather than the norm. kfuncs which have been promoted to stable are >>> then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to >>> a BPF helper addition is up to the maintainers to decide. >>> >>> 1. Stable kfuncs will not change their function signature or functionality in >>> a way that may cause incompatibilities for BPF programs calling the function. >>> >>> 2. The BPF community will make every reasonable effort to keep stable kfuncs >>> around as long as they continue to be useful to real-world BPF applications. >>> >>> 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure >>> might be implemented for them as outlined below. >> >> "deprecation procedure might be implemented" could be interpreted as "we >> may implement a deprecation procedure, or we may just remove it without >> one". Which is presumably not what you meant? So maybe: >> >> 3. Should a stable kfunc turn out to be no longer useful, the BPF >> community may decide to eventually remove it. In this case, before >> being removed that kfunc will go through a deprecation procedure as >> outlined below. > > Yes, that sounds good to me. Awesome! -Toke
On 1/17/23 1:22 PM, Toke Høiland-Jørgensen wrote: > Daniel Borkmann <daniel@iogearbox.net> writes: >> On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote: >>> Daniel Borkmann <daniel@iogearbox.net> writes: >>>> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >>>>> Following up on the discussion at the BPF office hours, this patch adds a >>>>> description of the (new) concept of "stable kfuncs", which are kfuncs that >>>>> offer a "more stable" interface than what we have now, but is still not >>>>> part of UAPI. >>>>> >>>>> This is mostly meant as a straw man proposal to focus discussions around >>>>> stability guarantees. From the discussion, it seemed clear that there were >>>>> at least some people (myself included) who felt that there needs to be some >>>>> way to export functionality that we consider "stable" (in the sense of >>>>> "applications can rely on its continuing existence"). >>>>> >>>>> One option is to keep BPF helpers as the stable interface and implement >>>>> some technical solution for moving functionality from kfuncs to helpers >>>>> once it has stood the test of time and we're comfortable committing to it >>>>> as a stable API. Another is to freeze the helper definitions, and instead >>>>> use kfuncs for this purpose as well, by marking a subset of them as >>>>> "stable" in some way. Or we can do both and have multiple levels of "stable", >>>>> I suppose. >>>>> >>>>> This patch is an attempt to describe what the "stable kfuncs" idea might look >>>>> like, as well as to formulate some criteria for what we mean by "stable", and >>>>> describe an explicit deprecation procedure. Feel free to critique any part >>>>> of this (including rejecting the notion entirely). >>>>> >>>>> Some people mentioned (in the office hours) that should we decide to go in >>>>> this direction, there's some work that needs to be done in libbpf (and >>>>> probably the kernel too?) to bring the kfunc developer experience up to par >>>>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >>>>> them discoverable), and having CO-RE support for using them, etc. I kinda >>>>> consider that orthogonal to what's described here, but I added a >>>>> placeholder reference indicating that this (TBD) functionality exists. >>>> >>>> Thanks for the writeup.. I did some edits to your sections to make some parts >>>> more clear and to leave out other parts (e.g. libbpf-related bits which are not >>>> relevant in here and it's one of many libs). I also edited some parts to leave >>>> us more flexibility. Here would be my take mixed in: >>> >>> Edits LGTM, with just one nit, below: >>> >>>> 3. API (in)stability of kfuncs >>>> ============================== >>>> >>>> By default, kfuncs exported to BPF programs are considered a kernel-internal >>>> interface that can change between kernel versions. In the extreme case that >>>> could also include removal of a kfunc. This means that BPF programs using >>>> kfuncs might need to adapt to changes between kernel versions. In other words, >>>> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought >>>> of as being similar to internal kernel API functions exported using the >>>> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must >>>> initially start out as kfuncs. >>>> >>>> 3.1 Promotion to "stable" >>>> ------------------------- >>>> >>>> While kfuncs are by default considered unstable as described above, some kfuncs >>>> may warrant a stronger stability guarantee and could be marked as *stable*. The >>>> decision to move a kfunc to *stable* is taken on a case-by-case basis and has >>>> a high barrier, taking into account its usefulness under longer-term production >>>> deployment without any unforeseen API issues or limitations. In general, it is >> >> Forgot, we should probably also add after "[...] or limitations.": >> >> Such promotion request along with aforementioned argumentation on why a kfunc >> is ready to be stabilized must be driven from developer-side. > > What does "driven from developer-side" mean, exactly? And what kind of > developers (BPF app developers, or kernel devs)? Mainly to denote that this needs to be an explicit request from the community rather than something that would happen automagically after some time (e.g. where maintainers would just put the KF_STABLE stamp to it). 'kfunc xyz has been used in our fleet in production in the context of project abc for two years now and its API is sufficient to cover all foreseeable needs. The kfunc didn't need to get extended since it was added [...]', for example. The developer-hat can be both as long as there is a concrete relation to usage of the kfunc that can be provided to then make the case. >>>> not expected that every kfunc will turn into a stable one - think of it as an >>>> exception rather than the norm. kfuncs which have been promoted to stable are >>>> then marked using the ``KF_STABLE`` tag. The possibility from a stable kfunc to >>>> a BPF helper addition is up to the maintainers to decide. >>>> >>>> 1. Stable kfuncs will not change their function signature or functionality in >>>> a way that may cause incompatibilities for BPF programs calling the function. >>>> >>>> 2. The BPF community will make every reasonable effort to keep stable kfuncs >>>> around as long as they continue to be useful to real-world BPF applications. >>>> >>>> 3. Should a stable kfunc turn out to be no longer useful, a deprecation procedure >>>> might be implemented for them as outlined below. >>> >>> "deprecation procedure might be implemented" could be interpreted as "we >>> may implement a deprecation procedure, or we may just remove it without >>> one". Which is presumably not what you meant? So maybe: >>> >>> 3. Should a stable kfunc turn out to be no longer useful, the BPF >>> community may decide to eventually remove it. In this case, before >>> being removed that kfunc will go through a deprecation procedure as >>> outlined below. >> >> Yes, that sounds good to me. > > Awesome! > > -Toke >
Daniel Borkmann <daniel@iogearbox.net> writes: > On 1/17/23 1:22 PM, Toke Høiland-Jørgensen wrote: >> Daniel Borkmann <daniel@iogearbox.net> writes: >>> On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote: >>>> Daniel Borkmann <daniel@iogearbox.net> writes: >>>>> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >>>>>> Following up on the discussion at the BPF office hours, this patch adds a >>>>>> description of the (new) concept of "stable kfuncs", which are kfuncs that >>>>>> offer a "more stable" interface than what we have now, but is still not >>>>>> part of UAPI. >>>>>> >>>>>> This is mostly meant as a straw man proposal to focus discussions around >>>>>> stability guarantees. From the discussion, it seemed clear that there were >>>>>> at least some people (myself included) who felt that there needs to be some >>>>>> way to export functionality that we consider "stable" (in the sense of >>>>>> "applications can rely on its continuing existence"). >>>>>> >>>>>> One option is to keep BPF helpers as the stable interface and implement >>>>>> some technical solution for moving functionality from kfuncs to helpers >>>>>> once it has stood the test of time and we're comfortable committing to it >>>>>> as a stable API. Another is to freeze the helper definitions, and instead >>>>>> use kfuncs for this purpose as well, by marking a subset of them as >>>>>> "stable" in some way. Or we can do both and have multiple levels of "stable", >>>>>> I suppose. >>>>>> >>>>>> This patch is an attempt to describe what the "stable kfuncs" idea might look >>>>>> like, as well as to formulate some criteria for what we mean by "stable", and >>>>>> describe an explicit deprecation procedure. Feel free to critique any part >>>>>> of this (including rejecting the notion entirely). >>>>>> >>>>>> Some people mentioned (in the office hours) that should we decide to go in >>>>>> this direction, there's some work that needs to be done in libbpf (and >>>>>> probably the kernel too?) to bring the kfunc developer experience up to par >>>>>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >>>>>> them discoverable), and having CO-RE support for using them, etc. I kinda >>>>>> consider that orthogonal to what's described here, but I added a >>>>>> placeholder reference indicating that this (TBD) functionality exists. >>>>> >>>>> Thanks for the writeup.. I did some edits to your sections to make some parts >>>>> more clear and to leave out other parts (e.g. libbpf-related bits which are not >>>>> relevant in here and it's one of many libs). I also edited some parts to leave >>>>> us more flexibility. Here would be my take mixed in: >>>> >>>> Edits LGTM, with just one nit, below: >>>> >>>>> 3. API (in)stability of kfuncs >>>>> ============================== >>>>> >>>>> By default, kfuncs exported to BPF programs are considered a kernel-internal >>>>> interface that can change between kernel versions. In the extreme case that >>>>> could also include removal of a kfunc. This means that BPF programs using >>>>> kfuncs might need to adapt to changes between kernel versions. In other words, >>>>> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought >>>>> of as being similar to internal kernel API functions exported using the >>>>> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must >>>>> initially start out as kfuncs. >>>>> >>>>> 3.1 Promotion to "stable" >>>>> ------------------------- >>>>> >>>>> While kfuncs are by default considered unstable as described above, some kfuncs >>>>> may warrant a stronger stability guarantee and could be marked as *stable*. The >>>>> decision to move a kfunc to *stable* is taken on a case-by-case basis and has >>>>> a high barrier, taking into account its usefulness under longer-term production >>>>> deployment without any unforeseen API issues or limitations. In general, it is >>> >>> Forgot, we should probably also add after "[...] or limitations.": >>> >>> Such promotion request along with aforementioned argumentation on why a kfunc >>> is ready to be stabilized must be driven from developer-side. >> >> What does "driven from developer-side" mean, exactly? And what kind of >> developers (BPF app developers, or kernel devs)? > > Mainly to denote that this needs to be an explicit request from the community > rather than something that would happen automagically after some time (e.g. > where maintainers would just put the KF_STABLE stamp to it). 'kfunc xyz has > been used in our fleet in production in the context of project abc for two > years now and its API is sufficient to cover all foreseeable needs. The > kfunc didn't need to get extended since it was added [...]', for example. > The developer-hat can be both as long as there is a concrete relation to > usage of the kfunc that can be provided to then make the case. Right, makes sense! So how about: "The process for requesting a kfunc be marked as stable consists of submitting a patch to the bpf@vger.kernel.org mailing list adding the KF_STABLE tag to that kfunc's definition. The patch description must include the rationale for why the kfunc should be promoted to stable, including references to existing production uses, etc." -Toke
On 1/17/23 3:38 PM, Toke Høiland-Jørgensen wrote: > Daniel Borkmann <daniel@iogearbox.net> writes: >> On 1/17/23 1:22 PM, Toke Høiland-Jørgensen wrote: >>> Daniel Borkmann <daniel@iogearbox.net> writes: >>>> On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote: >>>>> Daniel Borkmann <daniel@iogearbox.net> writes: >>>>>> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >>>>>>> Following up on the discussion at the BPF office hours, this patch adds a >>>>>>> description of the (new) concept of "stable kfuncs", which are kfuncs that >>>>>>> offer a "more stable" interface than what we have now, but is still not >>>>>>> part of UAPI. >>>>>>> >>>>>>> This is mostly meant as a straw man proposal to focus discussions around >>>>>>> stability guarantees. From the discussion, it seemed clear that there were >>>>>>> at least some people (myself included) who felt that there needs to be some >>>>>>> way to export functionality that we consider "stable" (in the sense of >>>>>>> "applications can rely on its continuing existence"). >>>>>>> >>>>>>> One option is to keep BPF helpers as the stable interface and implement >>>>>>> some technical solution for moving functionality from kfuncs to helpers >>>>>>> once it has stood the test of time and we're comfortable committing to it >>>>>>> as a stable API. Another is to freeze the helper definitions, and instead >>>>>>> use kfuncs for this purpose as well, by marking a subset of them as >>>>>>> "stable" in some way. Or we can do both and have multiple levels of "stable", >>>>>>> I suppose. >>>>>>> >>>>>>> This patch is an attempt to describe what the "stable kfuncs" idea might look >>>>>>> like, as well as to formulate some criteria for what we mean by "stable", and >>>>>>> describe an explicit deprecation procedure. Feel free to critique any part >>>>>>> of this (including rejecting the notion entirely). >>>>>>> >>>>>>> Some people mentioned (in the office hours) that should we decide to go in >>>>>>> this direction, there's some work that needs to be done in libbpf (and >>>>>>> probably the kernel too?) to bring the kfunc developer experience up to par >>>>>>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >>>>>>> them discoverable), and having CO-RE support for using them, etc. I kinda >>>>>>> consider that orthogonal to what's described here, but I added a >>>>>>> placeholder reference indicating that this (TBD) functionality exists. >>>>>> >>>>>> Thanks for the writeup.. I did some edits to your sections to make some parts >>>>>> more clear and to leave out other parts (e.g. libbpf-related bits which are not >>>>>> relevant in here and it's one of many libs). I also edited some parts to leave >>>>>> us more flexibility. Here would be my take mixed in: >>>>> >>>>> Edits LGTM, with just one nit, below: >>>>> >>>>>> 3. API (in)stability of kfuncs >>>>>> ============================== >>>>>> >>>>>> By default, kfuncs exported to BPF programs are considered a kernel-internal >>>>>> interface that can change between kernel versions. In the extreme case that >>>>>> could also include removal of a kfunc. This means that BPF programs using >>>>>> kfuncs might need to adapt to changes between kernel versions. In other words, >>>>>> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought >>>>>> of as being similar to internal kernel API functions exported using the >>>>>> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must >>>>>> initially start out as kfuncs. >>>>>> >>>>>> 3.1 Promotion to "stable" >>>>>> ------------------------- >>>>>> >>>>>> While kfuncs are by default considered unstable as described above, some kfuncs >>>>>> may warrant a stronger stability guarantee and could be marked as *stable*. The >>>>>> decision to move a kfunc to *stable* is taken on a case-by-case basis and has >>>>>> a high barrier, taking into account its usefulness under longer-term production >>>>>> deployment without any unforeseen API issues or limitations. In general, it is >>>> >>>> Forgot, we should probably also add after "[...] or limitations.": >>>> >>>> Such promotion request along with aforementioned argumentation on why a kfunc >>>> is ready to be stabilized must be driven from developer-side. >>> >>> What does "driven from developer-side" mean, exactly? And what kind of >>> developers (BPF app developers, or kernel devs)? >> >> Mainly to denote that this needs to be an explicit request from the community >> rather than something that would happen automagically after some time (e.g. >> where maintainers would just put the KF_STABLE stamp to it). 'kfunc xyz has >> been used in our fleet in production in the context of project abc for two >> years now and its API is sufficient to cover all foreseeable needs. The >> kfunc didn't need to get extended since it was added [...]', for example. >> The developer-hat can be both as long as there is a concrete relation to >> usage of the kfunc that can be provided to then make the case. > > Right, makes sense! So how about: > > "The process for requesting a kfunc be marked as stable consists of > submitting a patch to the bpf@vger.kernel.org mailing list adding the > KF_STABLE tag to that kfunc's definition. The patch description must > include the rationale for why the kfunc should be promoted to stable, > including references to existing production uses, etc." Sounds good to me! Thanks, Daniel
Daniel Borkmann <daniel@iogearbox.net> writes: > On 1/17/23 3:38 PM, Toke Høiland-Jørgensen wrote: >> Daniel Borkmann <daniel@iogearbox.net> writes: >>> On 1/17/23 1:22 PM, Toke Høiland-Jørgensen wrote: >>>> Daniel Borkmann <daniel@iogearbox.net> writes: >>>>> On 1/17/23 12:30 PM, Toke Høiland-Jørgensen wrote: >>>>>> Daniel Borkmann <daniel@iogearbox.net> writes: >>>>>>> On 1/16/23 11:57 PM, Toke Høiland-Jørgensen wrote: >>>>>>>> Following up on the discussion at the BPF office hours, this patch adds a >>>>>>>> description of the (new) concept of "stable kfuncs", which are kfuncs that >>>>>>>> offer a "more stable" interface than what we have now, but is still not >>>>>>>> part of UAPI. >>>>>>>> >>>>>>>> This is mostly meant as a straw man proposal to focus discussions around >>>>>>>> stability guarantees. From the discussion, it seemed clear that there were >>>>>>>> at least some people (myself included) who felt that there needs to be some >>>>>>>> way to export functionality that we consider "stable" (in the sense of >>>>>>>> "applications can rely on its continuing existence"). >>>>>>>> >>>>>>>> One option is to keep BPF helpers as the stable interface and implement >>>>>>>> some technical solution for moving functionality from kfuncs to helpers >>>>>>>> once it has stood the test of time and we're comfortable committing to it >>>>>>>> as a stable API. Another is to freeze the helper definitions, and instead >>>>>>>> use kfuncs for this purpose as well, by marking a subset of them as >>>>>>>> "stable" in some way. Or we can do both and have multiple levels of "stable", >>>>>>>> I suppose. >>>>>>>> >>>>>>>> This patch is an attempt to describe what the "stable kfuncs" idea might look >>>>>>>> like, as well as to formulate some criteria for what we mean by "stable", and >>>>>>>> describe an explicit deprecation procedure. Feel free to critique any part >>>>>>>> of this (including rejecting the notion entirely). >>>>>>>> >>>>>>>> Some people mentioned (in the office hours) that should we decide to go in >>>>>>>> this direction, there's some work that needs to be done in libbpf (and >>>>>>>> probably the kernel too?) to bring the kfunc developer experience up to par >>>>>>>> with helpers. Things like exporting kfunc definitions to vmlinux.h (to make >>>>>>>> them discoverable), and having CO-RE support for using them, etc. I kinda >>>>>>>> consider that orthogonal to what's described here, but I added a >>>>>>>> placeholder reference indicating that this (TBD) functionality exists. >>>>>>> >>>>>>> Thanks for the writeup.. I did some edits to your sections to make some parts >>>>>>> more clear and to leave out other parts (e.g. libbpf-related bits which are not >>>>>>> relevant in here and it's one of many libs). I also edited some parts to leave >>>>>>> us more flexibility. Here would be my take mixed in: >>>>>> >>>>>> Edits LGTM, with just one nit, below: >>>>>> >>>>>>> 3. API (in)stability of kfuncs >>>>>>> ============================== >>>>>>> >>>>>>> By default, kfuncs exported to BPF programs are considered a kernel-internal >>>>>>> interface that can change between kernel versions. In the extreme case that >>>>>>> could also include removal of a kfunc. This means that BPF programs using >>>>>>> kfuncs might need to adapt to changes between kernel versions. In other words, >>>>>>> kfuncs are _not_ part of the kernel UAPI! Rather, these kfuncs can be thought >>>>>>> of as being similar to internal kernel API functions exported using the >>>>>>> ``EXPORT_SYMBOL_GPL`` macro. All new BPF kernel helper-like functionality must >>>>>>> initially start out as kfuncs. >>>>>>> >>>>>>> 3.1 Promotion to "stable" >>>>>>> ------------------------- >>>>>>> >>>>>>> While kfuncs are by default considered unstable as described above, some kfuncs >>>>>>> may warrant a stronger stability guarantee and could be marked as *stable*. The >>>>>>> decision to move a kfunc to *stable* is taken on a case-by-case basis and has >>>>>>> a high barrier, taking into account its usefulness under longer-term production >>>>>>> deployment without any unforeseen API issues or limitations. In general, it is >>>>> >>>>> Forgot, we should probably also add after "[...] or limitations.": >>>>> >>>>> Such promotion request along with aforementioned argumentation on why a kfunc >>>>> is ready to be stabilized must be driven from developer-side. >>>> >>>> What does "driven from developer-side" mean, exactly? And what kind of >>>> developers (BPF app developers, or kernel devs)? >>> >>> Mainly to denote that this needs to be an explicit request from the community >>> rather than something that would happen automagically after some time (e.g. >>> where maintainers would just put the KF_STABLE stamp to it). 'kfunc xyz has >>> been used in our fleet in production in the context of project abc for two >>> years now and its API is sufficient to cover all foreseeable needs. The >>> kfunc didn't need to get extended since it was added [...]', for example. >>> The developer-hat can be both as long as there is a concrete relation to >>> usage of the kfunc that can be provided to then make the case. >> >> Right, makes sense! So how about: >> >> "The process for requesting a kfunc be marked as stable consists of >> submitting a patch to the bpf@vger.kernel.org mailing list adding the >> KF_STABLE tag to that kfunc's definition. The patch description must >> include the rationale for why the kfunc should be promoted to stable, >> including references to existing production uses, etc." > > Sounds good to me! Cool. I'll incorporate your changes (+ what we discussed) and send a v2 to make it easier for others to chime in... -Toke
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 9fd7fb539f85..c40726c5d3bb 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -7,9 +7,9 @@ BPF Kernel Functions (kfuncs) BPF Kernel Functions or more commonly known as kfuncs are functions in the Linux kernel which are exposed for use by BPF programs. Unlike normal BPF helpers, -kfuncs do not have a stable interface and can change from one kernel release to -another. Hence, BPF programs need to be updated in response to changes in the -kernel. +kfuncs by default do not have a stable interface and can change from one kernel +release to another. Hence, BPF programs may need to be updated in response to +changes in the kernel. See :ref:`BPF_kfunc_stability`. 2. Defining a kfunc =================== @@ -223,14 +223,81 @@ type. An example is shown below:: } late_initcall(init_subsystem); -3. Core kfuncs + +.. _BPF_kfunc_stability: + +3. API stability of kfuncs +========================== + +By default, kfuncs exported to BPF programs are considered a kernel-internal +interface that can change between kernel versions. This means that BPF programs +using kfuncs need to adapt to changes between kernel versions; these kfuncs can +be thought of as being similar to internal kernel API functions exported using +the ``EXPORT_SYMBOL_GPL`` macro. + +The libbpf library contains functionality that can help applications discover +which kfuncs are available and the CO-RE functionality can be used to handle +differences in kfunc availability across kernel versions as described in (TBD, +once this is implemented). + +3.1 Stable kfuncs +----------------- + +While kfuncs are by default considered unstable as described above, some kfuncs +warrant a stronger stability guarantee and are marked as *stable*. The decision +to move a kfunc to *stable* is taken on a case-by-case basis based on demand for +a stable interface, and only once a function has proven to be useful in practice +without any unforeseen API issues. + +Stable kfuncs are marked with the ``KF_STABLE`` tag in their definition, and +provide the following stability guarantees: + +1. Stable kfuncs will not change their function signature or functionality in a + way that may cause incompatibilities for BPF programs calling the function. + +2. The BPF community will make every reasonable effort to keep stable kfuncs + around as long as they continue to be useful to real-world BPF applications. + +3. Should a stable kfunc turn out to be no longer useful, or otherwise become + enough of a maintenance burden that it has to be removed, removal will only + happen following the deprecation procedure outlined below. + +3.2 Deprecation of kfuncs +------------------------- + +As described above, the community will make every reasonable effort to keep +kfuncs available through future kernel versions once they are marked as stable. +However, it may be the case that BPF development moves in a direction where even +a stable kfunc is no longer useful and/or becomes an unreasonable maintenance +burden for further development. + +In this case, stable kfuncs can be marked as *deprecated* using the +``KF_DEPRECATED`` tag. This will have the following effect: + +1. When using a deprecated kfunc, libbpf will emit a warning that the function + will be removed in a future kernel version. + +2. Deprecated kfuncs will be kept in the kernel for a minimum of 10 kernel + releases after it is first marked as deprecated (corresponding to roughly two + years of development time). + +3. Deprecated functions will be documented in the kernel docs, including a + recommendation for new functionality that can replace the usage of the + deprecated function (or an explanation for why no such replacement exists). + +4. After the deprecation period, the kfunc will be removed and the function name + will be marked as invalid inside the kernel (to ensure that no new kfunc is + accidentally introduced with the same name in the future). After this + happens, BPF programs calling the kfunc will be refused by the verifier. + +4. Core kfuncs ============== The BPF subsystem provides a number of "core" kfuncs that are potentially applicable to a wide variety of different possible use cases and programs. Those kfuncs are documented here. -3.1 struct task_struct * kfuncs +4.1 struct task_struct * kfuncs ------------------------------- There are a number of kfuncs that allow ``struct task_struct *`` objects to be @@ -306,7 +373,7 @@ Here is an example of it being used: return 0; } -3.2 struct cgroup * kfuncs +4.2 struct cgroup * kfuncs -------------------------- ``struct cgroup *`` objects also have acquire and release functions:
Following up on the discussion at the BPF office hours, this patch adds a description of the (new) concept of "stable kfuncs", which are kfuncs that offer a "more stable" interface than what we have now, but is still not part of UAPI. This is mostly meant as a straw man proposal to focus discussions around stability guarantees. From the discussion, it seemed clear that there were at least some people (myself included) who felt that there needs to be some way to export functionality that we consider "stable" (in the sense of "applications can rely on its continuing existence"). One option is to keep BPF helpers as the stable interface and implement some technical solution for moving functionality from kfuncs to helpers once it has stood the test of time and we're comfortable committing to it as a stable API. Another is to freeze the helper definitions, and instead use kfuncs for this purpose as well, by marking a subset of them as "stable" in some way. Or we can do both and have multiple levels of "stable", I suppose. This patch is an attempt to describe what the "stable kfuncs" idea might look like, as well as to formulate some criteria for what we mean by "stable", and describe an explicit deprecation procedure. Feel free to critique any part of this (including rejecting the notion entirely). Some people mentioned (in the office hours) that should we decide to go in this direction, there's some work that needs to be done in libbpf (and probably the kernel too?) to bring the kfunc developer experience up to par with helpers. Things like exporting kfunc definitions to vmlinux.h (to make them discoverable), and having CO-RE support for using them, etc. I kinda consider that orthogonal to what's described here, but I added a placeholder reference indicating that this (TBD) functionality exists. Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> --- Documentation/bpf/kfuncs.rst | 79 +++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 6 deletions(-)