| Message ID | 20240403123303.1452184-1-aspsk@isovalent.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | f91717007217d975aa975ddabd91ae1a107b9bff |
| Delegated to: | BPF |
| Headers | show |
| Series | [bpf-next] bpf: pack struct bpf_fib_lookup | expand |
From: Anton Protopopov <aspsk@isovalent.com> Date: Wed, 3 Apr 2024 14:33:03 +0200 > The struct bpf_fib_lookup is supposed to be of size 64. A recent commit > 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added > a static assertion to check this property so that future changes to the > structure will not accidentally break this assumption. > > As it immediately turned out, on some 32-bit arm systems, when AEABI=n, > the total size of the structure was equal to 68, see [1]. This happened > because the bpf_fib_lookup structure contains a union of two 16-bit > fields: > > union { > __u16 tot_len; > __u16 mtu_result; > }; > > which was supposed to compile to a 16-bit-aligned 16-bit field. On the > aforementioned setups it was instead both aligned and padded to 32-bits. > > Declare this inner union as __attribute__((packed, aligned(2))) such > that it always is of size 2 and is aligned to 16 bits. > > [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t > > Signed-off-by: Anton Protopopov <aspsk@isovalent.com> Reviewed-by: Alexander Lobakin <aleksander.lobakin@intel.com> > --- > include/uapi/linux/bpf.h | 2 +- > tools/include/uapi/linux/bpf.h | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h > index 79c548276b6b..6fe9f11c8abe 100644 > --- a/include/uapi/linux/bpf.h > +++ b/include/uapi/linux/bpf.h > @@ -7157,7 +7157,7 @@ struct bpf_fib_lookup { > > /* output: MTU value */ > __u16 mtu_result; > - }; > + } __attribute__((packed, aligned(2))); > /* input: L3 device index for lookup > * output: device index from FIB lookup > */ > diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h > index 79c548276b6b..6fe9f11c8abe 100644 > --- a/tools/include/uapi/linux/bpf.h > +++ b/tools/include/uapi/linux/bpf.h > @@ -7157,7 +7157,7 @@ struct bpf_fib_lookup { > > /* output: MTU value */ > __u16 mtu_result; > - }; > + } __attribute__((packed, aligned(2))); > /* input: L3 device index for lookup > * output: device index from FIB lookup > */ Thanks, Olek
On 4/3/24 2:33 PM, Anton Protopopov wrote: > The struct bpf_fib_lookup is supposed to be of size 64. A recent commit > 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added > a static assertion to check this property so that future changes to the > structure will not accidentally break this assumption. > > As it immediately turned out, on some 32-bit arm systems, when AEABI=n, > the total size of the structure was equal to 68, see [1]. This happened > because the bpf_fib_lookup structure contains a union of two 16-bit > fields: > > union { > __u16 tot_len; > __u16 mtu_result; > }; > > which was supposed to compile to a 16-bit-aligned 16-bit field. On the > aforementioned setups it was instead both aligned and padded to 32-bits. > > Declare this inner union as __attribute__((packed, aligned(2))) such > that it always is of size 2 and is aligned to 16 bits. > > [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t > > Signed-off-by: Anton Protopopov <aspsk@isovalent.com> Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org> Acked-by: Daniel Borkmann <daniel@iogearbox.net>
On Wed, Apr 3, 2024 at 6:14 AM Daniel Borkmann <daniel@iogearbox.net> wrote: > > On 4/3/24 2:33 PM, Anton Protopopov wrote: > > The struct bpf_fib_lookup is supposed to be of size 64. A recent commit > > 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added > > a static assertion to check this property so that future changes to the > > structure will not accidentally break this assumption. > > > > As it immediately turned out, on some 32-bit arm systems, when AEABI=n, > > the total size of the structure was equal to 68, see [1]. This happened > > because the bpf_fib_lookup structure contains a union of two 16-bit > > fields: > > > > union { > > __u16 tot_len; > > __u16 mtu_result; > > }; > > > > which was supposed to compile to a 16-bit-aligned 16-bit field. On the > > aforementioned setups it was instead both aligned and padded to 32-bits. > > > > Declare this inner union as __attribute__((packed, aligned(2))) such > > that it always is of size 2 and is aligned to 16 bits. > > > > [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t > > > > Signed-off-by: Anton Protopopov <aspsk@isovalent.com> > > Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org> > Acked-by: Daniel Borkmann <daniel@iogearbox.net> Let's continue the tag fest :) Also fixes tag: Fixes: e1850ea9bd9e ("bpf: bpf_fib_lookup return MTU value as output when looked up") Should this also go through the bpf tree instead of bpf-next?
On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote: > The struct bpf_fib_lookup is supposed to be of size 64. A recent commit > 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added > a static assertion to check this property so that future changes to the > structure will not accidentally break this assumption. > > As it immediately turned out, on some 32-bit arm systems, when AEABI=n, > the total size of the structure was equal to 68, see [1]. This happened > because the bpf_fib_lookup structure contains a union of two 16-bit > fields: > > union { > __u16 tot_len; > __u16 mtu_result; > }; This union was introduced three years ago, so fixing it now means another incompatible ABI change. While you clearly change it back to what it should have been the entire time, it's not obvious that this is the correct thing to do after three years. I do wonder to what degree we still want to care about OABI at all. As I mentioned, I stopped testing it myself because it is no longer relevant to most people, and there are probably a number of other ABI issues. > which was supposed to compile to a 16-bit-aligned 16-bit field. On the > aforementioned setups it was instead both aligned and padded to 32-bits. > > Declare this inner union as __attribute__((packed, aligned(2))) such > that it always is of size 2 and is aligned to 16 bits. I think you probably want 32-bit alignment for the structure, to keep the ABI unchanged on all other architectures. Arnd
On 4/3/24 10:09 PM, Arnd Bergmann wrote: > On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote: >> The struct bpf_fib_lookup is supposed to be of size 64. A recent commit >> 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added >> a static assertion to check this property so that future changes to the >> structure will not accidentally break this assumption. >> >> As it immediately turned out, on some 32-bit arm systems, when AEABI=n, >> the total size of the structure was equal to 68, see [1]. This happened >> because the bpf_fib_lookup structure contains a union of two 16-bit >> fields: >> >> union { >> __u16 tot_len; >> __u16 mtu_result; >> }; > > This union was introduced three years ago, so fixing it now means > another incompatible ABI change. While you clearly change it back > to what it should have been the entire time, it's not obvious that > this is the correct thing to do after three years. > > I do wonder to what degree we still want to care about OABI > at all. As I mentioned, I stopped testing it myself because it > is no longer relevant to most people, and there are probably > a number of other ABI issues. > >> which was supposed to compile to a 16-bit-aligned 16-bit field. On the >> aforementioned setups it was instead both aligned and padded to 32-bits. >> >> Declare this inner union as __attribute__((packed, aligned(2))) such >> that it always is of size 2 and is aligned to 16 bits. > > I think you probably want 32-bit alignment for the structure, > to keep the ABI unchanged on all other architectures. Fwiw, on x86 nothing should change on this regard, see below pahole dump before/after. I think similar might be true for other archs as otherwise we should have seen a kbuild bot complaint on hitting the size assert. So it seems at this point only for the reported case of AEABI=n ... if it is no longer relevant to most people, I'd think the likelihood of such users and users who utilize this specific helper is even smaller. But we could queue it to bpf-next to give it some time. Before patch: struct bpf_fib_lookup { __u8 family; /* 0 1 */ __u8 l4_protocol; /* 1 1 */ __be16 sport; /* 2 2 */ __be16 dport; /* 4 2 */ union { __u16 tot_len; /* 6 2 */ __u16 mtu_result; /* 6 2 */ }; /* 6 2 */ __u32 ifindex; /* 8 4 */ union { __u8 tos; /* 12 1 */ __be32 flowinfo; /* 12 4 */ __u32 rt_metric; /* 12 4 */ }; /* 12 4 */ union { __be32 ipv4_src; /* 16 4 */ __u32 ipv6_src[4]; /* 16 16 */ }; /* 16 16 */ union { __be32 ipv4_dst; /* 32 4 */ __u32 ipv6_dst[4]; /* 32 16 */ }; /* 32 16 */ union { struct { __be16 h_vlan_proto; /* 48 2 */ __be16 h_vlan_TCI; /* 50 2 */ }; /* 48 4 */ __u32 tbid; /* 48 4 */ }; /* 48 4 */ __u8 smac[6]; /* 52 6 */ __u8 dmac[6]; /* 58 6 */ /* size: 64, cachelines: 1, members: 12 */ }; After patch: struct bpf_fib_lookup { __u8 family; /* 0 1 */ __u8 l4_protocol; /* 1 1 */ __be16 sport; /* 2 2 */ __be16 dport; /* 4 2 */ union { __u16 tot_len; /* 6 2 */ __u16 mtu_result; /* 6 2 */ } __attribute__((__aligned__(2))); /* 6 2 */ __u32 ifindex; /* 8 4 */ union { __u8 tos; /* 12 1 */ __be32 flowinfo; /* 12 4 */ __u32 rt_metric; /* 12 4 */ }; /* 12 4 */ union { __be32 ipv4_src; /* 16 4 */ __u32 ipv6_src[4]; /* 16 16 */ }; /* 16 16 */ union { __be32 ipv4_dst; /* 32 4 */ __u32 ipv6_dst[4]; /* 32 16 */ }; /* 32 16 */ union { struct { __be16 h_vlan_proto; /* 48 2 */ __be16 h_vlan_TCI; /* 50 2 */ }; /* 48 4 */ __u32 tbid; /* 48 4 */ }; /* 48 4 */ __u8 smac[6]; /* 52 6 */ __u8 dmac[6]; /* 58 6 */ /* size: 64, cachelines: 1, members: 12 */ /* forced alignments: 1 */ } __attribute__((__aligned__(4))); Thanks, Daniel
On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote: > On 4/3/24 10:09 PM, Arnd Bergmann wrote: >> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote: >>> >>> Declare this inner union as __attribute__((packed, aligned(2))) such >>> that it always is of size 2 and is aligned to 16 bits. >> >> I think you probably want 32-bit alignment for the structure, >> to keep the ABI unchanged on all other architectures. > > Fwiw, on x86 nothing should change on this regard, see below pahole dump > before/after. I think similar might be true for other archs as otherwise > we should have seen a kbuild bot complaint on hitting the size assert. It's not the structure layout that changes, just its alignment. Of course this is unlikely to cause actual bugs, but if there there is no real need to change it, I would leave the alignment the same as before. Arnd
On 24/04/04 12:31, Arnd Bergmann wrote: > On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote: > > On 4/3/24 10:09 PM, Arnd Bergmann wrote: > >> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote: > >>> > >>> Declare this inner union as __attribute__((packed, aligned(2))) such > >>> that it always is of size 2 and is aligned to 16 bits. > >> > >> I think you probably want 32-bit alignment for the structure, > >> to keep the ABI unchanged on all other architectures. > > > > Fwiw, on x86 nothing should change on this regard, see below pahole dump > > before/after. I think similar might be true for other archs as otherwise > > we should have seen a kbuild bot complaint on hitting the size assert. > > It's not the structure layout that changes, just its alignment. > Of course this is unlikely to cause actual bugs, but if there there > is no real need to change it, I would leave the alignment the same > as before. I think the struct will now be automatically 4-byte aligned, as it has the following layout: struct { u8 a; u8 b; u16 c; u16 d; union { u16 e; u16 f; } __aligned__(2); ... }; So if the union is 2-byte aligned, then the struct is automatically 4-byte aligned, because its address is 6 bytes less than the address of the union. In fact, as Daniel posted above, pahole shows that the struct actually has __aligned__(4) attribute in the patched version. I can add explicit __aligned__(4) to make this clear. > Arnd
From: Anton Protopopov <aspsk@isovalent.com> Date: Thu, 4 Apr 2024 09:56:51 +0200 > On 24/04/04 12:31, Arnd Bergmann wrote: >> On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote: >>> On 4/3/24 10:09 PM, Arnd Bergmann wrote: >>>> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote: >>>>> >>>>> Declare this inner union as __attribute__((packed, aligned(2))) such >>>>> that it always is of size 2 and is aligned to 16 bits. >>>> >>>> I think you probably want 32-bit alignment for the structure, >>>> to keep the ABI unchanged on all other architectures. >>> >>> Fwiw, on x86 nothing should change on this regard, see below pahole dump >>> before/after. I think similar might be true for other archs as otherwise >>> we should have seen a kbuild bot complaint on hitting the size assert. >> >> It's not the structure layout that changes, just its alignment. >> Of course this is unlikely to cause actual bugs, but if there there >> is no real need to change it, I would leave the alignment the same >> as before. > > I think the struct will now be automatically 4-byte aligned, as it > has the following layout: > > struct { > u8 a; > u8 b; > u16 c; > u16 d; > union { u16 e; u16 f; } __aligned__(2); > ... > }; > > So if the union is 2-byte aligned, then the struct is automatically > 4-byte aligned, because its address is 6 bytes less than the address > of the union. In fact, as Daniel posted above, pahole shows that the > struct actually has __aligned__(4) attribute in the patched version. > I can add explicit __aligned__(4) to make this clear. I think it would be fine to not introduce an explicit attribute as long as pahole shows the structure is already 4-byte aligned. Regarding aligning it to 32 or 64 bytes -- the actual benefit can be checked via scripts/bloat-o-meter. Usually alignment bigger than 8 bytes don't change anything in the object code, at least on x86_64. > >> Arnd Thanks, Olek
On Thu, Apr 4, 2024, at 09:56, Anton Protopopov wrote: > On 24/04/04 12:31, Arnd Bergmann wrote: >> On Wed, Apr 3, 2024, at 23:00, Daniel Borkmann wrote: >> > On 4/3/24 10:09 PM, Arnd Bergmann wrote: >> >> On Wed, Apr 3, 2024, at 14:33, Anton Protopopov wrote: >> >>> >> >>> Declare this inner union as __attribute__((packed, aligned(2))) such >> >>> that it always is of size 2 and is aligned to 16 bits. >> >> >> >> I think you probably want 32-bit alignment for the structure, >> >> to keep the ABI unchanged on all other architectures. >> > >> > Fwiw, on x86 nothing should change on this regard, see below pahole dump >> > before/after. I think similar might be true for other archs as otherwise >> > we should have seen a kbuild bot complaint on hitting the size assert. >> >> It's not the structure layout that changes, just its alignment. >> Of course this is unlikely to cause actual bugs, but if there there >> is no real need to change it, I would leave the alignment the same >> as before. > > I think the struct will now be automatically 4-byte aligned, as it > has the following layout: > > struct { > u8 a; > u8 b; > u16 c; > u16 d; > union { u16 e; u16 f; } __aligned__(2); > ... > }; > > So if the union is 2-byte aligned, then the struct is automatically > 4-byte aligned, because its address is 6 bytes less than the address > of the union. In fact, as Daniel posted above, pahole shows that the > struct actually has __aligned__(4) attribute in the patched version. > I can add explicit __aligned__(4) to make this clear. Ah right. I misread your patch and assumed that you gave the outer structure 2-byte alignment as well, but it's only the inner union that needs it, which you did correctly. Arnd
Hello: This patch was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <andrii@kernel.org>: On Wed, 3 Apr 2024 14:33:03 +0200 you wrote: > The struct bpf_fib_lookup is supposed to be of size 64. A recent commit > 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added > a static assertion to check this property so that future changes to the > structure will not accidentally break this assumption. > > As it immediately turned out, on some 32-bit arm systems, when AEABI=n, > the total size of the structure was equal to 68, see [1]. This happened > because the bpf_fib_lookup structure contains a union of two 16-bit > fields: > > [...] Here is the summary with links: - [bpf-next] bpf: pack struct bpf_fib_lookup https://git.kernel.org/bpf/bpf-next/c/f91717007217 You are awesome, thank you!
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 79c548276b6b..6fe9f11c8abe 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7157,7 +7157,7 @@ struct bpf_fib_lookup { /* output: MTU value */ __u16 mtu_result; - }; + } __attribute__((packed, aligned(2))); /* input: L3 device index for lookup * output: device index from FIB lookup */ diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 79c548276b6b..6fe9f11c8abe 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7157,7 +7157,7 @@ struct bpf_fib_lookup { /* output: MTU value */ __u16 mtu_result; - }; + } __attribute__((packed, aligned(2))); /* input: L3 device index for lookup * output: device index from FIB lookup */
The struct bpf_fib_lookup is supposed to be of size 64. A recent commit 59b418c7063d ("bpf: Add a check for struct bpf_fib_lookup size") added a static assertion to check this property so that future changes to the structure will not accidentally break this assumption. As it immediately turned out, on some 32-bit arm systems, when AEABI=n, the total size of the structure was equal to 68, see [1]. This happened because the bpf_fib_lookup structure contains a union of two 16-bit fields: union { __u16 tot_len; __u16 mtu_result; }; which was supposed to compile to a 16-bit-aligned 16-bit field. On the aforementioned setups it was instead both aligned and padded to 32-bits. Declare this inner union as __attribute__((packed, aligned(2))) such that it always is of size 2 and is aligned to 16 bits. [1] https://lore.kernel.org/all/CA+G9fYtsoP51f-oP_Sp5MOq-Ffv8La2RztNpwvE6+R1VtFiLrw@mail.gmail.com/#t Signed-off-by: Anton Protopopov <aspsk@isovalent.com> --- include/uapi/linux/bpf.h | 2 +- tools/include/uapi/linux/bpf.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)