| Message ID | 20231026081959.3477034-3-lixiaoyan@google.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | Analyze and Reorganize core Networking Structs to optimize cacheline consumption | expand |
On Thu, Oct 26, 2023 at 10:20 AM Coco Li <lixiaoyan@google.com> wrote: > > Set up build time warnings to safegaurd against future header changes of safeguard > organized structs. > > Signed-off-by: Coco Li <lixiaoyan@google.com> > Suggested-by: Daniel Borkmann <daniel@iogearbox.net> > Acked-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Eric Dumazet <edumazet@google.com> > --- > include/linux/cache.h | 18 ++++++++++++++++++ > 1 file changed, 18 insertions(+) > > diff --git a/include/linux/cache.h b/include/linux/cache.h > index 9900d20b76c28..4e547beccd6a5 100644 > --- a/include/linux/cache.h > +++ b/include/linux/cache.h > @@ -85,6 +85,24 @@ > #define cache_line_size() L1_CACHE_BYTES > #endif > > +#ifndef __cacheline_group_begin > +#define __cacheline_group_begin(GROUP) \ > + __u8 __cacheline_group_begin__##GROUP[0] > +#endif > + > +#ifndef __cacheline_group_end > +#define __cacheline_group_end(GROUP) \ > + __u8 __cacheline_group_end__##GROUP[0] > +#endif > + > +#ifndef CACHELINE_ASSERT_GROUP_MEMBER > +#define CACHELINE_ASSERT_GROUP_MEMBER(TYPE, GROUP, MEMBER) \ > + BUILD_BUG_ON(!(offsetof(TYPE, MEMBER) >= \ > + offsetofend(TYPE, __cacheline_group_begin__##GROUP) && \ > + offsetofend(TYPE, MEMBER) <= \ > + offsetof(TYPE, __cacheline_group_end__##GROUP))) > +#endif > + > /* > * Helper to add padding within a struct to ensure data fall into separate > * cachelines. > -- > 2.42.0.758.gaed0368e0e-goog >
On Thu, 26 Oct 2023 08:19:55 +0000 Coco Li wrote: > Set up build time warnings to safegaurd against future header changes > of organized structs. TBH I had some doubts about the value of these asserts, I thought it was just me but I was talking to Vadim F and he brought up the same question. IIUC these markings will protect us from people moving the members out of the cache lines. Does that actually happen? It'd be less typing to assert the _size_ of each group, which protects from both moving out, and adding stuff haphazardly, which I'd guess is more common. Perhaps we should do that in addition?
From: Jakub Kicinski <kuba@kernel.org> Date: Thu, 26 Oct 2023 07:17:01 -0700 > On Thu, 26 Oct 2023 08:19:55 +0000 Coco Li wrote: > > Set up build time warnings to safegaurd against future header changes > > of organized structs. > > TBH I had some doubts about the value of these asserts, I thought > it was just me but I was talking to Vadim F and he brought up > the same question. > > IIUC these markings will protect us from people moving the members > out of the cache lines. Does that actually happen? > > It'd be less typing to assert the _size_ of each group, which protects > from both moving out, and adding stuff haphazardly, which I'd guess is > more common. Perhaps we should do that in addition? Also, we could assert the size of the struct itself and further add ____cacheline_aligned_in_smp to __cacheline_group_begin() ? If someone adds/removes a member before __cacheline_group_begin(), two groups could share the same cacheline.
On Thu, Oct 26, 2023 at 4:39 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > From: Jakub Kicinski <kuba@kernel.org> > Date: Thu, 26 Oct 2023 07:17:01 -0700 > > On Thu, 26 Oct 2023 08:19:55 +0000 Coco Li wrote: > > > Set up build time warnings to safegaurd against future header changes > > > of organized structs. > > > > TBH I had some doubts about the value of these asserts, I thought > > it was just me but I was talking to Vadim F and he brought up > > the same question. > > > > IIUC these markings will protect us from people moving the members > > out of the cache lines. Does that actually happen? > > > > It'd be less typing to assert the _size_ of each group, which protects > > from both moving out, and adding stuff haphazardly, which I'd guess is > > more common. Perhaps we should do that in addition? SGTM, will add in next patch. > > Also, we could assert the size of the struct itself and further > add ____cacheline_aligned_in_smp to __cacheline_group_begin() ? > > If someone adds/removes a member before __cacheline_group_begin(), > two groups could share the same cacheline. > > I think we shouldn't add ____cacheline_aligned_in_smp/____cacheline_aligned together with cacheline_group_begin, because especially for read-only cache lines that are side by side, enforcing them to be in separate cache lines will result in more total cache lines when we don't care about the same cache line being shared by multiple cpus. An example would be tx_read_only group vs rx_read_only group vs txrx_read_only groups, since there were suggestions that we mark these cache groups separately. For cache line separations that we care about (i.e. tcp_sock in tcp.h) where read and write might potentially be mixed, the ____cacheline_aligned should probably still be only the in header file only.
On Fri, Oct 27, 2023 at 1:39 AM Kuniyuki Iwashima <kuniyu@amazon.com> wrote: > > From: Jakub Kicinski <kuba@kernel.org> > Date: Thu, 26 Oct 2023 07:17:01 -0700 > > On Thu, 26 Oct 2023 08:19:55 +0000 Coco Li wrote: > > > Set up build time warnings to safegaurd against future header changes > > > of organized structs. > > > > TBH I had some doubts about the value of these asserts, I thought > > it was just me but I was talking to Vadim F and he brought up > > the same question. > > > > IIUC these markings will protect us from people moving the members > > out of the cache lines. Does that actually happen? > > > > It'd be less typing to assert the _size_ of each group, which protects > > from both moving out, and adding stuff haphazardly, which I'd guess is > > more common. Perhaps we should do that in addition? > > Also, we could assert the size of the struct itself and further > add ____cacheline_aligned_in_smp to __cacheline_group_begin() ? Nope, automatically adding ____cacheline_aligned_in_smp to each group is not beneficial. We ran a lot of experiments and concluded that grouping was the best strategy. Adding ____cacheline_aligned_in_smp adds holes and TX + RX traffic (RPC) would use more cache lines than necessary. > > If someone adds/removes a member before __cacheline_group_begin(), > two groups could share the same cacheline. > >
On 10/26/23 4:17 PM, Jakub Kicinski wrote: > On Thu, 26 Oct 2023 08:19:55 +0000 Coco Li wrote: >> Set up build time warnings to safegaurd against future header changes >> of organized structs. > > TBH I had some doubts about the value of these asserts, I thought > it was just me but I was talking to Vadim F and he brought up > the same question. > > IIUC these markings will protect us from people moving the members > out of the cache lines. Does that actually happen? > > It'd be less typing to assert the _size_ of each group, which protects > from both moving out, and adding stuff haphazardly, which I'd guess is > more common. Perhaps we should do that in addition? Size would be good, I also had that in the prototype in [0], I think blowing up the size is a bigger risk than moving existing members to somewhere else in the struct, and this way it is kind of a forcing factor to think deeper when this triggers, and helps in reviews hopefully as well since it's an explicit change when the size is bumped. Having this in addition would be nice imo. Thanks, Daniel [0] https://lore.kernel.org/netdev/50ca7bc1-e5c1-cb79-b2af-e5cd83b54dab@iogearbox.net/
diff --git a/include/linux/cache.h b/include/linux/cache.h index 9900d20b76c28..4e547beccd6a5 100644 --- a/include/linux/cache.h +++ b/include/linux/cache.h @@ -85,6 +85,24 @@ #define cache_line_size() L1_CACHE_BYTES #endif +#ifndef __cacheline_group_begin +#define __cacheline_group_begin(GROUP) \ + __u8 __cacheline_group_begin__##GROUP[0] +#endif + +#ifndef __cacheline_group_end +#define __cacheline_group_end(GROUP) \ + __u8 __cacheline_group_end__##GROUP[0] +#endif + +#ifndef CACHELINE_ASSERT_GROUP_MEMBER +#define CACHELINE_ASSERT_GROUP_MEMBER(TYPE, GROUP, MEMBER) \ + BUILD_BUG_ON(!(offsetof(TYPE, MEMBER) >= \ + offsetofend(TYPE, __cacheline_group_begin__##GROUP) && \ + offsetofend(TYPE, MEMBER) <= \ + offsetof(TYPE, __cacheline_group_end__##GROUP))) +#endif + /* * Helper to add padding within a struct to ensure data fall into separate * cachelines.