| Message ID | 20221111132258.762033-1-xukuohai@huaweicloud.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | BPF |
| Headers | show |
| Series | [bpf-next,v5] bpf: Initialize same number of free nodes for each pcpu_freelist | expand |
On Fri, Nov 11, 2022 at 5:06 AM Xu Kuohai <xukuohai@huaweicloud.com> wrote: > > From: Xu Kuohai <xukuohai@huawei.com> > > pcpu_freelist_populate() initializes nr_elems / num_possible_cpus() + 1 > free nodes for some CPUs, and then possibly one CPU with fewer nodes, > followed by remaining cpus with 0 nodes. For example, when nr_elems == 256 > and num_possible_cpus() == 32, CPU 0~27 each gets 9 free nodes, CPU 28 gets > 4 free nodes, CPU 29~31 get 0 free nodes, while in fact each CPU should get > 8 nodes equally. > > This patch initializes nr_elems / num_possible_cpus() free nodes for each > CPU firstly, then allocates the remaining free nodes by one for each CPU > until no free nodes left. > > Signed-off-by: Xu Kuohai <xukuohai@huawei.com> > Acked-by: Yonghong Song <yhs@fb.com> > --- > v5: Move "head = per_cpu_ptr ..." out of the i-loop as suggesetd by wuqiang I've applied v4 (before seeing v5) with head moved out of innermost loop. But I do prefer v4 anyways. > v4: Remove unneeded min() > v3: Simplify code as suggested by Andrii > v2: Update commit message and add Yonghong's ack > --- > kernel/bpf/percpu_freelist.c | 21 +++++++++------------ > 1 file changed, 9 insertions(+), 12 deletions(-) > > diff --git a/kernel/bpf/percpu_freelist.c b/kernel/bpf/percpu_freelist.c > index b6e7f5c5b9ab..de336e3dffee 100644 > --- a/kernel/bpf/percpu_freelist.c > +++ b/kernel/bpf/percpu_freelist.c > @@ -100,22 +100,19 @@ void pcpu_freelist_populate(struct pcpu_freelist *s, void *buf, u32 elem_size, > u32 nr_elems) > { > struct pcpu_freelist_head *head; > - int i, cpu, pcpu_entries; > + unsigned int cpu_idx = 0, cpu, i, j, n, m; > > - pcpu_entries = nr_elems / num_possible_cpus() + 1; > - i = 0; > + n = nr_elems / num_possible_cpus(); > + m = nr_elems % num_possible_cpus(); > > for_each_possible_cpu(cpu) { > -again: > head = per_cpu_ptr(s->freelist, cpu); > - /* No locking required as this is not visible yet. */ > - pcpu_freelist_push_node(head, buf); > - i++; > - buf += elem_size; > - if (i == nr_elems) > - break; > - if (i % pcpu_entries) > - goto again; > + j = n + (cpu_idx++ < m ? 1 : 0); > + for (i = 0; i < j; i++) { > + /* No locking required as this is not visible yet. */ > + pcpu_freelist_push_node(head, buf); > + buf += elem_size; > + } > } > } > > -- > 2.30.2 >
diff --git a/kernel/bpf/percpu_freelist.c b/kernel/bpf/percpu_freelist.c index b6e7f5c5b9ab..de336e3dffee 100644 --- a/kernel/bpf/percpu_freelist.c +++ b/kernel/bpf/percpu_freelist.c @@ -100,22 +100,19 @@ void pcpu_freelist_populate(struct pcpu_freelist *s, void *buf, u32 elem_size, u32 nr_elems) { struct pcpu_freelist_head *head; - int i, cpu, pcpu_entries; + unsigned int cpu_idx = 0, cpu, i, j, n, m; - pcpu_entries = nr_elems / num_possible_cpus() + 1; - i = 0; + n = nr_elems / num_possible_cpus(); + m = nr_elems % num_possible_cpus(); for_each_possible_cpu(cpu) { -again: head = per_cpu_ptr(s->freelist, cpu); - /* No locking required as this is not visible yet. */ - pcpu_freelist_push_node(head, buf); - i++; - buf += elem_size; - if (i == nr_elems) - break; - if (i % pcpu_entries) - goto again; + j = n + (cpu_idx++ < m ? 1 : 0); + for (i = 0; i < j; i++) { + /* No locking required as this is not visible yet. */ + pcpu_freelist_push_node(head, buf); + buf += elem_size; + } } }