| Message ID | cover.1697093455.git.hengqi@linux.alibaba.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | virtio-net: support dynamic coalescing moderation | expand |
On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > Now, virtio-net already supports per-queue moderation parameter > setting. Based on this, we use the netdim library of linux to support > dynamic coalescing moderation for virtio-net. > > Due to hardware scheduling issues, we only tested rx dim. Do you have PPS numbers? And TX numbers are also important as the throughput could be misleading due to various reasons. Thanks > > @Test env > rxq0 has affinity to cpu0. > > @Test cmd > client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > server: taskset -c 0 sockperf sr --tcp > > @Test res > The second column is the ratio of the result returned by client > when rx dim is enabled to the result returned by client when > rx dim is disabled. > -------------------------------------- > | msg_size | rx_dim=on / rx_dim=off | > -------------------------------------- > | 14B | + 3% | > -------------------------------------- > | 100B | + 16% | > -------------------------------------- > | 500B | + 25% | > -------------------------------------- > | 1400B | + 28% | > -------------------------------------- > | 2048B | + 22% | > -------------------------------------- > | 4096B | + 5% | > -------------------------------------- > > --- > This patch set was part of the previous netdim patch set[1]. > [1] was split into a merged bugfix set[2] and the current set. > The previous relevant commentators have been Cced. > > [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > > Heng Qi (5): > virtio-net: returns whether napi is complete > virtio-net: separate rx/tx coalescing moderation cmds > virtio-net: extract virtqueue coalescig cmd for reuse > virtio-net: support rx netdim > virtio-net: support tx netdim > > drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > 1 file changed, 322 insertions(+), 72 deletions(-) > > -- > 2.19.1.6.gb485710b > >
On Thu, Oct 12, 2023 at 4:29 PM Jason Wang <jasowang@redhat.com> wrote: > > On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > > > Now, virtio-net already supports per-queue moderation parameter > > setting. Based on this, we use the netdim library of linux to support > > dynamic coalescing moderation for virtio-net. > > > > Due to hardware scheduling issues, we only tested rx dim. > > Do you have PPS numbers? And TX numbers are also important as the > throughput could be misleading due to various reasons. Another consideration: We currently have two TX mode, NAPI by default, and orphan. Simple pktgen test shows NAPI can only achieve 30% of orphan. So we need to make sure: 1) dim helps for NAPI, and if NAPI can compete with orphan, we can drop the orphan code completely which is a great release and simplification of the codes. And it means we can have BQL, and other good stuff on top easily. 2) dim doesn't cause regression for orphan Thanks > > Thanks > > > > > @Test env > > rxq0 has affinity to cpu0. > > > > @Test cmd > > client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > > server: taskset -c 0 sockperf sr --tcp > > > > @Test res > > The second column is the ratio of the result returned by client > > when rx dim is enabled to the result returned by client when > > rx dim is disabled. > > -------------------------------------- > > | msg_size | rx_dim=on / rx_dim=off | > > -------------------------------------- > > | 14B | + 3% | > > -------------------------------------- > > | 100B | + 16% | > > -------------------------------------- > > | 500B | + 25% | > > -------------------------------------- > > | 1400B | + 28% | > > -------------------------------------- > > | 2048B | + 22% | > > -------------------------------------- > > | 4096B | + 5% | > > -------------------------------------- > > > > --- > > This patch set was part of the previous netdim patch set[1]. > > [1] was split into a merged bugfix set[2] and the current set. > > The previous relevant commentators have been Cced. > > > > [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > > [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > > > > Heng Qi (5): > > virtio-net: returns whether napi is complete > > virtio-net: separate rx/tx coalescing moderation cmds > > virtio-net: extract virtqueue coalescig cmd for reuse > > virtio-net: support rx netdim > > virtio-net: support tx netdim > > > > drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > > 1 file changed, 322 insertions(+), 72 deletions(-) > > > > -- > > 2.19.1.6.gb485710b > > > >
在 2023/10/12 下午4:29, Jason Wang 写道: > On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >> Now, virtio-net already supports per-queue moderation parameter >> setting. Based on this, we use the netdim library of linux to support >> dynamic coalescing moderation for virtio-net. >> >> Due to hardware scheduling issues, we only tested rx dim. > Do you have PPS numbers? And TX numbers are also important as the > throughput could be misleading due to various reasons. OK, will reply until the test environment is ready (the test machine is currently locked by someone else). Thanks! > > Thanks > >> @Test env >> rxq0 has affinity to cpu0. >> >> @Test cmd >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >> server: taskset -c 0 sockperf sr --tcp >> >> @Test res >> The second column is the ratio of the result returned by client >> when rx dim is enabled to the result returned by client when >> rx dim is disabled. >> -------------------------------------- >> | msg_size | rx_dim=on / rx_dim=off | >> -------------------------------------- >> | 14B | + 3% | >> -------------------------------------- >> | 100B | + 16% | >> -------------------------------------- >> | 500B | + 25% | >> -------------------------------------- >> | 1400B | + 28% | >> -------------------------------------- >> | 2048B | + 22% | >> -------------------------------------- >> | 4096B | + 5% | >> -------------------------------------- >> >> --- >> This patch set was part of the previous netdim patch set[1]. >> [1] was split into a merged bugfix set[2] and the current set. >> The previous relevant commentators have been Cced. >> >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >> >> Heng Qi (5): >> virtio-net: returns whether napi is complete >> virtio-net: separate rx/tx coalescing moderation cmds >> virtio-net: extract virtqueue coalescig cmd for reuse >> virtio-net: support rx netdim >> virtio-net: support tx netdim >> >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >> 1 file changed, 322 insertions(+), 72 deletions(-) >> >> -- >> 2.19.1.6.gb485710b >> >>
在 2023/10/12 下午4:29, Jason Wang 写道: > On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >> Now, virtio-net already supports per-queue moderation parameter >> setting. Based on this, we use the netdim library of linux to support >> dynamic coalescing moderation for virtio-net. >> >> Due to hardware scheduling issues, we only tested rx dim. > Do you have PPS numbers? And TX numbers are also important as the > throughput could be misleading due to various reasons. Hi Jason! The comparison of rx netdim performance is as follows: (the backend supporting tx dim is not yet ready) I. Sockperf UDP ================================================= 1. Env rxq_0 is affinity to cpu_0 2. Cmd client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B server: taskset -c 0 sockperf sr -p 8989 3. Result dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. ================================================= II. Redis ================================================= 1. Env There are 8 rxqs and rxq_i is affinity to cpu_i. 2. Result When all cpus are 100%, ops/sec of memtier_benchmark client is dim off: 978437.23 dim on: 1143638.28 ================================================= III. Nginx ================================================= 1. Env There are 8 rxqs and rxq_i is affinity to cpu_i. 2. Result When all cpus are 100%, requests/sec of wrk client is dim off: 877931.67 dim on: 1019160.31 ================================================= Thanks! > > Thanks > >> @Test env >> rxq0 has affinity to cpu0. >> >> @Test cmd >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >> server: taskset -c 0 sockperf sr --tcp >> >> @Test res >> The second column is the ratio of the result returned by client >> when rx dim is enabled to the result returned by client when >> rx dim is disabled. >> -------------------------------------- >> | msg_size | rx_dim=on / rx_dim=off | >> -------------------------------------- >> | 14B | + 3% | >> -------------------------------------- >> | 100B | + 16% | >> -------------------------------------- >> | 500B | + 25% | >> -------------------------------------- >> | 1400B | + 28% | >> -------------------------------------- >> | 2048B | + 22% | >> -------------------------------------- >> | 4096B | + 5% | >> -------------------------------------- >> >> --- >> This patch set was part of the previous netdim patch set[1]. >> [1] was split into a merged bugfix set[2] and the current set. >> The previous relevant commentators have been Cced. >> >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >> >> Heng Qi (5): >> virtio-net: returns whether napi is complete >> virtio-net: separate rx/tx coalescing moderation cmds >> virtio-net: extract virtqueue coalescig cmd for reuse >> virtio-net: support rx netdim >> virtio-net: support tx netdim >> >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >> 1 file changed, 322 insertions(+), 72 deletions(-) >> >> -- >> 2.19.1.6.gb485710b >> >>
On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > > > 在 2023/10/12 下午4:29, Jason Wang 写道: > > On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > >> Now, virtio-net already supports per-queue moderation parameter > >> setting. Based on this, we use the netdim library of linux to support > >> dynamic coalescing moderation for virtio-net. > >> > >> Due to hardware scheduling issues, we only tested rx dim. > > Do you have PPS numbers? And TX numbers are also important as the > > throughput could be misleading due to various reasons. > > Hi Jason! > > The comparison of rx netdim performance is as follows: > (the backend supporting tx dim is not yet ready) Thanks a lot for the numbers. I'd still expect the TX result as I did play tx interrupt coalescing about 10 years ago. I will start to review the series but let's try to have some TX numbers as well. Btw, it would be more convenient to have a raw PPS benchmark. E.g you can try to use a software or hardware packet generator. Thanks > > > I. Sockperf UDP > ================================================= > 1. Env > rxq_0 is affinity to cpu_0 > > 2. Cmd > client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > server: taskset -c 0 sockperf sr -p 8989 > > 3. Result > dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > ================================================= > > > II. Redis > ================================================= > 1. Env > There are 8 rxqs and rxq_i is affinity to cpu_i. > > 2. Result > When all cpus are 100%, ops/sec of memtier_benchmark client is > dim off: 978437.23 > dim on: 1143638.28 > ================================================= > > > III. Nginx > ================================================= > 1. Env > There are 8 rxqs and rxq_i is affinity to cpu_i. > > 2. Result > When all cpus are 100%, requests/sec of wrk client is > dim off: 877931.67 > dim on: 1019160.31 > ================================================= > > Thanks! > > > > > Thanks > > > >> @Test env > >> rxq0 has affinity to cpu0. > >> > >> @Test cmd > >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > >> server: taskset -c 0 sockperf sr --tcp > >> > >> @Test res > >> The second column is the ratio of the result returned by client > >> when rx dim is enabled to the result returned by client when > >> rx dim is disabled. > >> -------------------------------------- > >> | msg_size | rx_dim=on / rx_dim=off | > >> -------------------------------------- > >> | 14B | + 3% | > >> -------------------------------------- > >> | 100B | + 16% | > >> -------------------------------------- > >> | 500B | + 25% | > >> -------------------------------------- > >> | 1400B | + 28% | > >> -------------------------------------- > >> | 2048B | + 22% | > >> -------------------------------------- > >> | 4096B | + 5% | > >> -------------------------------------- > >> > >> --- > >> This patch set was part of the previous netdim patch set[1]. > >> [1] was split into a merged bugfix set[2] and the current set. > >> The previous relevant commentators have been Cced. > >> > >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > >> > >> Heng Qi (5): > >> virtio-net: returns whether napi is complete > >> virtio-net: separate rx/tx coalescing moderation cmds > >> virtio-net: extract virtqueue coalescig cmd for reuse > >> virtio-net: support rx netdim > >> virtio-net: support tx netdim > >> > >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > >> 1 file changed, 322 insertions(+), 72 deletions(-) > >> > >> -- > >> 2.19.1.6.gb485710b > >> > >> > >
On Thu, Oct 12, 2023 at 03:44:04PM +0800, Heng Qi wrote: > Now, virtio-net already supports per-queue moderation parameter > setting. Based on this, we use the netdim library of linux to support > dynamic coalescing moderation for virtio-net. > > Due to hardware scheduling issues, we only tested rx dim. So patches 1 to 4 look ok but patch 5 is untested - we should probably wait until it's tested properly. > @Test env > rxq0 has affinity to cpu0. > > @Test cmd > client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > server: taskset -c 0 sockperf sr --tcp > > @Test res > The second column is the ratio of the result returned by client > when rx dim is enabled to the result returned by client when > rx dim is disabled. > -------------------------------------- > | msg_size | rx_dim=on / rx_dim=off | > -------------------------------------- > | 14B | + 3% | > -------------------------------------- > | 100B | + 16% | > -------------------------------------- > | 500B | + 25% | > -------------------------------------- > | 1400B | + 28% | > -------------------------------------- > | 2048B | + 22% | > -------------------------------------- > | 4096B | + 5% | > -------------------------------------- > > --- > This patch set was part of the previous netdim patch set[1]. > [1] was split into a merged bugfix set[2] and the current set. > The previous relevant commentators have been Cced. > > [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > > Heng Qi (5): > virtio-net: returns whether napi is complete > virtio-net: separate rx/tx coalescing moderation cmds > virtio-net: extract virtqueue coalescig cmd for reuse > virtio-net: support rx netdim > virtio-net: support tx netdim > > drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > 1 file changed, 322 insertions(+), 72 deletions(-) > > -- > 2.19.1.6.gb485710b
On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote: > On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > > > > > > > 在 2023/10/12 下午4:29, Jason Wang 写道: > > > On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > >> Now, virtio-net already supports per-queue moderation parameter > > >> setting. Based on this, we use the netdim library of linux to support > > >> dynamic coalescing moderation for virtio-net. > > >> > > >> Due to hardware scheduling issues, we only tested rx dim. > > > Do you have PPS numbers? And TX numbers are also important as the > > > throughput could be misleading due to various reasons. > > > > Hi Jason! > > > > The comparison of rx netdim performance is as follows: > > (the backend supporting tx dim is not yet ready) > > Thanks a lot for the numbers. > > I'd still expect the TX result as I did play tx interrupt coalescing > about 10 years ago. > > I will start to review the series but let's try to have some TX numbers as well. > > Btw, it would be more convenient to have a raw PPS benchmark. E.g you > can try to use a software or hardware packet generator. > > Thanks Latency results are also kind of interesting. > > > > > > I. Sockperf UDP > > ================================================= > > 1. Env > > rxq_0 is affinity to cpu_0 > > > > 2. Cmd > > client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > > server: taskset -c 0 sockperf sr -p 8989 > > > > 3. Result > > dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > > dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > > ================================================= > > > > > > II. Redis > > ================================================= > > 1. Env > > There are 8 rxqs and rxq_i is affinity to cpu_i. > > > > 2. Result > > When all cpus are 100%, ops/sec of memtier_benchmark client is > > dim off: 978437.23 > > dim on: 1143638.28 > > ================================================= > > > > > > III. Nginx > > ================================================= > > 1. Env > > There are 8 rxqs and rxq_i is affinity to cpu_i. > > > > 2. Result > > When all cpus are 100%, requests/sec of wrk client is > > dim off: 877931.67 > > dim on: 1019160.31 > > ================================================= > > > > Thanks! > > > > > > > > Thanks > > > > > >> @Test env > > >> rxq0 has affinity to cpu0. > > >> > > >> @Test cmd > > >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > > >> server: taskset -c 0 sockperf sr --tcp > > >> > > >> @Test res > > >> The second column is the ratio of the result returned by client > > >> when rx dim is enabled to the result returned by client when > > >> rx dim is disabled. > > >> -------------------------------------- > > >> | msg_size | rx_dim=on / rx_dim=off | > > >> -------------------------------------- > > >> | 14B | + 3% | > > >> -------------------------------------- > > >> | 100B | + 16% | > > >> -------------------------------------- > > >> | 500B | + 25% | > > >> -------------------------------------- > > >> | 1400B | + 28% | > > >> -------------------------------------- > > >> | 2048B | + 22% | > > >> -------------------------------------- > > >> | 4096B | + 5% | > > >> -------------------------------------- > > >> > > >> --- > > >> This patch set was part of the previous netdim patch set[1]. > > >> [1] was split into a merged bugfix set[2] and the current set. > > >> The previous relevant commentators have been Cced. > > >> > > >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > > >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > > >> > > >> Heng Qi (5): > > >> virtio-net: returns whether napi is complete > > >> virtio-net: separate rx/tx coalescing moderation cmds > > >> virtio-net: extract virtqueue coalescig cmd for reuse > > >> virtio-net: support rx netdim > > >> virtio-net: support tx netdim > > >> > > >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > > >> 1 file changed, 322 insertions(+), 72 deletions(-) > > >> > > >> -- > > >> 2.19.1.6.gb485710b > > >> > > >> > > > >
在 2023/10/25 上午9:18, Jason Wang 写道: > On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >> >> >> 在 2023/10/12 下午4:29, Jason Wang 写道: >>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>>> Now, virtio-net already supports per-queue moderation parameter >>>> setting. Based on this, we use the netdim library of linux to support >>>> dynamic coalescing moderation for virtio-net. >>>> >>>> Due to hardware scheduling issues, we only tested rx dim. >>> Do you have PPS numbers? And TX numbers are also important as the >>> throughput could be misleading due to various reasons. >> Hi Jason! >> >> The comparison of rx netdim performance is as follows: >> (the backend supporting tx dim is not yet ready) > Thanks a lot for the numbers. > > I'd still expect the TX result as I did play tx interrupt coalescing Hi, Jason. Sorry for the late reply to this! Our team has been blocked by other priorities the past few days. For tx dim, we have a fixed empirical value internally. This value performs better overall than manually adjusting the tx timer register --> I'll do not have tx numbers. :( So in the short term I no longer try to push [5/5] patch for tx dim and try to return -EOPNOTSUPP for it, sorry for this. > about 10 years ago. > > I will start to review the series but let's try to have some TX numbers as well. > > Btw, it would be more convenient to have a raw PPS benchmark. E.g you I got some raw pps data using pktgen from linux/sample/pktgen: 1. tx cmd ./pktgen_sample02_multiqueue.sh -i eth1 -s 44 -d ${dst_ip} -m ${dst_mac} -t 8 -f 0 -n 0 This uses 8 kpktgend threads to inject data into eth1. 2. Rx side loads a simple xdp prog which drops all received udp packets. 3. Data pps: ~1000w rx dim off: cpu idle= ~35% rx dim on: cpu idle= ~76% Thanks! > can try to use a software or hardware packet generator. > > Thanks > >> >> I. Sockperf UDP >> ================================================= >> 1. Env >> rxq_0 is affinity to cpu_0 >> >> 2. Cmd >> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B >> server: taskset -c 0 sockperf sr -p 8989 >> >> 3. Result >> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. >> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. >> ================================================= >> >> >> II. Redis >> ================================================= >> 1. Env >> There are 8 rxqs and rxq_i is affinity to cpu_i. >> >> 2. Result >> When all cpus are 100%, ops/sec of memtier_benchmark client is >> dim off: 978437.23 >> dim on: 1143638.28 >> ================================================= >> >> >> III. Nginx >> ================================================= >> 1. Env >> There are 8 rxqs and rxq_i is affinity to cpu_i. >> >> 2. Result >> When all cpus are 100%, requests/sec of wrk client is >> dim off: 877931.67 >> dim on: 1019160.31 >> ================================================= >> >> Thanks! >> >>> Thanks >>> >>>> @Test env >>>> rxq0 has affinity to cpu0. >>>> >>>> @Test cmd >>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >>>> server: taskset -c 0 sockperf sr --tcp >>>> >>>> @Test res >>>> The second column is the ratio of the result returned by client >>>> when rx dim is enabled to the result returned by client when >>>> rx dim is disabled. >>>> -------------------------------------- >>>> | msg_size | rx_dim=on / rx_dim=off | >>>> -------------------------------------- >>>> | 14B | + 3% | >>>> -------------------------------------- >>>> | 100B | + 16% | >>>> -------------------------------------- >>>> | 500B | + 25% | >>>> -------------------------------------- >>>> | 1400B | + 28% | >>>> -------------------------------------- >>>> | 2048B | + 22% | >>>> -------------------------------------- >>>> | 4096B | + 5% | >>>> -------------------------------------- >>>> >>>> --- >>>> This patch set was part of the previous netdim patch set[1]. >>>> [1] was split into a merged bugfix set[2] and the current set. >>>> The previous relevant commentators have been Cced. >>>> >>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >>>> >>>> Heng Qi (5): >>>> virtio-net: returns whether napi is complete >>>> virtio-net: separate rx/tx coalescing moderation cmds >>>> virtio-net: extract virtqueue coalescig cmd for reuse >>>> virtio-net: support rx netdim >>>> virtio-net: support tx netdim >>>> >>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >>>> 1 file changed, 322 insertions(+), 72 deletions(-) >>>> >>>> -- >>>> 2.19.1.6.gb485710b >>>> >>>> >>
在 2023/10/25 下午1:49, Michael S. Tsirkin 写道: > On Thu, Oct 12, 2023 at 03:44:04PM +0800, Heng Qi wrote: >> Now, virtio-net already supports per-queue moderation parameter >> setting. Based on this, we use the netdim library of linux to support >> dynamic coalescing moderation for virtio-net. >> >> Due to hardware scheduling issues, we only tested rx dim. > So patches 1 to 4 look ok but patch 5 is untested - we should > probably wait until it's tested properly. Hi, Michael. For a few reasons (reply to Jason's thread), I won't be trying to push tx dim any more in the short term. Please review the remaining patches. Thanks a lot! > > >> @Test env >> rxq0 has affinity to cpu0. >> >> @Test cmd >> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >> server: taskset -c 0 sockperf sr --tcp >> >> @Test res >> The second column is the ratio of the result returned by client >> when rx dim is enabled to the result returned by client when >> rx dim is disabled. >> -------------------------------------- >> | msg_size | rx_dim=on / rx_dim=off | >> -------------------------------------- >> | 14B | + 3% | >> -------------------------------------- >> | 100B | + 16% | >> -------------------------------------- >> | 500B | + 25% | >> -------------------------------------- >> | 1400B | + 28% | >> -------------------------------------- >> | 2048B | + 22% | >> -------------------------------------- >> | 4096B | + 5% | >> -------------------------------------- >> >> --- >> This patch set was part of the previous netdim patch set[1]. >> [1] was split into a merged bugfix set[2] and the current set. >> The previous relevant commentators have been Cced. >> >> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >> >> Heng Qi (5): >> virtio-net: returns whether napi is complete >> virtio-net: separate rx/tx coalescing moderation cmds >> virtio-net: extract virtqueue coalescig cmd for reuse >> virtio-net: support rx netdim >> virtio-net: support tx netdim >> >> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >> 1 file changed, 322 insertions(+), 72 deletions(-) >> >> -- >> 2.19.1.6.gb485710b
On Wed, Nov 01, 2023 at 05:40:30PM +0800, Heng Qi wrote: > > > 在 2023/10/25 下午1:49, Michael S. Tsirkin 写道: > > On Thu, Oct 12, 2023 at 03:44:04PM +0800, Heng Qi wrote: > > > Now, virtio-net already supports per-queue moderation parameter > > > setting. Based on this, we use the netdim library of linux to support > > > dynamic coalescing moderation for virtio-net. > > > > > > Due to hardware scheduling issues, we only tested rx dim. > > So patches 1 to 4 look ok but patch 5 is untested - we should > > probably wait until it's tested properly. > > Hi, Michael. > > For a few reasons (reply to Jason's thread), I won't be trying to push tx > dim any more in the short term. > > Please review the remaining patches. > > Thanks a lot! You got a bunch of comments from Jason - want to address them in a new version then, and I'll review that? > > > > > > > @Test env > > > rxq0 has affinity to cpu0. > > > > > > @Test cmd > > > client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > > > server: taskset -c 0 sockperf sr --tcp > > > > > > @Test res > > > The second column is the ratio of the result returned by client > > > when rx dim is enabled to the result returned by client when > > > rx dim is disabled. > > > -------------------------------------- > > > | msg_size | rx_dim=on / rx_dim=off | > > > -------------------------------------- > > > | 14B | + 3% | > > > -------------------------------------- > > > | 100B | + 16% | > > > -------------------------------------- > > > | 500B | + 25% | > > > -------------------------------------- > > > | 1400B | + 28% | > > > -------------------------------------- > > > | 2048B | + 22% | > > > -------------------------------------- > > > | 4096B | + 5% | > > > -------------------------------------- > > > > > > --- > > > This patch set was part of the previous netdim patch set[1]. > > > [1] was split into a merged bugfix set[2] and the current set. > > > The previous relevant commentators have been Cced. > > > > > > [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > > > [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > > > > > > Heng Qi (5): > > > virtio-net: returns whether napi is complete > > > virtio-net: separate rx/tx coalescing moderation cmds > > > virtio-net: extract virtqueue coalescig cmd for reuse > > > virtio-net: support rx netdim > > > virtio-net: support tx netdim > > > > > > drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > > > 1 file changed, 322 insertions(+), 72 deletions(-) > > > > > > -- > > > 2.19.1.6.gb485710b
在 2023/11/1 下午6:44, Michael S. Tsirkin 写道: > On Wed, Nov 01, 2023 at 05:40:30PM +0800, Heng Qi wrote: >> >> 在 2023/10/25 下午1:49, Michael S. Tsirkin 写道: >>> On Thu, Oct 12, 2023 at 03:44:04PM +0800, Heng Qi wrote: >>>> Now, virtio-net already supports per-queue moderation parameter >>>> setting. Based on this, we use the netdim library of linux to support >>>> dynamic coalescing moderation for virtio-net. >>>> >>>> Due to hardware scheduling issues, we only tested rx dim. >>> So patches 1 to 4 look ok but patch 5 is untested - we should >>> probably wait until it's tested properly. >> Hi, Michael. >> >> For a few reasons (reply to Jason's thread), I won't be trying to push tx >> dim any more in the short term. >> >> Please review the remaining patches. >> >> Thanks a lot! > > You got a bunch of comments from Jason - want to address them > in a new version then, and I'll review that? Certainly! I have replied to Jason's comments, and will push the next version once he acks my reply. Thanks! > >>> >>>> @Test env >>>> rxq0 has affinity to cpu0. >>>> >>>> @Test cmd >>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >>>> server: taskset -c 0 sockperf sr --tcp >>>> >>>> @Test res >>>> The second column is the ratio of the result returned by client >>>> when rx dim is enabled to the result returned by client when >>>> rx dim is disabled. >>>> -------------------------------------- >>>> | msg_size | rx_dim=on / rx_dim=off | >>>> -------------------------------------- >>>> | 14B | + 3% | >>>> -------------------------------------- >>>> | 100B | + 16% | >>>> -------------------------------------- >>>> | 500B | + 25% | >>>> -------------------------------------- >>>> | 1400B | + 28% | >>>> -------------------------------------- >>>> | 2048B | + 22% | >>>> -------------------------------------- >>>> | 4096B | + 5% | >>>> -------------------------------------- >>>> >>>> --- >>>> This patch set was part of the previous netdim patch set[1]. >>>> [1] was split into a merged bugfix set[2] and the current set. >>>> The previous relevant commentators have been Cced. >>>> >>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >>>> >>>> Heng Qi (5): >>>> virtio-net: returns whether napi is complete >>>> virtio-net: separate rx/tx coalescing moderation cmds >>>> virtio-net: extract virtqueue coalescig cmd for reuse >>>> virtio-net: support rx netdim >>>> virtio-net: support tx netdim >>>> >>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >>>> 1 file changed, 322 insertions(+), 72 deletions(-) >>>> >>>> -- >>>> 2.19.1.6.gb485710b
在 2023/10/25 下午1:53, Michael S. Tsirkin 写道: > On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote: >> On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>> >>> >>> 在 2023/10/12 下午4:29, Jason Wang 写道: >>>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>>>> Now, virtio-net already supports per-queue moderation parameter >>>>> setting. Based on this, we use the netdim library of linux to support >>>>> dynamic coalescing moderation for virtio-net. >>>>> >>>>> Due to hardware scheduling issues, we only tested rx dim. >>>> Do you have PPS numbers? And TX numbers are also important as the >>>> throughput could be misleading due to various reasons. >>> Hi Jason! >>> >>> The comparison of rx netdim performance is as follows: >>> (the backend supporting tx dim is not yet ready) >> Thanks a lot for the numbers. >> >> I'd still expect the TX result as I did play tx interrupt coalescing >> about 10 years ago. >> >> I will start to review the series but let's try to have some TX numbers as well. >> >> Btw, it would be more convenient to have a raw PPS benchmark. E.g you >> can try to use a software or hardware packet generator. >> >> Thanks > Latency results are also kind of interesting. I test the latency using sockperf pp: @Rx cmd taskset -c 0 sockperf sr -p 8989 @Tx cmd taskset -c 0 sockperf pp -i ${ip} -p 8989 -t 10 After running this cmd 5 times and averaging the results, we get the following data: dim off: 17.7735 usec dim on: 18.0110 usec Thanks! > > >>> >>> I. Sockperf UDP >>> ================================================= >>> 1. Env >>> rxq_0 is affinity to cpu_0 >>> >>> 2. Cmd >>> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B >>> server: taskset -c 0 sockperf sr -p 8989 >>> >>> 3. Result >>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. >>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. >>> ================================================= >>> >>> >>> II. Redis >>> ================================================= >>> 1. Env >>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>> >>> 2. Result >>> When all cpus are 100%, ops/sec of memtier_benchmark client is >>> dim off: 978437.23 >>> dim on: 1143638.28 >>> ================================================= >>> >>> >>> III. Nginx >>> ================================================= >>> 1. Env >>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>> >>> 2. Result >>> When all cpus are 100%, requests/sec of wrk client is >>> dim off: 877931.67 >>> dim on: 1019160.31 >>> ================================================= >>> >>> Thanks! >>> >>>> Thanks >>>> >>>>> @Test env >>>>> rxq0 has affinity to cpu0. >>>>> >>>>> @Test cmd >>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >>>>> server: taskset -c 0 sockperf sr --tcp >>>>> >>>>> @Test res >>>>> The second column is the ratio of the result returned by client >>>>> when rx dim is enabled to the result returned by client when >>>>> rx dim is disabled. >>>>> -------------------------------------- >>>>> | msg_size | rx_dim=on / rx_dim=off | >>>>> -------------------------------------- >>>>> | 14B | + 3% | >>>>> -------------------------------------- >>>>> | 100B | + 16% | >>>>> -------------------------------------- >>>>> | 500B | + 25% | >>>>> -------------------------------------- >>>>> | 1400B | + 28% | >>>>> -------------------------------------- >>>>> | 2048B | + 22% | >>>>> -------------------------------------- >>>>> | 4096B | + 5% | >>>>> -------------------------------------- >>>>> >>>>> --- >>>>> This patch set was part of the previous netdim patch set[1]. >>>>> [1] was split into a merged bugfix set[2] and the current set. >>>>> The previous relevant commentators have been Cced. >>>>> >>>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >>>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >>>>> >>>>> Heng Qi (5): >>>>> virtio-net: returns whether napi is complete >>>>> virtio-net: separate rx/tx coalescing moderation cmds >>>>> virtio-net: extract virtqueue coalescig cmd for reuse >>>>> virtio-net: support rx netdim >>>>> virtio-net: support tx netdim >>>>> >>>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >>>>> 1 file changed, 322 insertions(+), 72 deletions(-) >>>>> >>>>> -- >>>>> 2.19.1.6.gb485710b >>>>> >>>>> >>>
On Wed, Nov 1, 2023 at 7:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > > > 在 2023/10/25 下午1:53, Michael S. Tsirkin 写道: > > On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote: > >> On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > >>> > >>> > >>> 在 2023/10/12 下午4:29, Jason Wang 写道: > >>>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > >>>>> Now, virtio-net already supports per-queue moderation parameter > >>>>> setting. Based on this, we use the netdim library of linux to support > >>>>> dynamic coalescing moderation for virtio-net. > >>>>> > >>>>> Due to hardware scheduling issues, we only tested rx dim. > >>>> Do you have PPS numbers? And TX numbers are also important as the > >>>> throughput could be misleading due to various reasons. > >>> Hi Jason! > >>> > >>> The comparison of rx netdim performance is as follows: > >>> (the backend supporting tx dim is not yet ready) > >> Thanks a lot for the numbers. > >> > >> I'd still expect the TX result as I did play tx interrupt coalescing > >> about 10 years ago. > >> > >> I will start to review the series but let's try to have some TX numbers as well. > >> > >> Btw, it would be more convenient to have a raw PPS benchmark. E.g you > >> can try to use a software or hardware packet generator. > >> > >> Thanks > > Latency results are also kind of interesting. > > I test the latency using sockperf pp: > > @Rx cmd > taskset -c 0 sockperf sr -p 8989 > > @Tx cmd > taskset -c 0 sockperf pp -i ${ip} -p 8989 -t 10 > > After running this cmd 5 times and averaging the results, > we get the following data: > > dim off: 17.7735 usec > dim on: 18.0110 usec Let's add those numbers to the changelog of the next version. Thanks > > Thanks! > > > > > > >>> > >>> I. Sockperf UDP > >>> ================================================= > >>> 1. Env > >>> rxq_0 is affinity to cpu_0 > >>> > >>> 2. Cmd > >>> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > >>> server: taskset -c 0 sockperf sr -p 8989 > >>> > >>> 3. Result > >>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > >>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > >>> ================================================= > >>> > >>> > >>> II. Redis > >>> ================================================= > >>> 1. Env > >>> There are 8 rxqs and rxq_i is affinity to cpu_i. > >>> > >>> 2. Result > >>> When all cpus are 100%, ops/sec of memtier_benchmark client is > >>> dim off: 978437.23 > >>> dim on: 1143638.28 > >>> ================================================= > >>> > >>> > >>> III. Nginx > >>> ================================================= > >>> 1. Env > >>> There are 8 rxqs and rxq_i is affinity to cpu_i. > >>> > >>> 2. Result > >>> When all cpus are 100%, requests/sec of wrk client is > >>> dim off: 877931.67 > >>> dim on: 1019160.31 > >>> ================================================= > >>> > >>> Thanks! > >>> > >>>> Thanks > >>>> > >>>>> @Test env > >>>>> rxq0 has affinity to cpu0. > >>>>> > >>>>> @Test cmd > >>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > >>>>> server: taskset -c 0 sockperf sr --tcp > >>>>> > >>>>> @Test res > >>>>> The second column is the ratio of the result returned by client > >>>>> when rx dim is enabled to the result returned by client when > >>>>> rx dim is disabled. > >>>>> -------------------------------------- > >>>>> | msg_size | rx_dim=on / rx_dim=off | > >>>>> -------------------------------------- > >>>>> | 14B | + 3% | > >>>>> -------------------------------------- > >>>>> | 100B | + 16% | > >>>>> -------------------------------------- > >>>>> | 500B | + 25% | > >>>>> -------------------------------------- > >>>>> | 1400B | + 28% | > >>>>> -------------------------------------- > >>>>> | 2048B | + 22% | > >>>>> -------------------------------------- > >>>>> | 4096B | + 5% | > >>>>> -------------------------------------- > >>>>> > >>>>> --- > >>>>> This patch set was part of the previous netdim patch set[1]. > >>>>> [1] was split into a merged bugfix set[2] and the current set. > >>>>> The previous relevant commentators have been Cced. > >>>>> > >>>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > >>>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > >>>>> > >>>>> Heng Qi (5): > >>>>> virtio-net: returns whether napi is complete > >>>>> virtio-net: separate rx/tx coalescing moderation cmds > >>>>> virtio-net: extract virtqueue coalescig cmd for reuse > >>>>> virtio-net: support rx netdim > >>>>> virtio-net: support tx netdim > >>>>> > >>>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > >>>>> 1 file changed, 322 insertions(+), 72 deletions(-) > >>>>> > >>>>> -- > >>>>> 2.19.1.6.gb485710b > >>>>> > >>>>> > >>> >
On Wed, Nov 1, 2023 at 5:38 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > > > > 在 2023/10/25 上午9:18, Jason Wang 写道: > > On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > >> > >> > >> 在 2023/10/12 下午4:29, Jason Wang 写道: > >>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: > >>>> Now, virtio-net already supports per-queue moderation parameter > >>>> setting. Based on this, we use the netdim library of linux to support > >>>> dynamic coalescing moderation for virtio-net. > >>>> > >>>> Due to hardware scheduling issues, we only tested rx dim. > >>> Do you have PPS numbers? And TX numbers are also important as the > >>> throughput could be misleading due to various reasons. > >> Hi Jason! > >> > >> The comparison of rx netdim performance is as follows: > >> (the backend supporting tx dim is not yet ready) > > Thanks a lot for the numbers. > > > > I'd still expect the TX result as I did play tx interrupt coalescing > > Hi, Jason. > > Sorry for the late reply to this! Our team has been blocked by other > priorities the past few days. > > For tx dim, we have a fixed empirical value internally. > This value performs better overall than manually adjusting the tx timer > register --> > I'll do not have tx numbers. :( So in the short term I no longer try to > push [5/5] > patch for tx dim and try to return -EOPNOTSUPP for it, sorry for this. > > > about 10 years ago. > > > > I will start to review the series but let's try to have some TX numbers as well. > > > > Btw, it would be more convenient to have a raw PPS benchmark. E.g you > > I got some raw pps data using pktgen from linux/sample/pktgen: > > 1. tx cmd > ./pktgen_sample02_multiqueue.sh -i eth1 -s 44 -d ${dst_ip} -m ${dst_mac} > -t 8 -f 0 -n 0 > > This uses 8 kpktgend threads to inject data into eth1. > > 2. Rx side loads a simple xdp prog which drops all received udp packets. > > 3. Data > pps: ~1000w For "w" did you mean 10 million? Looks too huge to me? > rx dim off: cpu idle= ~35% > rx dim on: cpu idle= ~76% This looks promising. Thanks > > Thanks! > > > can try to use a software or hardware packet generator. > > > > Thanks > > > >> > >> I. Sockperf UDP > >> ================================================= > >> 1. Env > >> rxq_0 is affinity to cpu_0 > >> > >> 2. Cmd > >> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B > >> server: taskset -c 0 sockperf sr -p 8989 > >> > >> 3. Result > >> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. > >> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. > >> ================================================= > >> > >> > >> II. Redis > >> ================================================= > >> 1. Env > >> There are 8 rxqs and rxq_i is affinity to cpu_i. > >> > >> 2. Result > >> When all cpus are 100%, ops/sec of memtier_benchmark client is > >> dim off: 978437.23 > >> dim on: 1143638.28 > >> ================================================= > >> > >> > >> III. Nginx > >> ================================================= > >> 1. Env > >> There are 8 rxqs and rxq_i is affinity to cpu_i. > >> > >> 2. Result > >> When all cpus are 100%, requests/sec of wrk client is > >> dim off: 877931.67 > >> dim on: 1019160.31 > >> ================================================= > >> > >> Thanks! > >> > >>> Thanks > >>> > >>>> @Test env > >>>> rxq0 has affinity to cpu0. > >>>> > >>>> @Test cmd > >>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} > >>>> server: taskset -c 0 sockperf sr --tcp > >>>> > >>>> @Test res > >>>> The second column is the ratio of the result returned by client > >>>> when rx dim is enabled to the result returned by client when > >>>> rx dim is disabled. > >>>> -------------------------------------- > >>>> | msg_size | rx_dim=on / rx_dim=off | > >>>> -------------------------------------- > >>>> | 14B | + 3% | > >>>> -------------------------------------- > >>>> | 100B | + 16% | > >>>> -------------------------------------- > >>>> | 500B | + 25% | > >>>> -------------------------------------- > >>>> | 1400B | + 28% | > >>>> -------------------------------------- > >>>> | 2048B | + 22% | > >>>> -------------------------------------- > >>>> | 4096B | + 5% | > >>>> -------------------------------------- > >>>> > >>>> --- > >>>> This patch set was part of the previous netdim patch set[1]. > >>>> [1] was split into a merged bugfix set[2] and the current set. > >>>> The previous relevant commentators have been Cced. > >>>> > >>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ > >>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ > >>>> > >>>> Heng Qi (5): > >>>> virtio-net: returns whether napi is complete > >>>> virtio-net: separate rx/tx coalescing moderation cmds > >>>> virtio-net: extract virtqueue coalescig cmd for reuse > >>>> virtio-net: support rx netdim > >>>> virtio-net: support tx netdim > >>>> > >>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- > >>>> 1 file changed, 322 insertions(+), 72 deletions(-) > >>>> > >>>> -- > >>>> 2.19.1.6.gb485710b > >>>> > >>>> > >> >
在 2023/11/2 下午12:33, Jason Wang 写道: > On Wed, Nov 1, 2023 at 7:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >> >> >> 在 2023/10/25 下午1:53, Michael S. Tsirkin 写道: >>> On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote: >>>> On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>>>> >>>>> 在 2023/10/12 下午4:29, Jason Wang 写道: >>>>>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>>>>>> Now, virtio-net already supports per-queue moderation parameter >>>>>>> setting. Based on this, we use the netdim library of linux to support >>>>>>> dynamic coalescing moderation for virtio-net. >>>>>>> >>>>>>> Due to hardware scheduling issues, we only tested rx dim. >>>>>> Do you have PPS numbers? And TX numbers are also important as the >>>>>> throughput could be misleading due to various reasons. >>>>> Hi Jason! >>>>> >>>>> The comparison of rx netdim performance is as follows: >>>>> (the backend supporting tx dim is not yet ready) >>>> Thanks a lot for the numbers. >>>> >>>> I'd still expect the TX result as I did play tx interrupt coalescing >>>> about 10 years ago. >>>> >>>> I will start to review the series but let's try to have some TX numbers as well. >>>> >>>> Btw, it would be more convenient to have a raw PPS benchmark. E.g you >>>> can try to use a software or hardware packet generator. >>>> >>>> Thanks >>> Latency results are also kind of interesting. >> I test the latency using sockperf pp: >> >> @Rx cmd >> taskset -c 0 sockperf sr -p 8989 >> >> @Tx cmd >> taskset -c 0 sockperf pp -i ${ip} -p 8989 -t 10 >> >> After running this cmd 5 times and averaging the results, >> we get the following data: >> >> dim off: 17.7735 usec >> dim on: 18.0110 usec > Let's add those numbers to the changelog of the next version. Ok. Thanks! > > Thanks > >> Thanks! >> >>> >>>>> I. Sockperf UDP >>>>> ================================================= >>>>> 1. Env >>>>> rxq_0 is affinity to cpu_0 >>>>> >>>>> 2. Cmd >>>>> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B >>>>> server: taskset -c 0 sockperf sr -p 8989 >>>>> >>>>> 3. Result >>>>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. >>>>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. >>>>> ================================================= >>>>> >>>>> >>>>> II. Redis >>>>> ================================================= >>>>> 1. Env >>>>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>>>> >>>>> 2. Result >>>>> When all cpus are 100%, ops/sec of memtier_benchmark client is >>>>> dim off: 978437.23 >>>>> dim on: 1143638.28 >>>>> ================================================= >>>>> >>>>> >>>>> III. Nginx >>>>> ================================================= >>>>> 1. Env >>>>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>>>> >>>>> 2. Result >>>>> When all cpus are 100%, requests/sec of wrk client is >>>>> dim off: 877931.67 >>>>> dim on: 1019160.31 >>>>> ================================================= >>>>> >>>>> Thanks! >>>>> >>>>>> Thanks >>>>>> >>>>>>> @Test env >>>>>>> rxq0 has affinity to cpu0. >>>>>>> >>>>>>> @Test cmd >>>>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >>>>>>> server: taskset -c 0 sockperf sr --tcp >>>>>>> >>>>>>> @Test res >>>>>>> The second column is the ratio of the result returned by client >>>>>>> when rx dim is enabled to the result returned by client when >>>>>>> rx dim is disabled. >>>>>>> -------------------------------------- >>>>>>> | msg_size | rx_dim=on / rx_dim=off | >>>>>>> -------------------------------------- >>>>>>> | 14B | + 3% | >>>>>>> -------------------------------------- >>>>>>> | 100B | + 16% | >>>>>>> -------------------------------------- >>>>>>> | 500B | + 25% | >>>>>>> -------------------------------------- >>>>>>> | 1400B | + 28% | >>>>>>> -------------------------------------- >>>>>>> | 2048B | + 22% | >>>>>>> -------------------------------------- >>>>>>> | 4096B | + 5% | >>>>>>> -------------------------------------- >>>>>>> >>>>>>> --- >>>>>>> This patch set was part of the previous netdim patch set[1]. >>>>>>> [1] was split into a merged bugfix set[2] and the current set. >>>>>>> The previous relevant commentators have been Cced. >>>>>>> >>>>>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >>>>>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >>>>>>> >>>>>>> Heng Qi (5): >>>>>>> virtio-net: returns whether napi is complete >>>>>>> virtio-net: separate rx/tx coalescing moderation cmds >>>>>>> virtio-net: extract virtqueue coalescig cmd for reuse >>>>>>> virtio-net: support rx netdim >>>>>>> virtio-net: support tx netdim >>>>>>> >>>>>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >>>>>>> 1 file changed, 322 insertions(+), 72 deletions(-) >>>>>>> >>>>>>> -- >>>>>>> 2.19.1.6.gb485710b >>>>>>> >>>>>>>
在 2023/11/2 下午12:34, Jason Wang 写道: > On Wed, Nov 1, 2023 at 5:38 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >> >> >> 在 2023/10/25 上午9:18, Jason Wang 写道: >>> On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>>> >>>> 在 2023/10/12 下午4:29, Jason Wang 写道: >>>>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>>>>> Now, virtio-net already supports per-queue moderation parameter >>>>>> setting. Based on this, we use the netdim library of linux to support >>>>>> dynamic coalescing moderation for virtio-net. >>>>>> >>>>>> Due to hardware scheduling issues, we only tested rx dim. >>>>> Do you have PPS numbers? And TX numbers are also important as the >>>>> throughput could be misleading due to various reasons. >>>> Hi Jason! >>>> >>>> The comparison of rx netdim performance is as follows: >>>> (the backend supporting tx dim is not yet ready) >>> Thanks a lot for the numbers. >>> >>> I'd still expect the TX result as I did play tx interrupt coalescing >> Hi, Jason. >> >> Sorry for the late reply to this! Our team has been blocked by other >> priorities the past few days. >> >> For tx dim, we have a fixed empirical value internally. >> This value performs better overall than manually adjusting the tx timer >> register --> >> I'll do not have tx numbers. :( So in the short term I no longer try to >> push [5/5] >> patch for tx dim and try to return -EOPNOTSUPP for it, sorry for this. >> >>> about 10 years ago. >>> >>> I will start to review the series but let's try to have some TX numbers as well. >>> >>> Btw, it would be more convenient to have a raw PPS benchmark. E.g you >> I got some raw pps data using pktgen from linux/sample/pktgen: >> >> 1. tx cmd >> ./pktgen_sample02_multiqueue.sh -i eth1 -s 44 -d ${dst_ip} -m ${dst_mac} >> -t 8 -f 0 -n 0 >> >> This uses 8 kpktgend threads to inject data into eth1. >> >> 2. Rx side loads a simple xdp prog which drops all received udp packets. >> >> 3. Data >> pps: ~1000w > For "w" did you mean 10 million? Looks too huge to me? Yes, all cpus in tx are 100% sys, rx uses xdp to drop all received udp packets. Then this means rx receiving ability is strong. If there was no xdp in rx, I remember tx sent 10million pps, but rx could only receive 7.3+ million pps. Thanks! > >> rx dim off: cpu idle= ~35% >> rx dim on: cpu idle= ~76% > This looks promising. > > Thanks > >> Thanks! >> >>> can try to use a software or hardware packet generator. >>> >>> Thanks >>> >>>> I. Sockperf UDP >>>> ================================================= >>>> 1. Env >>>> rxq_0 is affinity to cpu_0 >>>> >>>> 2. Cmd >>>> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B >>>> server: taskset -c 0 sockperf sr -p 8989 >>>> >>>> 3. Result >>>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. >>>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. >>>> ================================================= >>>> >>>> >>>> II. Redis >>>> ================================================= >>>> 1. Env >>>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>>> >>>> 2. Result >>>> When all cpus are 100%, ops/sec of memtier_benchmark client is >>>> dim off: 978437.23 >>>> dim on: 1143638.28 >>>> ================================================= >>>> >>>> >>>> III. Nginx >>>> ================================================= >>>> 1. Env >>>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>>> >>>> 2. Result >>>> When all cpus are 100%, requests/sec of wrk client is >>>> dim off: 877931.67 >>>> dim on: 1019160.31 >>>> ================================================= >>>> >>>> Thanks! >>>> >>>>> Thanks >>>>> >>>>>> @Test env >>>>>> rxq0 has affinity to cpu0. >>>>>> >>>>>> @Test cmd >>>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} >>>>>> server: taskset -c 0 sockperf sr --tcp >>>>>> >>>>>> @Test res >>>>>> The second column is the ratio of the result returned by client >>>>>> when rx dim is enabled to the result returned by client when >>>>>> rx dim is disabled. >>>>>> -------------------------------------- >>>>>> | msg_size | rx_dim=on / rx_dim=off | >>>>>> -------------------------------------- >>>>>> | 14B | + 3% | >>>>>> -------------------------------------- >>>>>> | 100B | + 16% | >>>>>> -------------------------------------- >>>>>> | 500B | + 25% | >>>>>> -------------------------------------- >>>>>> | 1400B | + 28% | >>>>>> -------------------------------------- >>>>>> | 2048B | + 22% | >>>>>> -------------------------------------- >>>>>> | 4096B | + 5% | >>>>>> -------------------------------------- >>>>>> >>>>>> --- >>>>>> This patch set was part of the previous netdim patch set[1]. >>>>>> [1] was split into a merged bugfix set[2] and the current set. >>>>>> The previous relevant commentators have been Cced. >>>>>> >>>>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >>>>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >>>>>> >>>>>> Heng Qi (5): >>>>>> virtio-net: returns whether napi is complete >>>>>> virtio-net: separate rx/tx coalescing moderation cmds >>>>>> virtio-net: extract virtqueue coalescig cmd for reuse >>>>>> virtio-net: support rx netdim >>>>>> virtio-net: support tx netdim >>>>>> >>>>>> drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- >>>>>> 1 file changed, 322 insertions(+), 72 deletions(-) >>>>>> >>>>>> -- >>>>>> 2.19.1.6.gb485710b >>>>>> >>>>>>
在 2023/11/2 下午12:51, Heng Qi 写道: > > > 在 2023/11/2 下午12:34, Jason Wang 写道: >> On Wed, Nov 1, 2023 at 5:38 PM Heng Qi <hengqi@linux.alibaba.com> wrote: >>> >>> >>> 在 2023/10/25 上午9:18, Jason Wang 写道: >>>> On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@linux.alibaba.com> >>>> wrote: >>>>> >>>>> 在 2023/10/12 下午4:29, Jason Wang 写道: >>>>>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi >>>>>> <hengqi@linux.alibaba.com> wrote: >>>>>>> Now, virtio-net already supports per-queue moderation parameter >>>>>>> setting. Based on this, we use the netdim library of linux to >>>>>>> support >>>>>>> dynamic coalescing moderation for virtio-net. >>>>>>> >>>>>>> Due to hardware scheduling issues, we only tested rx dim. >>>>>> Do you have PPS numbers? And TX numbers are also important as the >>>>>> throughput could be misleading due to various reasons. >>>>> Hi Jason! >>>>> >>>>> The comparison of rx netdim performance is as follows: >>>>> (the backend supporting tx dim is not yet ready) >>>> Thanks a lot for the numbers. >>>> >>>> I'd still expect the TX result as I did play tx interrupt coalescing >>> Hi, Jason. >>> >>> Sorry for the late reply to this! Our team has been blocked by other >>> priorities the past few days. >>> >>> For tx dim, we have a fixed empirical value internally. >>> This value performs better overall than manually adjusting the tx timer >>> register --> >>> I'll do not have tx numbers. :( So in the short term I no longer try to >>> push [5/5] >>> patch for tx dim and try to return -EOPNOTSUPP for it, sorry for this. >>> >>>> about 10 years ago. >>>> >>>> I will start to review the series but let's try to have some TX >>>> numbers as well. >>>> >>>> Btw, it would be more convenient to have a raw PPS benchmark. E.g you >>> I got some raw pps data using pktgen from linux/sample/pktgen: >>> >>> 1. tx cmd >>> ./pktgen_sample02_multiqueue.sh -i eth1 -s 44 -d ${dst_ip} -m >>> ${dst_mac} >>> -t 8 -f 0 -n 0 >>> >>> This uses 8 kpktgend threads to inject data into eth1. >>> >>> 2. Rx side loads a simple xdp prog which drops all received udp >>> packets. >>> >>> 3. Data >>> pps: ~1000w >> For "w" did you mean 10 million? Looks too huge to me? > > Yes, all cpus in tx are 100% sys, rx uses xdp to drop all received udp > packets. > Then this means rx receiving ability is strong. > > If there was no xdp in rx, I remember tx sent 10million pps, but rx > could only receive 7.3+ million pps. In addition, in the test environment of pktgen, rx has 8 cpu and 8 queue. Thanks! > > Thanks! > >> >>> rx dim off: cpu idle= ~35% >>> rx dim on: cpu idle= ~76% >> This looks promising. >> >> Thanks >> >>> Thanks! >>> >>>> can try to use a software or hardware packet generator. >>>> >>>> Thanks >>>> >>>>> I. Sockperf UDP >>>>> ================================================= >>>>> 1. Env >>>>> rxq_0 is affinity to cpu_0 >>>>> >>>>> 2. Cmd >>>>> client: taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B >>>>> server: taskset -c 0 sockperf sr -p 8989 >>>>> >>>>> 3. Result >>>>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%. >>>>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%. >>>>> ================================================= >>>>> >>>>> >>>>> II. Redis >>>>> ================================================= >>>>> 1. Env >>>>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>>>> >>>>> 2. Result >>>>> When all cpus are 100%, ops/sec of memtier_benchmark client is >>>>> dim off: 978437.23 >>>>> dim on: 1143638.28 >>>>> ================================================= >>>>> >>>>> >>>>> III. Nginx >>>>> ================================================= >>>>> 1. Env >>>>> There are 8 rxqs and rxq_i is affinity to cpu_i. >>>>> >>>>> 2. Result >>>>> When all cpus are 100%, requests/sec of wrk client is >>>>> dim off: 877931.67 >>>>> dim on: 1019160.31 >>>>> ================================================= >>>>> >>>>> Thanks! >>>>> >>>>>> Thanks >>>>>> >>>>>>> @Test env >>>>>>> rxq0 has affinity to cpu0. >>>>>>> >>>>>>> @Test cmd >>>>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m >>>>>>> ${msg_size} >>>>>>> server: taskset -c 0 sockperf sr --tcp >>>>>>> >>>>>>> @Test res >>>>>>> The second column is the ratio of the result returned by client >>>>>>> when rx dim is enabled to the result returned by client when >>>>>>> rx dim is disabled. >>>>>>> -------------------------------------- >>>>>>> | msg_size | rx_dim=on / rx_dim=off | >>>>>>> -------------------------------------- >>>>>>> | 14B | + 3% | >>>>>>> -------------------------------------- >>>>>>> | 100B | + 16% | >>>>>>> -------------------------------------- >>>>>>> | 500B | + 25% | >>>>>>> -------------------------------------- >>>>>>> | 1400B | + 28% | >>>>>>> -------------------------------------- >>>>>>> | 2048B | + 22% | >>>>>>> -------------------------------------- >>>>>>> | 4096B | + 5% | >>>>>>> -------------------------------------- >>>>>>> >>>>>>> --- >>>>>>> This patch set was part of the previous netdim patch set[1]. >>>>>>> [1] was split into a merged bugfix set[2] and the current set. >>>>>>> The previous relevant commentators have been Cced. >>>>>>> >>>>>>> [1] >>>>>>> https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ >>>>>>> [2] >>>>>>> https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ >>>>>>> >>>>>>> Heng Qi (5): >>>>>>> virtio-net: returns whether napi is complete >>>>>>> virtio-net: separate rx/tx coalescing moderation cmds >>>>>>> virtio-net: extract virtqueue coalescig cmd for reuse >>>>>>> virtio-net: support rx netdim >>>>>>> virtio-net: support tx netdim >>>>>>> >>>>>>> drivers/net/virtio_net.c | 394 >>>>>>> ++++++++++++++++++++++++++++++++------- >>>>>>> 1 file changed, 322 insertions(+), 72 deletions(-) >>>>>>> >>>>>>> -- >>>>>>> 2.19.1.6.gb485710b >>>>>>> >>>>>>> >
Now, virtio-net already supports per-queue moderation parameter setting. Based on this, we use the netdim library of linux to support dynamic coalescing moderation for virtio-net. Due to hardware scheduling issues, we only tested rx dim. @Test env rxq0 has affinity to cpu0. @Test cmd client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size} server: taskset -c 0 sockperf sr --tcp @Test res The second column is the ratio of the result returned by client when rx dim is enabled to the result returned by client when rx dim is disabled. -------------------------------------- | msg_size | rx_dim=on / rx_dim=off | -------------------------------------- | 14B | + 3% | -------------------------------------- | 100B | + 16% | -------------------------------------- | 500B | + 25% | -------------------------------------- | 1400B | + 28% | -------------------------------------- | 2048B | + 22% | -------------------------------------- | 4096B | + 5% | -------------------------------------- --- This patch set was part of the previous netdim patch set[1]. [1] was split into a merged bugfix set[2] and the current set. The previous relevant commentators have been Cced. [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/ [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/ Heng Qi (5): virtio-net: returns whether napi is complete virtio-net: separate rx/tx coalescing moderation cmds virtio-net: extract virtqueue coalescig cmd for reuse virtio-net: support rx netdim virtio-net: support tx netdim drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++------- 1 file changed, 322 insertions(+), 72 deletions(-)