[net-next,0/3] Reduce locks scope of-smc_xxx_lgr_pending

Message

D. Wythe Nov. 13, 2024, 7:14 a.m. UTC

This patch set attempts to optimize the parallelism of SMC-R connections by reducing
locks scope of-smc_xxx_lgr_pending. This is a balance between large-scale refactoring
and performance optimization, where this patch attempts to achieve performance optimization
with as few changes as possible to minimize unexpected impacts.

Although there are still many bottlenecks that affect the connection performance of SMC, 
This also has a certain performance improvement after this patches.

Short-lived conenction wrk/nginx benchmark test:

+--------------+--------+--------+--------+-------+-------+-------+-------+
|conns/qps     |   c8   |  c16   |  c32   |  c64  | c512  | c1024 | c2048 |
+--------------+--------+--------+--------+-------+-------+-------+-------+
|SMC-R before  |10481.84|10761.04|10283.01|9006.88|9140.88|9255.41|7296.20|
+--------------+--------+--------+--------+-------+-------+-------+-------+
|SMC-R origin  |7328.86 |7256.99 |7288.53 |7239.55|5787.56|5371.17|3065.74|
+--------------+--------+--------+--------+-------+-------+-------+-------+

D. Wythe (3):
  net/smc: refactoring lgr pending lock
  net/smc: reduce locks scope of smc_xxx_lgr_pending
  net/smc: Isolate the smc_xxx_lgr_pending with ib_device

 net/smc/af_smc.c   | 36 +++++++++++++++++++-----------------
 net/smc/smc_core.c | 17 +++++++++++++++--
 net/smc/smc_core.h | 29 +++++++++++++++++++++++++++++
 net/smc/smc_ib.c   |  2 ++
 net/smc/smc_ib.h   |  2 ++
 5 files changed, 67 insertions(+), 19 deletions(-)

Comments

Dust Li Nov. 14, 2024, 1:27 a.m. UTC | #1

On 2024-11-13 15:14:02, D. Wythe wrote:
>This patch set attempts to optimize the parallelism of SMC-R connections by reducing
>locks scope of-smc_xxx_lgr_pending. This is a balance between large-scale refactoring
>and performance optimization, where this patch attempts to achieve performance optimization
>with as few changes as possible to minimize unexpected impacts.

I think one of the main benefits you missed here is isolating the lock across
different network namespaces when the RDMA device is exclusive. This
isolation is crucial when managing multiple containers, each with its
own RDMA device.

With the global smc_server_lgr_pending lock, if one container holds this
lock, all other containers must wait.

Worse still, a malicious client could exploit this by forcing the server
to create new link groups according to the SMC-R protocol. In other
words, an attacker could easily compel the server to continuously create
new link groups and hold the smc_server_lgr_pending lock, causing all
SMC connections targeting other containers on the same server to wait.
This scenario effectively constitutes a denial-of-service (DoS) attack.

>
>Although there are still many bottlenecks that affect the connection performance of SMC, 
>This also has a certain performance improvement after this patches.
>
>Short-lived conenction wrk/nginx benchmark test:
>
>+--------------+--------+--------+--------+-------+-------+-------+-------+
>|conns/qps     |   c8   |  c16   |  c32   |  c64  | c512  | c1024 | c2048 |
>+--------------+--------+--------+--------+-------+-------+-------+-------+
>|SMC-R before  |10481.84|10761.04|10283.01|9006.88|9140.88|9255.41|7296.20|
         ^^^ after ?


>+--------------+--------+--------+--------+-------+-------+-------+-------+
>|SMC-R origin  |7328.86 |7256.99 |7288.53 |7239.55|5787.56|5371.17|3065.74|
>+--------------+--------+--------+--------+-------+-------+-------+-------+
>
>D. Wythe (3):
>  net/smc: refactoring lgr pending lock
>  net/smc: reduce locks scope of smc_xxx_lgr_pending
>  net/smc: Isolate the smc_xxx_lgr_pending with ib_device
>
> net/smc/af_smc.c   | 36 +++++++++++++++++++-----------------
> net/smc/smc_core.c | 17 +++++++++++++++--
> net/smc/smc_core.h | 29 +++++++++++++++++++++++++++++
> net/smc/smc_ib.c   |  2 ++
> net/smc/smc_ib.h   |  2 ++
> 5 files changed, 67 insertions(+), 19 deletions(-)
>
>-- 
>2.45.0
>