[v3,0/3] QIOChannel async_write & async_flush + MSG_ZEROCOPY + multifd

Message

Leonardo Bras Sept. 22, 2021, 10:24 p.m. UTC

This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make
use of it for multifd migration performance improvement.

Patch #1 creates new callbacks for QIOChannel, allowing the implementation
of asynchronous writing.

Patch #2 implements async_write and async_flush on QIOChannelSocket,

Patch #3 Makes use of async_write + async_flush to enable MSG_ZEROCOPY
for migration using multifd nocomp.

Results:
So far, the resource usage of __sys_sendmsg() reduced 15 times, and the
overall migration took 13-18% less time, based in synthetic workload.

The objective is to reduce migration time in hosts with heavy cpu usage.

---
Changes since v2:
- Patch #1: One more fallback
- Patch #2: Fall back to sync if fails to lock buffer memory in MSG_ZEROCOPY send.

Changes since v1:
- Reimplemented the patchset using async_write + async_flush approach.
- Implemented a flush to be able to tell whenever all data was written.

Leonardo Bras (3):
  QIOChannel: Add io_async_writev & io_async_flush callbacks
  QIOChannelSocket: Implement io_async_write & io_async_flush
  multifd: Send using asynchronous write on nocomp to send RAM pages.

 include/io/channel-socket.h |   2 +
 include/io/channel.h        |  94 ++++++++++++++++---
 io/channel-socket.c         | 176 ++++++++++++++++++++++++++++++++++--
 io/channel.c                |  66 +++++++++++---
 migration/multifd.c         |   3 +-
 5 files changed, 300 insertions(+), 41 deletions(-)

Comments

Peter Xu Sept. 28, 2021, 10:50 p.m. UTC | #1

On Wed, Sep 22, 2021 at 07:24:20PM -0300, Leonardo Bras wrote:
> This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make
> use of it for multifd migration performance improvement.
> 
> Patch #1 creates new callbacks for QIOChannel, allowing the implementation
> of asynchronous writing.
> 
> Patch #2 implements async_write and async_flush on QIOChannelSocket,
> 
> Patch #3 Makes use of async_write + async_flush to enable MSG_ZEROCOPY
> for migration using multifd nocomp.
> 
> Results:
> So far, the resource usage of __sys_sendmsg() reduced 15 times, and the
> overall migration took 13-18% less time, based in synthetic workload.
> 
> The objective is to reduce migration time in hosts with heavy cpu usage.

My previous memory is that we'll add a capability bit for migration, so it'll
not be enabled until user specified it.  Plan to do it in the next version?

It'll still be okay if you want to separate the work of (1) qio channel
zero-copy support on sockets, and (2) apply zero copy to migration, then that
can be done in part 2.  Your call. :)

Thanks,

Leonardo Bras Sept. 29, 2021, 6:34 p.m. UTC | #2

Hello Peter,

On Tue, Sep 28, 2021 at 7:51 PM Peter Xu <[e-mail]> wrote:
>
> On Wed, Sep 22, 2021 at 07:24:20PM -0300, Leonardo Bras wrote:
> > This patch series intends to enable MSG_ZEROCOPY in QIOChannel, and make
> > use of it for multifd migration performance improvement.
> >
> > Patch #1 creates new callbacks for QIOChannel, allowing the implementation
> > of asynchronous writing.
> >
> > Patch #2 implements async_write and async_flush on QIOChannelSocket,
> >
> > Patch #3 Makes use of async_write + async_flush to enable MSG_ZEROCOPY
> > for migration using multifd nocomp.
> >
> > Results:
> > So far, the resource usage of __sys_sendmsg() reduced 15 times, and the
> > overall migration took 13-18% less time, based in synthetic workload.
> >
> > The objective is to reduce migration time in hosts with heavy cpu usage.
>
> My previous memory is that we'll add a capability bit for migration, so it'll
> not be enabled until user specified it.  Plan to do it in the next version?

You mean like I did in pre-V1, enabling ZEROCOPY for multifd it in QMP?
Or is this something else?


>
> It'll still be okay if you want to separate the work of (1) qio channel
> zero-copy support on sockets, and (2) apply zero copy to migration, then that
> can be done in part 2.  Your call. :)

Ok  :)

>
> Thanks,
>
> --
> Peter Xu
>

Thank you Peter,

Best regards,
Leonardo

Peter Xu Sept. 29, 2021, 7:22 p.m. UTC | #3

On Wed, Sep 29, 2021 at 03:34:01PM -0300, Leonardo Bras Soares Passos wrote:
> You mean like I did in pre-V1, enabling ZEROCOPY for multifd it in QMP?
> Or is this something else?

I mean any way to still be able to turn zerocopy off? E.g. when the user has no
privilege on mlock.  Thanks,

Leonardo Bras Sept. 29, 2021, 7:48 p.m. UTC | #4

On Wed, Sep 29, 2021 at 4:23 PM Peter Xu <[e-mail]> wrote:
>
> On Wed, Sep 29, 2021 at 03:34:01PM -0300, Leonardo Bras Soares Passos wrote:
> > You mean like I did in pre-V1, enabling ZEROCOPY for multifd it in QMP?
> > Or is this something else?
>
> I mean any way to still be able to turn zerocopy off? E.g. when the user has no
> privilege on mlock.  Thanks,
>
> --
> Peter Xu
>

Yeah, that makes sense in the new approach of failing when zerocopy is
not possible.
I will make sure to do the in v3.

Best regards,
Leonardo