mbox series

[v16,00/20] Initial support for multi-process Qemu

Message ID cover.1610339529.git.jag.raman@oracle.com (mailing list archive)
Headers show
Series Initial support for multi-process Qemu | expand

Message

Jag Raman Jan. 11, 2021, 5:05 a.m. UTC
Hi

This is the v16 of the patchset. Thank you for your time reviewing v15.

This version has the following changes:

[PATCH v16 04/20] multi-process: Add config option for multi-process QEMU
  - Using “default_feature” value to enable/disable multiprocess

[PATCH v16 07/20] io: add qio_channel_writev_full_all helper
  - Removed local variable in qio_channel_writev_full_all(), setting arguments
    directly
  - Fixed indentation issues
  - Updated commit message

[PATCH v16 08/20] io: add qio_channel_readv_full_all_eof &
                  qio_channel_readv_full_all helpers
  - Added two variants of readv - _full_all_eof & _full_all based on feedback
  - Dropped errno return value
  - Updated commit message
  - Unable to remove local variables and set arguments directly as the
    arguments are later needed for cleanup (g_free/close) during failure

Switched to using OBJECT_DECLARE_{SIMPLE_TYPE, TYPE} macros in the
following patches:
  - [PATCH v16 05/20] multi-process: setup PCI host bridge for remote device
  - [PATCH v16 06/20] multi-process: setup a machine object for remote device
                      process
  - [PATCH v16 11/20] multi-process: Associate fd of a PCIDevice with its object
  - [PATCH v16 13/20] multi-process: introduce proxy object

Updated copyright text to use the year 2021 in the files that show them.

To touch upon the history of this project, we posted the Proof Of Concept
patches before the BoF session in 2018. Subsequently, we have posted 15
versions on the qemu-devel mailing list. You can find them by following
the links below ([1] - [15]). Following people contributed to the design and
implementation of this project:
Jagannathan Raman <jag.raman@oracle.com>
Elena Ufimtseva <elena.ufimtseva@oracle.com>
John G Johnson <john.g.johnson@oracle.com>
Stefan Hajnoczi <stefanha@redhat.com>
Konrad Wilk <konrad.wilk@oracle.com>
Kanth Ghatraju <kanth.ghatraju@oracle.com>

We would like to thank the QEMU community for your feedback in the
design and implementation of this project. Qemu wiki page:
https://wiki.qemu.org/Features/MultiProcessQEMU

For the full concept writeup about QEMU multi-process, please
refer to docs/devel/qemu-multiprocess.rst. Also, see
docs/qemu-multiprocess.txt for usage information.

Thank you for reviewing this series!

[POC]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
[1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
[2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
[3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
[4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
[5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
[6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
[7]: https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
[8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
[9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
[10]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
[11]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.html
[12]: https://patchew.org/QEMU/cover.1606853298.git.jag.raman@oracle.com/
[13]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg766825.html
[14]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg768376.html
[15]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg769178.html

Elena Ufimtseva (8):
  multi-process: add configure and usage information
  io: add qio_channel_writev_full_all helper
  io: add qio_channel_readv_full_all_eof & qio_channel_readv_full_all
    helpers
  multi-process: define MPQemuMsg format and transmission functions
  multi-process: introduce proxy object
  multi-process: add proxy communication functions
  multi-process: Forward PCI config space acceses to the remote process
  multi-process: perform device reset in the remote process

Jagannathan Raman (11):
  memory: alloc RAM from file at offset
  multi-process: Add config option for multi-process QEMU
  multi-process: setup PCI host bridge for remote device
  multi-process: setup a machine object for remote device process
  multi-process: Initialize message handler in remote device
  multi-process: Associate fd of a PCIDevice with its object
  multi-process: setup memory manager for remote device
  multi-process: PCI BAR read/write handling for proxy & remote
    endpoints
  multi-process: Synchronize remote memory
  multi-process: create IOHUB object to handle irq
  multi-process: Retrieve PCI info from remote process

John G Johnson (1):
  multi-process: add the concept description to
    docs/devel/qemu-multiprocess

 docs/devel/index.rst                      |   1 +
 docs/devel/multi-process.rst              | 966 ++++++++++++++++++++++++++++++
 docs/multi-process.rst                    |  64 ++
 configure                                 |  10 +
 meson.build                               |   5 +-
 hw/remote/trace.h                         |   1 +
 include/exec/memory.h                     |   2 +
 include/exec/ram_addr.h                   |   2 +-
 include/hw/pci-host/remote.h              |  30 +
 include/hw/pci/pci_ids.h                  |   3 +
 include/hw/remote/iohub.h                 |  42 ++
 include/hw/remote/machine.h               |  38 ++
 include/hw/remote/memory.h                |  19 +
 include/hw/remote/mpqemu-link.h           |  99 +++
 include/hw/remote/proxy-memory-listener.h |  28 +
 include/hw/remote/proxy.h                 |  48 ++
 include/io/channel.h                      |  76 +++
 include/qemu/mmap-alloc.h                 |   4 +-
 include/sysemu/iothread.h                 |   6 +
 backends/hostmem-memfd.c                  |   2 +-
 hw/misc/ivshmem.c                         |   3 +-
 hw/pci-host/remote.c                      |  75 +++
 hw/remote/iohub.c                         | 119 ++++
 hw/remote/machine.c                       |  80 +++
 hw/remote/memory.c                        |  65 ++
 hw/remote/message.c                       | 230 +++++++
 hw/remote/mpqemu-link.c                   | 267 +++++++++
 hw/remote/proxy-memory-listener.c         | 227 +++++++
 hw/remote/proxy.c                         | 379 ++++++++++++
 hw/remote/remote-obj.c                    | 203 +++++++
 io/channel.c                              |  88 ++-
 iothread.c                                |   6 +
 softmmu/memory.c                          |   3 +-
 softmmu/physmem.c                         |  11 +-
 util/mmap-alloc.c                         |   7 +-
 util/oslib-posix.c                        |   2 +-
 Kconfig.host                              |   4 +
 MAINTAINERS                               |  24 +
 hw/Kconfig                                |   1 +
 hw/meson.build                            |   1 +
 hw/pci-host/Kconfig                       |   3 +
 hw/pci-host/meson.build                   |   1 +
 hw/remote/Kconfig                         |   4 +
 hw/remote/meson.build                     |  13 +
 hw/remote/trace-events                    |   4 +
 45 files changed, 3234 insertions(+), 32 deletions(-)
 create mode 100644 docs/devel/multi-process.rst
 create mode 100644 docs/multi-process.rst
 create mode 100644 hw/remote/trace.h
 create mode 100644 include/hw/pci-host/remote.h
 create mode 100644 include/hw/remote/iohub.h
 create mode 100644 include/hw/remote/machine.h
 create mode 100644 include/hw/remote/memory.h
 create mode 100644 include/hw/remote/mpqemu-link.h
 create mode 100644 include/hw/remote/proxy-memory-listener.h
 create mode 100644 include/hw/remote/proxy.h
 create mode 100644 hw/pci-host/remote.c
 create mode 100644 hw/remote/iohub.c
 create mode 100644 hw/remote/machine.c
 create mode 100644 hw/remote/memory.c
 create mode 100644 hw/remote/message.c
 create mode 100644 hw/remote/mpqemu-link.c
 create mode 100644 hw/remote/proxy-memory-listener.c
 create mode 100644 hw/remote/proxy.c
 create mode 100644 hw/remote/remote-obj.c
 create mode 100644 hw/remote/Kconfig
 create mode 100644 hw/remote/meson.build
 create mode 100644 hw/remote/trace-events

Comments

Yonggang Luo Jan. 11, 2021, 5:20 a.m. UTC | #1
I have a question, does this support/test on Windows?

On Mon, Jan 11, 2021 at 1:08 PM Jagannathan Raman <jag.raman@oracle.com>
wrote:
>
> Hi
>
> This is the v16 of the patchset. Thank you for your time reviewing v15.
>
> This version has the following changes:
>
> [PATCH v16 04/20] multi-process: Add config option for multi-process QEMU
>   - Using “default_feature” value to enable/disable multiprocess
>
> [PATCH v16 07/20] io: add qio_channel_writev_full_all helper
>   - Removed local variable in qio_channel_writev_full_all(), setting
arguments
>     directly
>   - Fixed indentation issues
>   - Updated commit message
>
> [PATCH v16 08/20] io: add qio_channel_readv_full_all_eof &
>                   qio_channel_readv_full_all helpers
>   - Added two variants of readv - _full_all_eof & _full_all based on
feedback
>   - Dropped errno return value
>   - Updated commit message
>   - Unable to remove local variables and set arguments directly as the
>     arguments are later needed for cleanup (g_free/close) during failure
>
> Switched to using OBJECT_DECLARE_{SIMPLE_TYPE, TYPE} macros in the
> following patches:
>   - [PATCH v16 05/20] multi-process: setup PCI host bridge for remote
device
>   - [PATCH v16 06/20] multi-process: setup a machine object for remote
device
>                       process
>   - [PATCH v16 11/20] multi-process: Associate fd of a PCIDevice with its
object
>   - [PATCH v16 13/20] multi-process: introduce proxy object
>
> Updated copyright text to use the year 2021 in the files that show them.
>
> To touch upon the history of this project, we posted the Proof Of Concept
> patches before the BoF session in 2018. Subsequently, we have posted 15
> versions on the qemu-devel mailing list. You can find them by following
> the links below ([1] - [15]). Following people contributed to the design
and
> implementation of this project:
> Jagannathan Raman <jag.raman@oracle.com>
> Elena Ufimtseva <elena.ufimtseva@oracle.com>
> John G Johnson <john.g.johnson@oracle.com>
> Stefan Hajnoczi <stefanha@redhat.com>
> Konrad Wilk <konrad.wilk@oracle.com>
> Kanth Ghatraju <kanth.ghatraju@oracle.com>
>
> We would like to thank the QEMU community for your feedback in the
> design and implementation of this project. Qemu wiki page:
> https://wiki.qemu.org/Features/MultiProcessQEMU
>
> For the full concept writeup about QEMU multi-process, please
> refer to docs/devel/qemu-multiprocess.rst. Also, see
> docs/qemu-multiprocess.txt for usage information.
>
> Thank you for reviewing this series!
>
> [POC]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
> [1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
> [2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
> [3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
> [4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
> [5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
> [6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
> [7]:
https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
> [8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
> [9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
> [10]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
> [11]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.html
> [12]: https://patchew.org/QEMU/cover.1606853298.git.jag.raman@oracle.com/
> [13]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg766825.html
> [14]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg768376.html
> [15]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg769178.html
>
> Elena Ufimtseva (8):
>   multi-process: add configure and usage information
>   io: add qio_channel_writev_full_all helper
>   io: add qio_channel_readv_full_all_eof & qio_channel_readv_full_all
>     helpers
>   multi-process: define MPQemuMsg format and transmission functions
>   multi-process: introduce proxy object
>   multi-process: add proxy communication functions
>   multi-process: Forward PCI config space acceses to the remote process
>   multi-process: perform device reset in the remote process
>
> Jagannathan Raman (11):
>   memory: alloc RAM from file at offset
>   multi-process: Add config option for multi-process QEMU
>   multi-process: setup PCI host bridge for remote device
>   multi-process: setup a machine object for remote device process
>   multi-process: Initialize message handler in remote device
>   multi-process: Associate fd of a PCIDevice with its object
>   multi-process: setup memory manager for remote device
>   multi-process: PCI BAR read/write handling for proxy & remote
>     endpoints
>   multi-process: Synchronize remote memory
>   multi-process: create IOHUB object to handle irq
>   multi-process: Retrieve PCI info from remote process
>
> John G Johnson (1):
>   multi-process: add the concept description to
>     docs/devel/qemu-multiprocess
>
>  docs/devel/index.rst                      |   1 +
>  docs/devel/multi-process.rst              | 966
++++++++++++++++++++++++++++++
>  docs/multi-process.rst                    |  64 ++
>  configure                                 |  10 +
>  meson.build                               |   5 +-
>  hw/remote/trace.h                         |   1 +
>  include/exec/memory.h                     |   2 +
>  include/exec/ram_addr.h                   |   2 +-
>  include/hw/pci-host/remote.h              |  30 +
>  include/hw/pci/pci_ids.h                  |   3 +
>  include/hw/remote/iohub.h                 |  42 ++
>  include/hw/remote/machine.h               |  38 ++
>  include/hw/remote/memory.h                |  19 +
>  include/hw/remote/mpqemu-link.h           |  99 +++
>  include/hw/remote/proxy-memory-listener.h |  28 +
>  include/hw/remote/proxy.h                 |  48 ++
>  include/io/channel.h                      |  76 +++
>  include/qemu/mmap-alloc.h                 |   4 +-
>  include/sysemu/iothread.h                 |   6 +
>  backends/hostmem-memfd.c                  |   2 +-
>  hw/misc/ivshmem.c                         |   3 +-
>  hw/pci-host/remote.c                      |  75 +++
>  hw/remote/iohub.c                         | 119 ++++
>  hw/remote/machine.c                       |  80 +++
>  hw/remote/memory.c                        |  65 ++
>  hw/remote/message.c                       | 230 +++++++
>  hw/remote/mpqemu-link.c                   | 267 +++++++++
>  hw/remote/proxy-memory-listener.c         | 227 +++++++
>  hw/remote/proxy.c                         | 379 ++++++++++++
>  hw/remote/remote-obj.c                    | 203 +++++++
>  io/channel.c                              |  88 ++-
>  iothread.c                                |   6 +
>  softmmu/memory.c                          |   3 +-
>  softmmu/physmem.c                         |  11 +-
>  util/mmap-alloc.c                         |   7 +-
>  util/oslib-posix.c                        |   2 +-
>  Kconfig.host                              |   4 +
>  MAINTAINERS                               |  24 +
>  hw/Kconfig                                |   1 +
>  hw/meson.build                            |   1 +
>  hw/pci-host/Kconfig                       |   3 +
>  hw/pci-host/meson.build                   |   1 +
>  hw/remote/Kconfig                         |   4 +
>  hw/remote/meson.build                     |  13 +
>  hw/remote/trace-events                    |   4 +
>  45 files changed, 3234 insertions(+), 32 deletions(-)
>  create mode 100644 docs/devel/multi-process.rst
>  create mode 100644 docs/multi-process.rst
>  create mode 100644 hw/remote/trace.h
>  create mode 100644 include/hw/pci-host/remote.h
>  create mode 100644 include/hw/remote/iohub.h
>  create mode 100644 include/hw/remote/machine.h
>  create mode 100644 include/hw/remote/memory.h
>  create mode 100644 include/hw/remote/mpqemu-link.h
>  create mode 100644 include/hw/remote/proxy-memory-listener.h
>  create mode 100644 include/hw/remote/proxy.h
>  create mode 100644 hw/pci-host/remote.c
>  create mode 100644 hw/remote/iohub.c
>  create mode 100644 hw/remote/machine.c
>  create mode 100644 hw/remote/memory.c
>  create mode 100644 hw/remote/message.c
>  create mode 100644 hw/remote/mpqemu-link.c
>  create mode 100644 hw/remote/proxy-memory-listener.c
>  create mode 100644 hw/remote/proxy.c
>  create mode 100644 hw/remote/remote-obj.c
>  create mode 100644 hw/remote/Kconfig
>  create mode 100644 hw/remote/meson.build
>  create mode 100644 hw/remote/trace-events
>
> --
> 1.8.3.1
>
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo
Jag Raman Jan. 11, 2021, 6:02 p.m. UTC | #2
> On Jan 11, 2021, at 12:20 AM, 罗勇刚(Yonggang Luo) <luoyonggang@gmail.com> wrote:
> 
> I have a question, does this support/test on Windows?

Hi Yonggang, we have not tested this on Windows. The project only builds on Linux for now.

Thank you!
—
Jag

> 
> On Mon, Jan 11, 2021 at 1:08 PM Jagannathan Raman <jag.raman@oracle.com> wrote:
> >
> > Hi
> >
> > This is the v16 of the patchset. Thank you for your time reviewing v15.
> >
> > This version has the following changes:
> >
> > [PATCH v16 04/20] multi-process: Add config option for multi-process QEMU
> >   - Using “default_feature” value to enable/disable multiprocess
> >
> > [PATCH v16 07/20] io: add qio_channel_writev_full_all helper
> >   - Removed local variable in qio_channel_writev_full_all(), setting arguments
> >     directly
> >   - Fixed indentation issues
> >   - Updated commit message
> >
> > [PATCH v16 08/20] io: add qio_channel_readv_full_all_eof &
> >                   qio_channel_readv_full_all helpers
> >   - Added two variants of readv - _full_all_eof & _full_all based on feedback
> >   - Dropped errno return value
> >   - Updated commit message
> >   - Unable to remove local variables and set arguments directly as the
> >     arguments are later needed for cleanup (g_free/close) during failure
> >
> > Switched to using OBJECT_DECLARE_{SIMPLE_TYPE, TYPE} macros in the
> > following patches:
> >   - [PATCH v16 05/20] multi-process: setup PCI host bridge for remote device
> >   - [PATCH v16 06/20] multi-process: setup a machine object for remote device
> >                       process
> >   - [PATCH v16 11/20] multi-process: Associate fd of a PCIDevice with its object
> >   - [PATCH v16 13/20] multi-process: introduce proxy object
> >
> > Updated copyright text to use the year 2021 in the files that show them.
> >
> > To touch upon the history of this project, we posted the Proof Of Concept
> > patches before the BoF session in 2018. Subsequently, we have posted 15
> > versions on the qemu-devel mailing list. You can find them by following
> > the links below ([1] - [15]). Following people contributed to the design and
> > implementation of this project:
> > Jagannathan Raman <jag.raman@oracle.com>
> > Elena Ufimtseva <elena.ufimtseva@oracle.com>
> > John G Johnson <john.g.johnson@oracle.com>
> > Stefan Hajnoczi <stefanha@redhat.com>
> > Konrad Wilk <konrad.wilk@oracle.com>
> > Kanth Ghatraju <kanth.ghatraju@oracle.com>
> >
> > We would like to thank the QEMU community for your feedback in the
> > design and implementation of this project. Qemu wiki page:
> > https://wiki.qemu.org/Features/MultiProcessQEMU
> >
> > For the full concept writeup about QEMU multi-process, please
> > refer to docs/devel/qemu-multiprocess.rst. Also, see
> > docs/qemu-multiprocess.txt for usage information.
> >
> > Thank you for reviewing this series!
> >
> > [POC]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
> > [1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
> > [2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
> > [3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
> > [4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
> > [5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
> > [6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
> > [7]: https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
> > [8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
> > [9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
> > [10]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
> > [11]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.html
> > [12]: https://patchew.org/QEMU/cover.1606853298.git.jag.raman@oracle.com/
> > [13]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg766825.html
> > [14]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg768376.html
> > [15]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg769178.html
> >
> > Elena Ufimtseva (8):
> >   multi-process: add configure and usage information
> >   io: add qio_channel_writev_full_all helper
> >   io: add qio_channel_readv_full_all_eof & qio_channel_readv_full_all
> >     helpers
> >   multi-process: define MPQemuMsg format and transmission functions
> >   multi-process: introduce proxy object
> >   multi-process: add proxy communication functions
> >   multi-process: Forward PCI config space acceses to the remote process
> >   multi-process: perform device reset in the remote process
> >
> > Jagannathan Raman (11):
> >   memory: alloc RAM from file at offset
> >   multi-process: Add config option for multi-process QEMU
> >   multi-process: setup PCI host bridge for remote device
> >   multi-process: setup a machine object for remote device process
> >   multi-process: Initialize message handler in remote device
> >   multi-process: Associate fd of a PCIDevice with its object
> >   multi-process: setup memory manager for remote device
> >   multi-process: PCI BAR read/write handling for proxy & remote
> >     endpoints
> >   multi-process: Synchronize remote memory
> >   multi-process: create IOHUB object to handle irq
> >   multi-process: Retrieve PCI info from remote process
> >
> > John G Johnson (1):
> >   multi-process: add the concept description to
> >     docs/devel/qemu-multiprocess
> >
> >  docs/devel/index.rst                      |   1 +
> >  docs/devel/multi-process.rst              | 966 ++++++++++++++++++++++++++++++
> >  docs/multi-process.rst                    |  64 ++
> >  configure                                 |  10 +
> >  meson.build                               |   5 +-
> >  hw/remote/trace.h                         |   1 +
> >  include/exec/memory.h                     |   2 +
> >  include/exec/ram_addr.h                   |   2 +-
> >  include/hw/pci-host/remote.h              |  30 +
> >  include/hw/pci/pci_ids.h                  |   3 +
> >  include/hw/remote/iohub.h                 |  42 ++
> >  include/hw/remote/machine.h               |  38 ++
> >  include/hw/remote/memory.h                |  19 +
> >  include/hw/remote/mpqemu-link.h           |  99 +++
> >  include/hw/remote/proxy-memory-listener.h |  28 +
> >  include/hw/remote/proxy.h                 |  48 ++
> >  include/io/channel.h                      |  76 +++
> >  include/qemu/mmap-alloc.h                 |   4 +-
> >  include/sysemu/iothread.h                 |   6 +
> >  backends/hostmem-memfd.c                  |   2 +-
> >  hw/misc/ivshmem.c                         |   3 +-
> >  hw/pci-host/remote.c                      |  75 +++
> >  hw/remote/iohub.c                         | 119 ++++
> >  hw/remote/machine.c                       |  80 +++
> >  hw/remote/memory.c                        |  65 ++
> >  hw/remote/message.c                       | 230 +++++++
> >  hw/remote/mpqemu-link.c                   | 267 +++++++++
> >  hw/remote/proxy-memory-listener.c         | 227 +++++++
> >  hw/remote/proxy.c                         | 379 ++++++++++++
> >  hw/remote/remote-obj.c                    | 203 +++++++
> >  io/channel.c                              |  88 ++-
> >  iothread.c                                |   6 +
> >  softmmu/memory.c                          |   3 +-
> >  softmmu/physmem.c                         |  11 +-
> >  util/mmap-alloc.c                         |   7 +-
> >  util/oslib-posix.c                        |   2 +-
> >  Kconfig.host                              |   4 +
> >  MAINTAINERS                               |  24 +
> >  hw/Kconfig                                |   1 +
> >  hw/meson.build                            |   1 +
> >  hw/pci-host/Kconfig                       |   3 +
> >  hw/pci-host/meson.build                   |   1 +
> >  hw/remote/Kconfig                         |   4 +
> >  hw/remote/meson.build                     |  13 +
> >  hw/remote/trace-events                    |   4 +
> >  45 files changed, 3234 insertions(+), 32 deletions(-)
> >  create mode 100644 docs/devel/multi-process.rst
> >  create mode 100644 docs/multi-process.rst
> >  create mode 100644 hw/remote/trace.h
> >  create mode 100644 include/hw/pci-host/remote.h
> >  create mode 100644 include/hw/remote/iohub.h
> >  create mode 100644 include/hw/remote/machine.h
> >  create mode 100644 include/hw/remote/memory.h
> >  create mode 100644 include/hw/remote/mpqemu-link.h
> >  create mode 100644 include/hw/remote/proxy-memory-listener.h
> >  create mode 100644 include/hw/remote/proxy.h
> >  create mode 100644 hw/pci-host/remote.c
> >  create mode 100644 hw/remote/iohub.c
> >  create mode 100644 hw/remote/machine.c
> >  create mode 100644 hw/remote/memory.c
> >  create mode 100644 hw/remote/message.c
> >  create mode 100644 hw/remote/mpqemu-link.c
> >  create mode 100644 hw/remote/proxy-memory-listener.c
> >  create mode 100644 hw/remote/proxy.c
> >  create mode 100644 hw/remote/remote-obj.c
> >  create mode 100644 hw/remote/Kconfig
> >  create mode 100644 hw/remote/meson.build
> >  create mode 100644 hw/remote/trace-events
> >
> > --
> > 1.8.3.1
> >
> >
> 
> 
> --
>          此致
> 礼
> 罗勇刚
> Yours
>     sincerely,
> Yonggang Luo
Yonggang Luo Jan. 11, 2021, 8:13 p.m. UTC | #3
On Tue, Jan 12, 2021 at 2:02 AM Jag Raman <jag.raman@oracle.com> wrote:
>
>
>
> > On Jan 11, 2021, at 12:20 AM, 罗勇刚(Yonggang Luo) <luoyonggang@gmail.com>
wrote:
> >
> > I have a question, does this support/test on Windows?
>
> Hi Yonggang, we have not tested this on Windows. The project only builds
on Linux for now.
>
> Thank you!
I may interested add Windows support, what's the hard part?
If we add Windows/OSX support, we can create same code base for all three
platforms, and reduce #ifdef _WIN32 like macros


> —
> Jag
>
> >
> > On Mon, Jan 11, 2021 at 1:08 PM Jagannathan Raman <jag.raman@oracle.com>
wrote:
> > >
> > > Hi
> > >
> > > This is the v16 of the patchset. Thank you for your time reviewing
v15.
> > >
> > > This version has the following changes:
> > >
> > > [PATCH v16 04/20] multi-process: Add config option for multi-process
QEMU
> > >   - Using “default_feature” value to enable/disable multiprocess
> > >
> > > [PATCH v16 07/20] io: add qio_channel_writev_full_all helper
> > >   - Removed local variable in qio_channel_writev_full_all(), setting
arguments
> > >     directly
> > >   - Fixed indentation issues
> > >   - Updated commit message
> > >
> > > [PATCH v16 08/20] io: add qio_channel_readv_full_all_eof &
> > >                   qio_channel_readv_full_all helpers
> > >   - Added two variants of readv - _full_all_eof & _full_all based on
feedback
> > >   - Dropped errno return value
> > >   - Updated commit message
> > >   - Unable to remove local variables and set arguments directly as the
> > >     arguments are later needed for cleanup (g_free/close) during
failure
> > >
> > > Switched to using OBJECT_DECLARE_{SIMPLE_TYPE, TYPE} macros in the
> > > following patches:
> > >   - [PATCH v16 05/20] multi-process: setup PCI host bridge for remote
device
> > >   - [PATCH v16 06/20] multi-process: setup a machine object for
remote device
> > >                       process
> > >   - [PATCH v16 11/20] multi-process: Associate fd of a PCIDevice with
its object
> > >   - [PATCH v16 13/20] multi-process: introduce proxy object
> > >
> > > Updated copyright text to use the year 2021 in the files that show
them.
> > >
> > > To touch upon the history of this project, we posted the Proof Of
Concept
> > > patches before the BoF session in 2018. Subsequently, we have posted
15
> > > versions on the qemu-devel mailing list. You can find them by
following
> > > the links below ([1] - [15]). Following people contributed to the
design and
> > > implementation of this project:
> > > Jagannathan Raman <jag.raman@oracle.com>
> > > Elena Ufimtseva <elena.ufimtseva@oracle.com>
> > > John G Johnson <john.g.johnson@oracle.com>
> > > Stefan Hajnoczi <stefanha@redhat.com>
> > > Konrad Wilk <konrad.wilk@oracle.com>
> > > Kanth Ghatraju <kanth.ghatraju@oracle.com>
> > >
> > > We would like to thank the QEMU community for your feedback in the
> > > design and implementation of this project. Qemu wiki page:
> > > https://wiki.qemu.org/Features/MultiProcessQEMU
> > >
> > > For the full concept writeup about QEMU multi-process, please
> > > refer to docs/devel/qemu-multiprocess.rst. Also, see
> > > docs/qemu-multiprocess.txt for usage information.
> > >
> > > Thank you for reviewing this series!
> > >
> > > [POC]:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
> > > [1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
> > > [2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
> > > [3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
> > > [4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
> > > [5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
> > > [6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
> > > [7]:
https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
> > > [8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
> > > [9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
> > > [10]:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
> > > [11]:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.html
> > > [12]:
https://patchew.org/QEMU/cover.1606853298.git.jag.raman@oracle.com/
> > > [13]:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg766825.html
> > > [14]:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg768376.html
> > > [15]:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg769178.html
> > >
> > > Elena Ufimtseva (8):
> > >   multi-process: add configure and usage information
> > >   io: add qio_channel_writev_full_all helper
> > >   io: add qio_channel_readv_full_all_eof & qio_channel_readv_full_all
> > >     helpers
> > >   multi-process: define MPQemuMsg format and transmission functions
> > >   multi-process: introduce proxy object
> > >   multi-process: add proxy communication functions
> > >   multi-process: Forward PCI config space acceses to the remote
process
> > >   multi-process: perform device reset in the remote process
> > >
> > > Jagannathan Raman (11):
> > >   memory: alloc RAM from file at offset
> > >   multi-process: Add config option for multi-process QEMU
> > >   multi-process: setup PCI host bridge for remote device
> > >   multi-process: setup a machine object for remote device process
> > >   multi-process: Initialize message handler in remote device
> > >   multi-process: Associate fd of a PCIDevice with its object
> > >   multi-process: setup memory manager for remote device
> > >   multi-process: PCI BAR read/write handling for proxy & remote
> > >     endpoints
> > >   multi-process: Synchronize remote memory
> > >   multi-process: create IOHUB object to handle irq
> > >   multi-process: Retrieve PCI info from remote process
> > >
> > > John G Johnson (1):
> > >   multi-process: add the concept description to
> > >     docs/devel/qemu-multiprocess
> > >
> > >  docs/devel/index.rst                      |   1 +
> > >  docs/devel/multi-process.rst              | 966
++++++++++++++++++++++++++++++
> > >  docs/multi-process.rst                    |  64 ++
> > >  configure                                 |  10 +
> > >  meson.build                               |   5 +-
> > >  hw/remote/trace.h                         |   1 +
> > >  include/exec/memory.h                     |   2 +
> > >  include/exec/ram_addr.h                   |   2 +-
> > >  include/hw/pci-host/remote.h              |  30 +
> > >  include/hw/pci/pci_ids.h                  |   3 +
> > >  include/hw/remote/iohub.h                 |  42 ++
> > >  include/hw/remote/machine.h               |  38 ++
> > >  include/hw/remote/memory.h                |  19 +
> > >  include/hw/remote/mpqemu-link.h           |  99 +++
> > >  include/hw/remote/proxy-memory-listener.h |  28 +
> > >  include/hw/remote/proxy.h                 |  48 ++
> > >  include/io/channel.h                      |  76 +++
> > >  include/qemu/mmap-alloc.h                 |   4 +-
> > >  include/sysemu/iothread.h                 |   6 +
> > >  backends/hostmem-memfd.c                  |   2 +-
> > >  hw/misc/ivshmem.c                         |   3 +-
> > >  hw/pci-host/remote.c                      |  75 +++
> > >  hw/remote/iohub.c                         | 119 ++++
> > >  hw/remote/machine.c                       |  80 +++
> > >  hw/remote/memory.c                        |  65 ++
> > >  hw/remote/message.c                       | 230 +++++++
> > >  hw/remote/mpqemu-link.c                   | 267 +++++++++
> > >  hw/remote/proxy-memory-listener.c         | 227 +++++++
> > >  hw/remote/proxy.c                         | 379 ++++++++++++
> > >  hw/remote/remote-obj.c                    | 203 +++++++
> > >  io/channel.c                              |  88 ++-
> > >  iothread.c                                |   6 +
> > >  softmmu/memory.c                          |   3 +-
> > >  softmmu/physmem.c                         |  11 +-
> > >  util/mmap-alloc.c                         |   7 +-
> > >  util/oslib-posix.c                        |   2 +-
> > >  Kconfig.host                              |   4 +
> > >  MAINTAINERS                               |  24 +
> > >  hw/Kconfig                                |   1 +
> > >  hw/meson.build                            |   1 +
> > >  hw/pci-host/Kconfig                       |   3 +
> > >  hw/pci-host/meson.build                   |   1 +
> > >  hw/remote/Kconfig                         |   4 +
> > >  hw/remote/meson.build                     |  13 +
> > >  hw/remote/trace-events                    |   4 +
> > >  45 files changed, 3234 insertions(+), 32 deletions(-)
> > >  create mode 100644 docs/devel/multi-process.rst
> > >  create mode 100644 docs/multi-process.rst
> > >  create mode 100644 hw/remote/trace.h
> > >  create mode 100644 include/hw/pci-host/remote.h
> > >  create mode 100644 include/hw/remote/iohub.h
> > >  create mode 100644 include/hw/remote/machine.h
> > >  create mode 100644 include/hw/remote/memory.h
> > >  create mode 100644 include/hw/remote/mpqemu-link.h
> > >  create mode 100644 include/hw/remote/proxy-memory-listener.h
> > >  create mode 100644 include/hw/remote/proxy.h
> > >  create mode 100644 hw/pci-host/remote.c
> > >  create mode 100644 hw/remote/iohub.c
> > >  create mode 100644 hw/remote/machine.c
> > >  create mode 100644 hw/remote/memory.c
> > >  create mode 100644 hw/remote/message.c
> > >  create mode 100644 hw/remote/mpqemu-link.c
> > >  create mode 100644 hw/remote/proxy-memory-listener.c
> > >  create mode 100644 hw/remote/proxy.c
> > >  create mode 100644 hw/remote/remote-obj.c
> > >  create mode 100644 hw/remote/Kconfig
> > >  create mode 100644 hw/remote/meson.build
> > >  create mode 100644 hw/remote/trace-events
> > >
> > > --
> > > 1.8.3.1
> > >
> > >
> >
> >
> > --
> >          此致
> > 礼
> > 罗勇刚
> > Yours
> >     sincerely,
> > Yonggang Luo
>


--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo
Stefan Hajnoczi Jan. 12, 2021, 11:17 a.m. UTC | #4
On Mon, Jan 11, 2021 at 12:05:45AM -0500, Jagannathan Raman wrote:
> [PATCH v16 08/20] io: add qio_channel_readv_full_all_eof &
>                   qio_channel_readv_full_all helpers
>   - Added two variants of readv - _full_all_eof & _full_all based on feedback
>   - Dropped errno return value
>   - Updated commit message
>   - Unable to remove local variables and set arguments directly as the
>     arguments are later needed for cleanup (g_free/close) during failure

I noticed issues with the qio_channel_readv_full_all() error case and
posted comments on the patch.

Stefan
Jag Raman Jan. 13, 2021, 4:45 p.m. UTC | #5
> On Jan 11, 2021, at 3:13 PM, 罗勇刚(Yonggang Luo) <luoyonggang@gmail.com> wrote:
> 
> 
> 
> On Tue, Jan 12, 2021 at 2:02 AM Jag Raman <jag.raman@oracle.com> wrote:
> >
> >
> >
> > > On Jan 11, 2021, at 12:20 AM, 罗勇刚(Yonggang Luo) <luoyonggang@gmail.com> wrote:
> > >
> > > I have a question, does this support/test on Windows?
> >
> > Hi Yonggang, we have not tested this on Windows. The project only builds on Linux for now.
> >
> > Thank you!
> I may interested add Windows support, what's the hard part?

Hi Yonggang,

I think it would be the unix sockets and sharing file descriptors via them.

Thanks!

> If we add Windows/OSX support, we can create same code base for all three platforms, and reduce #ifdef _WIN32 like macros
> 
> 
> > —
> > Jag
> >
> > >
> > > On Mon, Jan 11, 2021 at 1:08 PM Jagannathan Raman <jag.raman@oracle.com> wrote:
> > > >
> > > > Hi
> > > >
> > > > This is the v16 of the patchset. Thank you for your time reviewing v15.
> > > >
> > > > This version has the following changes:
> > > >
> > > > [PATCH v16 04/20] multi-process: Add config option for multi-process QEMU
> > > >   - Using “default_feature” value to enable/disable multiprocess
> > > >
> > > > [PATCH v16 07/20] io: add qio_channel_writev_full_all helper
> > > >   - Removed local variable in qio_channel_writev_full_all(), setting arguments
> > > >     directly
> > > >   - Fixed indentation issues
> > > >   - Updated commit message
> > > >
> > > > [PATCH v16 08/20] io: add qio_channel_readv_full_all_eof &
> > > >                   qio_channel_readv_full_all helpers
> > > >   - Added two variants of readv - _full_all_eof & _full_all based on feedback
> > > >   - Dropped errno return value
> > > >   - Updated commit message
> > > >   - Unable to remove local variables and set arguments directly as the
> > > >     arguments are later needed for cleanup (g_free/close) during failure
> > > >
> > > > Switched to using OBJECT_DECLARE_{SIMPLE_TYPE, TYPE} macros in the
> > > > following patches:
> > > >   - [PATCH v16 05/20] multi-process: setup PCI host bridge for remote device
> > > >   - [PATCH v16 06/20] multi-process: setup a machine object for remote device
> > > >                       process
> > > >   - [PATCH v16 11/20] multi-process: Associate fd of a PCIDevice with its object
> > > >   - [PATCH v16 13/20] multi-process: introduce proxy object
> > > >
> > > > Updated copyright text to use the year 2021 in the files that show them.
> > > >
> > > > To touch upon the history of this project, we posted the Proof Of Concept
> > > > patches before the BoF session in 2018. Subsequently, we have posted 15
> > > > versions on the qemu-devel mailing list. You can find them by following
> > > > the links below ([1] - [15]). Following people contributed to the design and
> > > > implementation of this project:
> > > > Jagannathan Raman <jag.raman@oracle.com>
> > > > Elena Ufimtseva <elena.ufimtseva@oracle.com>
> > > > John G Johnson <john.g.johnson@oracle.com>
> > > > Stefan Hajnoczi <stefanha@redhat.com>
> > > > Konrad Wilk <konrad.wilk@oracle.com>
> > > > Kanth Ghatraju <kanth.ghatraju@oracle.com>
> > > >
> > > > We would like to thank the QEMU community for your feedback in the
> > > > design and implementation of this project. Qemu wiki page:
> > > > https://wiki.qemu.org/Features/MultiProcessQEMU
> > > >
> > > > For the full concept writeup about QEMU multi-process, please
> > > > refer to docs/devel/qemu-multiprocess.rst. Also, see
> > > > docs/qemu-multiprocess.txt for usage information.
> > > >
> > > > Thank you for reviewing this series!
> > > >
> > > > [POC]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
> > > > [1]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
> > > > [2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html
> > > > [3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg642000.html
> > > > [4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg655118.html
> > > > [5]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg682429.html
> > > > [6]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg697484.html
> > > > [7]: https://patchew.org/QEMU/cover.1593273671.git.elena.ufimtseva@oracle.com/
> > > > [8]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg727007.html
> > > > [9]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg734275.html
> > > > [10]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg747638.html
> > > > [11]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg750972.html
> > > > [12]: https://patchew.org/QEMU/cover.1606853298.git.jag.raman@oracle.com/
> > > > [13]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg766825.html
> > > > [14]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg768376.html
> > > > [15]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg769178.html
> > > >
> > > > Elena Ufimtseva (8):
> > > >   multi-process: add configure and usage information
> > > >   io: add qio_channel_writev_full_all helper
> > > >   io: add qio_channel_readv_full_all_eof & qio_channel_readv_full_all
> > > >     helpers
> > > >   multi-process: define MPQemuMsg format and transmission functions
> > > >   multi-process: introduce proxy object
> > > >   multi-process: add proxy communication functions
> > > >   multi-process: Forward PCI config space acceses to the remote process
> > > >   multi-process: perform device reset in the remote process
> > > >
> > > > Jagannathan Raman (11):
> > > >   memory: alloc RAM from file at offset
> > > >   multi-process: Add config option for multi-process QEMU
> > > >   multi-process: setup PCI host bridge for remote device
> > > >   multi-process: setup a machine object for remote device process
> > > >   multi-process: Initialize message handler in remote device
> > > >   multi-process: Associate fd of a PCIDevice with its object
> > > >   multi-process: setup memory manager for remote device
> > > >   multi-process: PCI BAR read/write handling for proxy & remote
> > > >     endpoints
> > > >   multi-process: Synchronize remote memory
> > > >   multi-process: create IOHUB object to handle irq
> > > >   multi-process: Retrieve PCI info from remote process
> > > >
> > > > John G Johnson (1):
> > > >   multi-process: add the concept description to
> > > >     docs/devel/qemu-multiprocess
> > > >
> > > >  docs/devel/index.rst                      |   1 +
> > > >  docs/devel/multi-process.rst              | 966 ++++++++++++++++++++++++++++++
> > > >  docs/multi-process.rst                    |  64 ++
> > > >  configure                                 |  10 +
> > > >  meson.build                               |   5 +-
> > > >  hw/remote/trace.h                         |   1 +
> > > >  include/exec/memory.h                     |   2 +
> > > >  include/exec/ram_addr.h                   |   2 +-
> > > >  include/hw/pci-host/remote.h              |  30 +
> > > >  include/hw/pci/pci_ids.h                  |   3 +
> > > >  include/hw/remote/iohub.h                 |  42 ++
> > > >  include/hw/remote/machine.h               |  38 ++
> > > >  include/hw/remote/memory.h                |  19 +
> > > >  include/hw/remote/mpqemu-link.h           |  99 +++
> > > >  include/hw/remote/proxy-memory-listener.h |  28 +
> > > >  include/hw/remote/proxy.h                 |  48 ++
> > > >  include/io/channel.h                      |  76 +++
> > > >  include/qemu/mmap-alloc.h                 |   4 +-
> > > >  include/sysemu/iothread.h                 |   6 +
> > > >  backends/hostmem-memfd.c                  |   2 +-
> > > >  hw/misc/ivshmem.c                         |   3 +-
> > > >  hw/pci-host/remote.c                      |  75 +++
> > > >  hw/remote/iohub.c                         | 119 ++++
> > > >  hw/remote/machine.c                       |  80 +++
> > > >  hw/remote/memory.c                        |  65 ++
> > > >  hw/remote/message.c                       | 230 +++++++
> > > >  hw/remote/mpqemu-link.c                   | 267 +++++++++
> > > >  hw/remote/proxy-memory-listener.c         | 227 +++++++
> > > >  hw/remote/proxy.c                         | 379 ++++++++++++
> > > >  hw/remote/remote-obj.c                    | 203 +++++++
> > > >  io/channel.c                              |  88 ++-
> > > >  iothread.c                                |   6 +
> > > >  softmmu/memory.c                          |   3 +-
> > > >  softmmu/physmem.c                         |  11 +-
> > > >  util/mmap-alloc.c                         |   7 +-
> > > >  util/oslib-posix.c                        |   2 +-
> > > >  Kconfig.host                              |   4 +
> > > >  MAINTAINERS                               |  24 +
> > > >  hw/Kconfig                                |   1 +
> > > >  hw/meson.build                            |   1 +
> > > >  hw/pci-host/Kconfig                       |   3 +
> > > >  hw/pci-host/meson.build                   |   1 +
> > > >  hw/remote/Kconfig                         |   4 +
> > > >  hw/remote/meson.build                     |  13 +
> > > >  hw/remote/trace-events                    |   4 +
> > > >  45 files changed, 3234 insertions(+), 32 deletions(-)
> > > >  create mode 100644 docs/devel/multi-process.rst
> > > >  create mode 100644 docs/multi-process.rst
> > > >  create mode 100644 hw/remote/trace.h
> > > >  create mode 100644 include/hw/pci-host/remote.h
> > > >  create mode 100644 include/hw/remote/iohub.h
> > > >  create mode 100644 include/hw/remote/machine.h
> > > >  create mode 100644 include/hw/remote/memory.h
> > > >  create mode 100644 include/hw/remote/mpqemu-link.h
> > > >  create mode 100644 include/hw/remote/proxy-memory-listener.h
> > > >  create mode 100644 include/hw/remote/proxy.h
> > > >  create mode 100644 hw/pci-host/remote.c
> > > >  create mode 100644 hw/remote/iohub.c
> > > >  create mode 100644 hw/remote/machine.c
> > > >  create mode 100644 hw/remote/memory.c
> > > >  create mode 100644 hw/remote/message.c
> > > >  create mode 100644 hw/remote/mpqemu-link.c
> > > >  create mode 100644 hw/remote/proxy-memory-listener.c
> > > >  create mode 100644 hw/remote/proxy.c
> > > >  create mode 100644 hw/remote/remote-obj.c
> > > >  create mode 100644 hw/remote/Kconfig
> > > >  create mode 100644 hw/remote/meson.build
> > > >  create mode 100644 hw/remote/trace-events
> > > >
> > > > --
> > > > 1.8.3.1
> > > >
> > > >
> > >
> > >
> > > --
> > >          此致
> > > 礼
> > > 罗勇刚
> > > Yours
> > >     sincerely,
> > > Yonggang Luo
> >
> 
> 
> --
>          此致
> 礼
> 罗勇刚
> Yours
>     sincerely,
> Yonggang Luo