mbox series

[v3,0/8] qapi-go: add generator for Golang interfaces

Message ID 20250110104946.74960-1-victortoso@redhat.com (mailing list archive)
Headers show
Series qapi-go: add generator for Golang interfaces | expand

Message

Victor Toso Jan. 10, 2025, 10:49 a.m. UTC
This patch series intent is to introduce a generator that produces a Go
module for Go applications to interact over QMP with QEMU.

The initial Goal is to have a Go module that works as intended and can
be improved upon. I'd consider initial releases to be alpha while we
work with utilities tools and libraries on top of this.

The generated code should reside in a separated Git repository, similar
to python-qemu-qmp.

Applications should be able to consume this under qemu.org
namespace (e.g: import "qemu.org/go/qemu"), see Daniel's suggestion:
https://lists.gnu.org/archive/html/qemu-devel/2023-09/msg07024.html

This is the third iteration:
v2: https://lists.gnu.org/archive/html/qemu-devel/2023-10/msg04785.html

I've pushed this series in my gitlab fork:
https://gitlab.com/victortoso/qapi-go/

The fork contains some tests, including tests that were generated from
QAPI's own examples from another generator created for testing, if you
are interested in it:
https://lists.gnu.org/archive/html/qemu-devel/2023-09/msg04946.html

I've generated the qapi-go module over each commit of this series, see:
https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v3-by-patch

I've also generated the qapi-go module over QEMU tags: v9.1.0, v9.2.0:
https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v3-by-tags

--

Sorry that its been awhile between v2 and v3, I had to prioritize other
things. I hope to get this back on track in 2025.

Cheers,
Victor

* Changes:

On generated go:
 - the output should be formatted as gofmt/goimports tools (Daniel)

 - Included QAPI's documentation too (Daniel), see:
   https://lists.gnu.org/archive/html/qemu-devel/2024-11/msg01621.html
    
 - Commands and Events should Marshal directly (Andrea)

On python script:
 - rebased: now uses QAPISchemaBranches, QAPISchemaAlternatives

 - use textwrap as much as possible (Andrea)

 - lots of changes to make the output like gofmt does

Victor Toso (8):
  qapi: golang: Generate enum type
  qapi: golang: Generate alternate types
  qapi: golang: Generate struct types
  qapi: golang: structs: Address nullable members
  qapi: golang: Generate union type
  qapi: golang: Generate event type
  qapi: golang: Generate command type
  docs: add notes on Golang code generator

 docs/devel/index-build.rst          |    1 +
 docs/devel/qapi-golang-code-gen.rst |  548 +++++++++
 scripts/qapi/golang.py              | 1645 +++++++++++++++++++++++++++
 scripts/qapi/main.py                |    3 +
 4 files changed, 2197 insertions(+)
 create mode 100644 docs/devel/qapi-golang-code-gen.rst
 create mode 100644 scripts/qapi/golang.py

Comments

Markus Armbruster Jan. 13, 2025, 12:52 p.m. UTC | #1
Victor Toso <victortoso@redhat.com> writes:

> This patch series intent is to introduce a generator that produces a Go
> module for Go applications to interact over QMP with QEMU.
>
> The initial Goal is to have a Go module that works as intended and can
> be improved upon. I'd consider initial releases to be alpha while we
> work with utilities tools and libraries on top of this.
>
> The generated code should reside in a separated Git repository, similar
> to python-qemu-qmp.
>
> Applications should be able to consume this under qemu.org
> namespace (e.g: import "qemu.org/go/qemu"), see Daniel's suggestion:
> https://lists.gnu.org/archive/html/qemu-devel/2023-09/msg07024.html
>
> This is the third iteration:
> v2: https://lists.gnu.org/archive/html/qemu-devel/2023-10/msg04785.html
>
> I've pushed this series in my gitlab fork:
> https://gitlab.com/victortoso/qapi-go/
>
> The fork contains some tests, including tests that were generated from
> QAPI's own examples from another generator created for testing, if you
> are interested in it:
> https://lists.gnu.org/archive/html/qemu-devel/2023-09/msg04946.html
>
> I've generated the qapi-go module over each commit of this series, see:
> https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v3-by-patch
>
> I've also generated the qapi-go module over QEMU tags: v9.1.0, v9.2.0:
> https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v3-by-tags
>
> --
>
> Sorry that its been awhile between v2 and v3, I had to prioritize other
> things. I hope to get this back on track in 2025.
>
> Cheers,
> Victor
>
> * Changes:
>
> On generated go:
>  - the output should be formatted as gofmt/goimports tools (Daniel)
>
>  - Included QAPI's documentation too (Daniel), see:
>    https://lists.gnu.org/archive/html/qemu-devel/2024-11/msg01621.html
>     
>  - Commands and Events should Marshal directly (Andrea)
>
> On python script:
>  - rebased: now uses QAPISchemaBranches, QAPISchemaAlternatives
>
>  - use textwrap as much as possible (Andrea)
>
>  - lots of changes to make the output like gofmt does
>
> Victor Toso (8):
>   qapi: golang: Generate enum type
>   qapi: golang: Generate alternate types
>   qapi: golang: Generate struct types
>   qapi: golang: structs: Address nullable members
>   qapi: golang: Generate union type
>   qapi: golang: Generate event type
>   qapi: golang: Generate command type
>   docs: add notes on Golang code generator
>
>  docs/devel/index-build.rst          |    1 +
>  docs/devel/qapi-golang-code-gen.rst |  548 +++++++++
>  scripts/qapi/golang.py              | 1645 +++++++++++++++++++++++++++
>  scripts/qapi/main.py                |    3 +
>  4 files changed, 2197 insertions(+)
>  create mode 100644 docs/devel/qapi-golang-code-gen.rst
>  create mode 100644 scripts/qapi/golang.py

This is series adds a backend that slots in cleanly, i.e. without any
changes to the core.  That makes it as low-risk to merge as it gets.

I'd like an Acked-by for the generated Go from someone familiar the kind
of software that could use it.

The Go backend is a single golang.py, which generates:

* Types:
  alternates.go enums.go structs.go unions.go

* Commands:
  commands.go

* Events:
  events.go

It doesn't generate visitors or introspection code.

Correct?

The existing C backend generates code for

* Types (types.py):
  qapi-builtin-types.[ch]
  qapi-types.[ch] qapi-types-*.[ch]

* Visitors (visit.py):

  qapi-builtin-visit.h
  qapi-visit.[ch] qapi-visit-*.[ch]

* Commands (commands.py):

  qapi-init-commands.h
  qapi-commands.[ch] qapi-commands-*.[ch]

* Events (events.py):

  qapi-emit-events.h
  qapi-events.[ch] qapi-events-*.[ch]

* Introspection (introspect.py):

  qapi-introspect.h

The -* files are all one pair of files per module (the things pulled in
with include directives), if any.  We do this to avoid "touch the QAPI
schema, recompile the world."

The generated Go is monolithic.  No "recompile the world" problem with
Go?

golang.py is somewhat big.  Whether splitting it up along the lines of
the C backend would improve things I can't say.  No need to worry about
that now.
Victor Toso Jan. 14, 2025, 9:44 a.m. UTC | #2
Hi,

On Mon, Jan 13, 2025 at 01:52:25PM +0100, Markus Armbruster wrote:
> Victor Toso <victortoso@redhat.com> writes:
> 
> > This patch series intent is to introduce a generator that produces a Go
> > module for Go applications to interact over QMP with QEMU.
> >
> > The initial Goal is to have a Go module that works as intended and can
> > be improved upon. I'd consider initial releases to be alpha while we
> > work with utilities tools and libraries on top of this.
> >
> > The generated code should reside in a separated Git repository, similar
> > to python-qemu-qmp.
> >
> > Applications should be able to consume this under qemu.org
> > namespace (e.g: import "qemu.org/go/qemu"), see Daniel's suggestion:
> > https://lists.gnu.org/archive/html/qemu-devel/2023-09/msg07024.html
> >
> > This is the third iteration:
> > v2: https://lists.gnu.org/archive/html/qemu-devel/2023-10/msg04785.html
> >
> > I've pushed this series in my gitlab fork:
> > https://gitlab.com/victortoso/qapi-go/
> >
> > The fork contains some tests, including tests that were generated from
> > QAPI's own examples from another generator created for testing, if you
> > are interested in it:
> > https://lists.gnu.org/archive/html/qemu-devel/2023-09/msg04946.html
> >
> > I've generated the qapi-go module over each commit of this series, see:
> > https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v3-by-patch
> >
> > I've also generated the qapi-go module over QEMU tags: v9.1.0, v9.2.0:
> > https://gitlab.com/victortoso/qapi-go/-/commits/qapi-golang-v3-by-tags
> >
> > --
> >
> > Sorry that its been awhile between v2 and v3, I had to prioritize other
> > things. I hope to get this back on track in 2025.
> >
> > Cheers,
> > Victor
> >
> > * Changes:
> >
> > On generated go:
> >  - the output should be formatted as gofmt/goimports tools (Daniel)
> >
> >  - Included QAPI's documentation too (Daniel), see:
> >    https://lists.gnu.org/archive/html/qemu-devel/2024-11/msg01621.html
> >     
> >  - Commands and Events should Marshal directly (Andrea)
> >
> > On python script:
> >  - rebased: now uses QAPISchemaBranches, QAPISchemaAlternatives
> >
> >  - use textwrap as much as possible (Andrea)
> >
> >  - lots of changes to make the output like gofmt does
> >
> > Victor Toso (8):
> >   qapi: golang: Generate enum type
> >   qapi: golang: Generate alternate types
> >   qapi: golang: Generate struct types
> >   qapi: golang: structs: Address nullable members
> >   qapi: golang: Generate union type
> >   qapi: golang: Generate event type
> >   qapi: golang: Generate command type
> >   docs: add notes on Golang code generator
> >
> >  docs/devel/index-build.rst          |    1 +
> >  docs/devel/qapi-golang-code-gen.rst |  548 +++++++++
> >  scripts/qapi/golang.py              | 1645 +++++++++++++++++++++++++++
> >  scripts/qapi/main.py                |    3 +
> >  4 files changed, 2197 insertions(+)
> >  create mode 100644 docs/devel/qapi-golang-code-gen.rst
> >  create mode 100644 scripts/qapi/golang.py
> 
> This is series adds a backend that slots in cleanly, i.e. without any
> changes to the core.  That makes it as low-risk to merge as it gets.
> 
> I'd like an Acked-by for the generated Go from someone familiar the kind
> of software that could use it.
> 
> The Go backend is a single golang.py, which generates:
> 
> * Types:
>   alternates.go enums.go structs.go unions.go
> 
> * Commands:
>   commands.go
> 
> * Events:
>   events.go
> 
> It doesn't generate visitors or introspection code.
> 
> Correct?

Correct. I've actually thought about following what we did with
libvirt-go-module which appends _generated to the files that are
generated, meaning that we would also have files that are without
_generated in the name, with custoam/manual code related to that
file.

Either way, that does not change the Go module (does not break
API/ABI) so we can customize it if needed at a later time.
 
> The existing C backend generates code for
> 
> * Types (types.py):
>   qapi-builtin-types.[ch]
>   qapi-types.[ch] qapi-types-*.[ch]
> 
> * Visitors (visit.py):
> 
>   qapi-builtin-visit.h
>   qapi-visit.[ch] qapi-visit-*.[ch]
> 
> * Commands (commands.py):
> 
>   qapi-init-commands.h
>   qapi-commands.[ch] qapi-commands-*.[ch]
> 
> * Events (events.py):
> 
>   qapi-emit-events.h
>   qapi-events.[ch] qapi-events-*.[ch]
> 
> * Introspection (introspect.py):
> 
>   qapi-introspect.h
> 
> The -* files are all one pair of files per module (the things pulled in
> with include directives), if any.  We do this to avoid "touch the QAPI
> schema, recompile the world."
> 
> The generated Go is monolithic.  No "recompile the world" problem with
> Go?

From my experience, Go compiler only builds when the .go file
changes. If the QAPI changes and the generator outputs different
.go files, it'll recompile at the project using it.
 
> golang.py is somewhat big.  Whether splitting it up along the lines of
> the C backend would improve things I can't say.  No need to worry about
> that now.

Yes, I'm not super experienced with python. I'm all ears to any
suggestions on how to organize the source better. I'm happy to do
it as a follow-up.

Cheers,
Victor