From patchwork Fri Feb 14 20:29:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8C22C02198 for ; Fri, 14 Feb 2025 20:31:25 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2K1-0003ye-F8; Fri, 14 Feb 2025 15:30:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2K0-0003yO-ML for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:00 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2Jy-0001lB-RU for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739564998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z9tecBY1ateQFfirQNGI5dKfCjw9CG3z9DtUVhLANyg=; b=e7Cj5yJVrQk3XhFFYnswPP/rnWISGC4ONTGoU3YXWD/bJc4QnEaLfUnQcpT97WgVjHNMJs nFIsAhpYADnRkfLfne3S5EXxGkmJRXgIv0q3ueb1DX1O3TLGuxJWy/QZ2QINynooNXW6BI I7RlxvQPdLoH41HBXKUGruBCBzRWtDw= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-39-o1ciReYhPh2am_39yQfxUw-1; Fri, 14 Feb 2025 15:29:56 -0500 X-MC-Unique: o1ciReYhPh2am_39yQfxUw-1 X-Mimecast-MFC-AGG-ID: o1ciReYhPh2am_39yQfxUw_1739564995 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CB960180036F for ; Fri, 14 Feb 2025 20:29:55 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D6E3C180056F; Fri, 14 Feb 2025 20:29:51 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 01/11] qapi: golang: first level unmarshalling type Date: Fri, 14 Feb 2025 21:29:34 +0100 Message-ID: <20250214202944.69897-2-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This first patch introduces protocol.go. It introduces the Message Go struct type that can unmarshall any QMP message. It does not handle deeper than 1st layer of the JSON object, that is, with: 1. { "execute": "query-machines", "arguments": { "compat-props": true } } 2. { "event": "BALLOON_CHANGE", "data": { "actual": 944766976 }, "timestamp": { "seconds": 1267020223, "microseconds": 435656 } } We will be able to know it is a query-machine command or a balloon-change event. Specific data type to handle arguments/data will be introduced further in the series. This patch also introduces the Visitor skeleton with a proper write() function to copy-over the protocol.go to the target destination. Note, you can execute any patch of this series with: python3 ./scripts/qapi-gen.py -o /tmp/out qapi/qapi-schema.json Signed-off-by: Victor Toso --- scripts/qapi/golang/__init__.py | 0 scripts/qapi/golang/golang.py | 135 ++++++++++++++++++++++++++++++++ scripts/qapi/golang/protocol.go | 48 ++++++++++++ scripts/qapi/main.py | 2 + 4 files changed, 185 insertions(+) create mode 100644 scripts/qapi/golang/__init__.py create mode 100644 scripts/qapi/golang/golang.py create mode 100644 scripts/qapi/golang/protocol.go diff --git a/scripts/qapi/golang/__init__.py b/scripts/qapi/golang/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py new file mode 100644 index 0000000000..333741b47b --- /dev/null +++ b/scripts/qapi/golang/golang.py @@ -0,0 +1,135 @@ +""" +Golang QAPI generator +""" + +# Copyright (c) 2025 Red Hat Inc. +# +# Authors: +# Victor Toso +# +# This work is licensed under the terms of the GNU GPL, version 2. +# See the COPYING file in the top-level directory. + +# Just for type hint on self +from __future__ import annotations + +import os, shutil +from typing import List, Optional + +from ..schema import ( + QAPISchema, + QAPISchemaBranches, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaIfCond, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaType, + QAPISchemaVariants, + QAPISchemaVisitor, +) +from ..source import QAPISourceInfo + + +def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: + vis = QAPISchemaGenGolangVisitor(prefix) + schema.visit(vis) + vis.write(output_dir) + + +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): + # pylint: disable=too-many-arguments + def __init__(self, _: str): + super().__init__() + gofiles = ("protocol.go",) + self.schema: QAPISchema + self.golang_package_name = "qapi" + self.duplicate = list(gofiles) + + def visit_begin(self, schema: QAPISchema) -> None: + self.schema = schema + + def visit_end(self) -> None: + del self.schema + + def visit_object_type( + self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + branches: Optional[QAPISchemaBranches], + ) -> None: + pass + + def visit_alternate_type( + self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants, + ) -> None: + pass + + def visit_enum_type( + self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str], + ) -> None: + pass + + def visit_array_type( + self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + element_type: QAPISchemaType, + ) -> None: + pass + + def visit_command( + self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool, + coroutine: bool, + ) -> None: + pass + + def visit_event( + self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + boxed: bool, + ) -> None: + pass + + def write(self, outdir: str) -> None: + godir = "go" + targetpath = os.path.join(outdir, godir) + os.makedirs(targetpath, exist_ok=True) + + # Content to be copied over + srcdir = os.path.dirname(os.path.realpath(__file__)) + for filename in self.duplicate: + srcpath = os.path.join(srcdir, filename) + dstpath = os.path.join(targetpath, filename) + shutil.copyfile(srcpath, dstpath) diff --git a/scripts/qapi/golang/protocol.go b/scripts/qapi/golang/protocol.go new file mode 100644 index 0000000000..4ff8d2f3fb --- /dev/null +++ b/scripts/qapi/golang/protocol.go @@ -0,0 +1,48 @@ +/* + * Copyright 2025 Red Hat, Inc. + * SPDX-License-Identifier: MIT-0 + * + * Authors: + * Victor Toso + * Daniel P. Berrange + */ +package qapi + +import ( + "encoding/json" + "time" +) + +/* Union of data for command, response, error, or event, + * since when receiving we don't know upfront which we + * must deserialize */ +type Message struct { + QMP *json.RawMessage `json:"QMP,omitempty"` + Execute string `json:"execute,omitempty"` + ExecOOB string `json:"exec-oob,omitempty"` + Event string `json:"event,omitempty"` + Error *json.RawMessage `json:"error,omitempty"` + Return *json.RawMessage `json:"return,omitempty"` + ID string `json:"id,omitempty"` + Timestamp *Timestamp `json:"timestamp,omitempty"` + Data *json.RawMessage `json:"data,omitempty"` + Arguments *json.RawMessage `json:"arguments,omitempty"` +} + +type QAPIError struct { + Class string `json:"class"` + Description string `json:"desc"` +} + +func (err *QAPIError) Error() string { + return err.Description +} + +type Timestamp struct { + Seconds int `json:"seconds"` + MicroSeconds int `json:"microseconds"` +} + +func (t *Timestamp) AsTime() time.Time { + return time.Unix(int64(t.Seconds), int64(t.MicroSeconds)*1000) +} diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py index 324081b9fc..af315c1ad1 100644 --- a/scripts/qapi/main.py +++ b/scripts/qapi/main.py @@ -16,6 +16,7 @@ from .error import QAPIError from .events import gen_events from .features import gen_features +from .golang import golang from .introspect import gen_introspect from .schema import QAPISchema from .types import gen_types @@ -55,6 +56,7 @@ def generate(schema_file: str, gen_commands(schema, output_dir, prefix, gen_tracing) gen_events(schema, output_dir, prefix) gen_introspect(schema, output_dir, prefix, unmask) + golang.gen_golang(schema, output_dir, prefix) def main() -> int: From patchwork Fri Feb 14 20:29:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C4B4C021A4 for ; Fri, 14 Feb 2025 20:31:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2K7-00043g-Lb; Fri, 14 Feb 2025 15:30:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2K6-00041h-13 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:06 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2K3-0001lb-1R for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565001; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6xqYsVDH8oWpPyMWyIgRluzniD956Fyoitoj7hGAZ6g=; b=NZzJt339NpG1VuE6YMIrezkEMHQnUZpa+HP/RDxHeI7eIvAYGN8zvkJlboh3Wp2xRhe4L8 tKHixpMMcZVQfkH9nElSW+GqYNVJrEmBK/nnhWq7ik9XJecFenodlJmCtyF6pLtlq07b3+ 6mOOReHbJhiB/GSmEHnUHLxCER8BIyk= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-388-4pNR8iaUMoqsSa4XiRNFYw-1; Fri, 14 Feb 2025 15:29:59 -0500 X-MC-Unique: 4pNR8iaUMoqsSa4XiRNFYw-1 X-Mimecast-MFC-AGG-ID: 4pNR8iaUMoqsSa4XiRNFYw_1739564998 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D514218004A7 for ; Fri, 14 Feb 2025 20:29:58 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9186A1800352; Fri, 14 Feb 2025 20:29:56 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 02/11] qapi: golang: Generate enum type Date: Fri, 14 Feb 2025 21:29:35 +0100 Message-ID: <20250214202944.69897-3-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch handles QAPI enum types and generates its equivalent in Go. We sort the output based on enum's type name. Enums are being handled as strings in Golang. 1. For each QAPI enum, we will define a string type in Go to be the assigned type of this specific enum. 2. Naming: CamelCase will be used in any identifier that we want to export, which is everything. Example: qapi: | ## | # @DisplayProtocol: | # | # Display protocols which support changing password options. | # | # Since: 7.0 | ## | { 'enum': 'DisplayProtocol', | 'data': [ 'vnc', 'spice' ] } go: | // Display protocols which support changing password options. | // | // Since: 7.0 | type DisplayProtocol string | | const ( | DisplayProtocolVnc DisplayProtocol = "vnc" | DisplayProtocolSpice DisplayProtocol = "spice" | ) Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 185 +++++++++++++++++++++++++++++++++- 1 file changed, 183 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index 333741b47b..f074ec1f6f 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -13,7 +13,7 @@ # Just for type hint on self from __future__ import annotations -import os, shutil +import os, shutil, textwrap from typing import List, Optional from ..schema import ( @@ -30,6 +30,65 @@ ) from ..source import QAPISourceInfo +TEMPLATE_GENERATED_HEADER = """ +/* + * Copyright 2025 Red Hat, Inc. + * SPDX-License-Identifier: (MIT-0 and GPL-2.0-or-later) + */ + +/**************************************************************************** + * THIS CODE HAS BEEN GENERATED. DO NOT CHANGE IT DIRECTLY * + ****************************************************************************/ +package {package_name} +""" + +TEMPLATE_GO_IMPORTS = """ +import ( +{imports} +) +""" + +TEMPLATE_ENUM = """ +type {name} string + +const ( +{fields} +) +""" + + +# Takes the documentation object of a specific type and returns +# that type's documentation and its member's docs. +def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): + if doc is None: + return "", {} + + cmt = "// " + fmt = textwrap.TextWrapper( + width=70, initial_indent=cmt, subsequent_indent=cmt + ) + main = fmt.fill(doc.body.text) + + for section in doc.sections: + # TODO is not a relevant section to Go applications + if section.tag in ["TODO"]: + continue + + if main != "": + # Give empty line as space for the tag. + main += "\n//\n" + + tag = "" if section.tag is None else f"{section.tag}: " + text = section.text.replace(" ", " ") + main += fmt.fill(f"{tag}{text}") + + fields = {} + for key, value in doc.args.items(): + if len(value.text) > 0: + fields[key] = " ".join(value.text.replace("\n", " ").split()) + + return main, fields + def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: vis = QAPISchemaGenGolangVisitor(prefix) @@ -37,20 +96,90 @@ def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: vis.write(output_dir) +def qapi_to_field_name_enum(name: str) -> str: + return name.title().replace("-", "") + + +def fetch_indent_blocks_over_enum_with_docs( + name: str, members: List[QAPISchemaEnumMember], docfields: Dict[str, str] +) -> Tuple[int]: + maxname = 0 + blocks: List[int] = [0] + for member in members: + # For simplicity, every time we have doc, we add a new indent block + hasdoc = member.name is not None and member.name in docfields + + enum_name = f"{name}{qapi_to_field_name_enum(member.name)}" + maxname = ( + max(maxname, len(enum_name)) if not hasdoc else len(enum_name) + ) + + if hasdoc: + blocks.append(maxname) + else: + blocks[-1] = maxname + + return blocks + + +def generate_content_from_dict(data: dict[str, str]) -> str: + content = "" + + for name in sorted(data): + content += data[name] + + return content.replace("\n\n\n", "\n\n") + + +def generate_template_imports(words: List[str]) -> str: + if len(words) == 0: + return "" + + if len(words) == 1: + return '\nimport "{words[0]}"\n' + + return TEMPLATE_GO_IMPORTS.format( + imports="\n".join(f'\t"{w}"' for w in words) + ) + + class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): # pylint: disable=too-many-arguments def __init__(self, _: str): super().__init__() gofiles = ("protocol.go",) + # Map each qapi type to the necessary Go imports + types = { + "enum": [], + } + self.schema: QAPISchema self.golang_package_name = "qapi" self.duplicate = list(gofiles) + self.enums: dict[str, str] = {} + self.docmap = {} + + self.types = dict.fromkeys(types, "") + self.types_import = types def visit_begin(self, schema: QAPISchema) -> None: self.schema = schema + # iterate once in schema.docs to map doc objects to its name + for doc in schema.docs: + if doc.symbol is None: + continue + self.docmap[doc.symbol] = doc + + for qapitype, imports in self.types_import.items(): + self.types[qapitype] = TEMPLATE_GENERATED_HEADER[1:].format( + package_name=self.golang_package_name + ) + self.types[qapitype] += generate_template_imports(imports) + def visit_end(self) -> None: del self.schema + self.types["enum"] += generate_content_from_dict(self.enums) def visit_object_type( self, @@ -83,7 +212,51 @@ def visit_enum_type( members: List[QAPISchemaEnumMember], prefix: Optional[str], ) -> None: - pass + assert name not in self.enums + doc = self.docmap.get(name, None) + maindoc, docfields = qapi_to_golang_struct_docs(doc) + + # The logic below is to generate QAPI enums as blocks of Go consts + # each with its own type for type safety inside Go applications. + # + # Block of const() blocks are vertically indented so we have to + # first iterate over all names to calculate space between + # $var_name and $var_type. This is achieved by helper function + # @fetch_indent_blocks_over_enum_with_docs() + # + # A new indentation block is defined by empty line or a comment. + + indent_block = iter( + fetch_indent_blocks_over_enum_with_docs(name, members, docfields) + ) + maxname = next(indent_block) + fields = "" + for index, member in enumerate(members): + # For simplicity, every time we have doc, we go to next indent block + hasdoc = member.name is not None and member.name in docfields + + if hasdoc: + maxname = next(indent_block) + + enum_name = f"{name}{qapi_to_field_name_enum(member.name)}" + name2type = " " * (maxname - len(enum_name) + 1) + + if hasdoc: + docstr = ( + textwrap.TextWrapper(width=80) + .fill(docfields[member.name]) + .replace("\n", "\n\t// ") + ) + fields += f"""\t// {docstr}\n""" + + fields += f"""\t{enum_name}{name2type}{name} = "{member.name}"\n""" + + if maindoc != "": + maindoc = f"\n{maindoc}" + + self.enums[name] = maindoc + TEMPLATE_ENUM.format( + name=name, fields=fields[:-1] + ) def visit_array_type( self, @@ -133,3 +306,11 @@ def write(self, outdir: str) -> None: srcpath = os.path.join(srcdir, filename) dstpath = os.path.join(targetpath, filename) shutil.copyfile(srcpath, dstpath) + + # Types to be generated + for qapitype, content in self.types.items(): + gofile = f"gen_type_{qapitype}.go" + pathname = os.path.join(targetpath, gofile) + + with open(pathname, "w", encoding="utf8") as outfile: + outfile.write(content) From patchwork Fri Feb 14 20:29:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5A9E1C02198 for ; Fri, 14 Feb 2025 20:30:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2KC-000471-FA; Fri, 14 Feb 2025 15:30:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KA-00045E-MG for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:10 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2K8-0001yO-0i for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4n33rQfGaBnwnQltob59qtttUzejZbi5C/XifU+Ps0I=; b=I8yuLUJI9TvP2YPx1rC7hko73Nkrv1rrGSau5QBIEvOic1b77Ou1gGM19oAss0yAlsRmVn qzuSYjYQg+ee9ONMhLXDXbBMDJtvTGjoXxvABQH3zFJtOlVXs1ZAeAHZygBwVMJq8wyGSv DQyGfYkI+R+BWj0Yz+PU4fMEkUh7lvs= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-220-zxSZQKp0P46P0RH8ZeDzDA-1; Fri, 14 Feb 2025 15:30:02 -0500 X-MC-Unique: zxSZQKp0P46P0RH8ZeDzDA-1 X-Mimecast-MFC-AGG-ID: zxSZQKp0P46P0RH8ZeDzDA_1739565001 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D0F5B180087C for ; Fri, 14 Feb 2025 20:30:01 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8B39B180056F; Fri, 14 Feb 2025 20:29:59 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 03/11] qapi: golang: Generate alternate types Date: Fri, 14 Feb 2025 21:29:36 +0100 Message-ID: <20250214202944.69897-4-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch handles QAPI alternate types and generates data structures in Go that handles it. Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. 1. Over the wire, we need to infer underlying value by its type 2. Pointer to types are mapped as optional. Absent value can be a valid value. 3. We use Go's standard 'encoding/json' library with its Marshal and Unmarshal interfaces. 4. As an exceptional but valid case, there are types that accept JSON NULL as value. Due to limitations with Go's standard library (point 3) combined with Absent being a possibility (point 2), we translante NULL values to a boolean field called 'IsNull'. See the second example and docs/devel/qapi-golang-code-gen.rst under Alternate section. * First example: qapi: | ## | # @BlockdevRef: | # | # Reference to a block device. | # | # @definition: defines a new block device inline | # | # @reference: references the ID of an existing block device | # | # Since: 2.9 | ## | { 'alternate': 'BlockdevRef', | 'data': { 'definition': 'BlockdevOptions', | 'reference': 'str' } } go: | // Reference to a block device. | // | // Since: 2.9 | type BlockdevRef struct { | // defines a new block device inline | Definition *BlockdevOptions | // references the ID of an existing block device | Reference *string | } | | func (s BlockdevRef) MarshalJSON() ([]byte, error) { | ... | } | | func (s *BlockdevRef) UnmarshalJSON(data []byte) error { | ... | } usage: | input := `{"driver":"qcow2","data-file":"/some/place/my-image"}` | k := BlockdevRef{} | err := json.Unmarshal([]byte(input), &k) | if err != nil { | panic(err) | } | // *k.Definition.Qcow2.DataFile.Reference == "/some/place/my-image" * Second example: qapi: | { 'alternate': 'StrOrNull', | 'data': { 's': 'str', | 'n': 'null' } } | // This is a string value or the explicit lack of a string (null | // pointer in C). Intended for cases when 'optional absent' already | // has a different meaning. | // | // Since: 2.10 | type StrOrNull struct { | // the string value | S *string | // no string value | IsNull bool | } | | // Helper function to get its underlying Go value or absent of value | func (s *StrOrNull) ToAnyOrAbsent() (any, bool) { | ... | } | | func (s StrOrNull) MarshalJSON() ([]byte, error) { | ... | } | | func (s *StrOrNull) UnmarshalJSON(data []byte) error { | ... | } Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 306 +++++++++++++++++++++++++++++++++- scripts/qapi/golang/utils.go | 26 +++ 2 files changed, 329 insertions(+), 3 deletions(-) create mode 100644 scripts/qapi/golang/utils.go diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index f074ec1f6f..aa1a18a501 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -14,10 +14,11 @@ from __future__ import annotations import os, shutil, textwrap -from typing import List, Optional +from typing import List, Optional, Tuple from ..schema import ( QAPISchema, + QAPISchemaAlternateType, QAPISchemaBranches, QAPISchemaEnumMember, QAPISchemaFeature, @@ -30,6 +31,8 @@ ) from ..source import QAPISourceInfo +FOUR_SPACES = " " + TEMPLATE_GENERATED_HEADER = """ /* * Copyright 2025 Red Hat, Inc. @@ -56,6 +59,57 @@ ) """ +TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL = """ + // Check for json-null first + if string(data) == "null" {{ + return errors.New(`null not supported for {name}`) + }}""" + +TEMPLATE_ALTERNATE_NULLABLE_CHECK = """ + }} else if s.{var_name} != nil {{ + return *s.{var_name}, false""" + +TEMPLATE_ALTERNATE_MARSHAL_CHECK = """ + if s.{var_name} != nil {{ + return json.Marshal(s.{var_name}) + }} else """ + +TEMPLATE_ALTERNATE_UNMARSHAL_CHECK = """ + // Check for {var_type} + {{ + s.{var_name} = new({var_type}) + if err := strictDecode(s.{var_name}, data); err == nil {{ + return nil + }} + s.{var_name} = nil + }} + +""" + +TEMPLATE_ALTERNATE_NULLABLE_MARSHAL_CHECK = """ + if s.IsNull { + return []byte("null"), nil + } else """ + +TEMPLATE_ALTERNATE_NULLABLE_UNMARSHAL_CHECK = """ + // Check for json-null first + if string(data) == "null" { + s.IsNull = true + return nil + }""" + +TEMPLATE_ALTERNATE_METHODS = """ +func (s {name}) MarshalJSON() ([]byte, error) {{ +{marshal_check_fields} + return {marshal_return_default} +}} + +func (s *{name}) UnmarshalJSON(data []byte) error {{ +{unmarshal_check_fields} + return fmt.Errorf("Can't convert to {name}: %s", string(data)) +}} +""" + # Takes the documentation object of a specific type and returns # that type's documentation and its member's docs. @@ -96,10 +150,88 @@ def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: vis.write(output_dir) +def qapi_to_field_name(name: str) -> str: + return name.title().replace("_", "").replace("-", "") + + def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") +def qapi_schema_type_to_go_type(qapitype: str) -> str: + schema_types_to_go = { + "str": "string", + "null": "nil", + "bool": "bool", + "number": "float64", + "size": "uint64", + "int": "int64", + "int8": "int8", + "int16": "int16", + "int32": "int32", + "int64": "int64", + "uint8": "uint8", + "uint16": "uint16", + "uint32": "uint32", + "uint64": "uint64", + "any": "any", + "QType": "QType", + } + + prefix = "" + if qapitype.endswith("List"): + prefix = "[]" + qapitype = qapitype[:-4] + + qapitype = schema_types_to_go.get(qapitype, qapitype) + return prefix + qapitype + + +# Helper for Alternate generation +def qapi_field_to_alternate_go_field( + member_name: str, type_name: str +) -> Tuple[str, str, str]: + # Nothing to generate on null types. We update some + # variables to handle json-null on marshalling methods. + if type_name == "null": + return "IsNull", "bool", "" + + # On Alternates, fields are optional represented in Go as pointer + return ( + qapi_to_field_name(member_name), + qapi_schema_type_to_go_type(type_name), + "*", + ) + + +def fetch_indent_blocks_over_args( + args: List[dict[str:str]], +) -> Tuple[int, int]: + maxname, maxtype = 0, 0 + blocks: tuple(int, int) = [] + for arg in args: + if "comment" in arg or "doc" in arg: + blocks.append((maxname, maxtype)) + maxname, maxtype = 0, 0 + + if "comment" in arg: + # They are single blocks + continue + + if "type" not in arg: + # Embed type are on top of the struct and the following + # fields do not consider it for formatting + blocks.append((maxname, maxtype)) + maxname, maxtype = 0, 0 + continue + + maxname = max(maxname, len(arg.get("name", ""))) + maxtype = max(maxtype, len(arg.get("type", ""))) + + blocks.append((maxname, maxtype)) + return blocks + + def fetch_indent_blocks_over_enum_with_docs( name: str, members: List[QAPISchemaEnumMember], docfields: Dict[str, str] ) -> Tuple[int]: @@ -122,6 +254,137 @@ def fetch_indent_blocks_over_enum_with_docs( return blocks +# Helper function for boxed or self contained structures. +def generate_struct_type( + type_name, + type_doc: str = "", + args: List[dict[str:str]] = None, + indent: int = 0, +) -> str: + base_indent = FOUR_SPACES * indent + + with_type = "" + if type_name != "": + with_type = f"\n{base_indent}type {type_name}" + + if type_doc != "": + # Append line jump only if type_doc exists + type_doc = f"\n{type_doc}" + + if args is None: + # No args, early return + return f"""{type_doc}{with_type} struct{{}}""" + + # The logic below is to generate fields of the struct. + # We have to be mindful of the different indentation possibilities between + # $var_name $var_type $var_tag that are vertically indented with gofmt. + # + # So, we first have to iterate over all args and find all indent blocks + # by calculating the spaces between (1) member and type and between (2) + # the type and tag. (1) and (2) is the tuple present in List returned + # by the helper function fetch_indent_blocks_over_args. + inner_indent = base_indent + FOUR_SPACES + doc_indent = inner_indent + "// " + fmt = textwrap.TextWrapper( + width=70, initial_indent=doc_indent, subsequent_indent=doc_indent + ) + + indent_block = iter(fetch_indent_blocks_over_args(args)) + maxname, maxtype = next(indent_block) + members = " {\n" + for index, arg in enumerate(args): + if "comment" in arg: + maxname, maxtype = next(indent_block) + members += f""" // {arg["comment"]}\n""" + # comments are single blocks, so we can skip to next arg + continue + + name2type = "" + if "doc" in arg: + maxname, maxtype = next(indent_block) + members += fmt.fill(arg["doc"]) + members += "\n" + + name = arg["name"] + if "type" in arg: + namelen = len(name) + name2type = " " * max(1, (maxname - namelen + 1)) + + type2tag = "" + if "tag" in arg: + typelen = len(arg["type"]) + type2tag = " " * max(1, (maxtype - typelen + 1)) + + gotype = arg.get("type", "") + tag = arg.get("tag", "") + members += ( + f"""{inner_indent}{name}{name2type}{gotype}{type2tag}{tag}\n""" + ) + + members += f"{base_indent}}}\n" + return f"""{type_doc}{with_type} struct{members}""" + + +def generate_template_alternate( + self: QAPISchemaGenGolangVisitor, + name: str, + variants: Optional[QAPISchemaVariants], +) -> str: + args: List[dict[str:str]] = [] + nullable = name in self.accept_null_types + if nullable: + # Examples in QEMU QAPI schema: StrOrNull and BlockdevRefOrNull + marshal_return_default = """[]byte("{}"), nil""" + marshal_check_fields = TEMPLATE_ALTERNATE_NULLABLE_MARSHAL_CHECK[1:] + unmarshal_check_fields = TEMPLATE_ALTERNATE_NULLABLE_UNMARSHAL_CHECK + else: + marshal_return_default = f'nil, errors.New("{name} has empty fields")' + marshal_check_fields = "" + unmarshal_check_fields = ( + TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL.format(name=name) + ) + + doc = self.docmap.get(name, None) + content, docfields = qapi_to_golang_struct_docs(doc) + if variants: + for var in variants.variants: + var_name, var_type, isptr = qapi_field_to_alternate_go_field( + var.name, var.type.name + ) + args.append( + { + "name": f"{var_name}", + "type": f"{isptr}{var_type}", + "doc": docfields.get(var.name, ""), + } + ) + # Null is special, handled first + if var.type.name == "null": + assert nullable + continue + + skip_indent = 1 + len(FOUR_SPACES) + if marshal_check_fields == "": + skip_indent = 1 + marshal_check_fields += TEMPLATE_ALTERNATE_MARSHAL_CHECK[ + skip_indent: + ].format(var_name=var_name) + unmarshal_check_fields += TEMPLATE_ALTERNATE_UNMARSHAL_CHECK[ + :-1 + ].format(var_name=var_name, var_type=var_type) + + content += string_to_code(generate_struct_type(name, args=args)) + content += string_to_code( + TEMPLATE_ALTERNATE_METHODS.format( + name=name, + marshal_check_fields=marshal_check_fields[:-6], + marshal_return_default=marshal_return_default, + unmarshal_check_fields=unmarshal_check_fields[1:], + ) + ) + return "\n" + content + + def generate_content_from_dict(data: dict[str, str]) -> str: content = "" @@ -131,6 +394,25 @@ def generate_content_from_dict(data: dict[str, str]) -> str: return content.replace("\n\n\n", "\n\n") +def string_to_code(text: str) -> str: + DOUBLE_BACKTICK = "``" + result = "" + for line in text.splitlines(): + # replace left four spaces with tabs + limit = len(line) - len(line.lstrip()) + result += line[:limit].replace(FOUR_SPACES, "\t") + + # work with the rest of the line + if line[limit : limit + 2] == "//": + # gofmt tool does not like comments with backticks. + result += line[limit:].replace(DOUBLE_BACKTICK, '"') + else: + result += line[limit:] + result += "\n" + + return result + + def generate_template_imports(words: List[str]) -> str: if len(words) == 0: return "" @@ -147,9 +429,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): # pylint: disable=too-many-arguments def __init__(self, _: str): super().__init__() - gofiles = ("protocol.go",) + gofiles = ("protocol.go", "utils.go") # Map each qapi type to the necessary Go imports types = { + "alternate": ["encoding/json", "errors", "fmt"], "enum": [], } @@ -157,6 +440,8 @@ def __init__(self, _: str): self.golang_package_name = "qapi" self.duplicate = list(gofiles) self.enums: dict[str, str] = {} + self.alternates: dict[str, str] = {} + self.accept_null_types = [] self.docmap = {} self.types = dict.fromkeys(types, "") @@ -165,6 +450,17 @@ def __init__(self, _: str): def visit_begin(self, schema: QAPISchema) -> None: self.schema = schema + # We need to be aware of any types that accept JSON NULL + for name, entity in self.schema._entity_dict.items(): + if not isinstance(entity, QAPISchemaAlternateType): + # Assume that only Alternate types accept JSON NULL + continue + + for var in entity.alternatives.variants: + if var.type.name == "null": + self.accept_null_types.append(name) + break + # iterate once in schema.docs to map doc objects to its name for doc in schema.docs: if doc.symbol is None: @@ -180,6 +476,7 @@ def visit_begin(self, schema: QAPISchema) -> None: def visit_end(self) -> None: del self.schema self.types["enum"] += generate_content_from_dict(self.enums) + self.types["alternate"] += generate_content_from_dict(self.alternates) def visit_object_type( self, @@ -201,7 +498,10 @@ def visit_alternate_type( features: List[QAPISchemaFeature], variants: QAPISchemaVariants, ) -> None: - pass + assert name not in self.alternates + self.alternates[name] = generate_template_alternate( + self, name, variants + ) def visit_enum_type( self, diff --git a/scripts/qapi/golang/utils.go b/scripts/qapi/golang/utils.go new file mode 100644 index 0000000000..f00c0a5d83 --- /dev/null +++ b/scripts/qapi/golang/utils.go @@ -0,0 +1,26 @@ +/* + * Copyright 2025 Red Hat, Inc. + * SPDX-License-Identifier: MIT-0 + * + * Authors: + * Victor Toso + */ +package qapi + +import ( + "encoding/json" + "strings" +) + +// Creates a decoder that errors on unknown Fields +// Returns nil if successfully decoded @from payload to @into type +// Returns error if failed to decode @from payload to @into type +func strictDecode(into interface{}, from []byte) error { + dec := json.NewDecoder(strings.NewReader(string(from))) + dec.DisallowUnknownFields() + + if err := dec.Decode(into); err != nil { + return err + } + return nil +} From patchwork Fri Feb 14 20:29:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C82FAC02198 for ; Fri, 14 Feb 2025 20:31:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2KD-00047A-VQ; Fri, 14 Feb 2025 15:30:14 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KB-00046R-Nd for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2K9-0001ya-8q for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565008; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/gdyGCh2dz9H4adBNlVDJUKRfJK6pdw2YnRUuj8wqQE=; b=fpPhBmQ29PSgampX4Eqz4dRWmOzwycvjKuRJTLMzJxfHIi9QjTgan954a/1zgUuvKh5tOL uc+WP0lNEHIecOMaTHFjJV6JZ1tafDKt7D593qGhnFiforwnzl8w365YWAdYzJlfIsvGsO GpmzY4owZqZuszj3wS1VxIcV34D28A0= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-360-w7VgUASBM-eqejd_IwSPoA-1; Fri, 14 Feb 2025 15:30:06 -0500 X-MC-Unique: w7VgUASBM-eqejd_IwSPoA-1 X-Mimecast-MFC-AGG-ID: w7VgUASBM-eqejd_IwSPoA_1739565005 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 06152180087B for ; Fri, 14 Feb 2025 20:30:05 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9D0D61800352; Fri, 14 Feb 2025 20:30:02 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 04/11] qapi: golang: Generate struct types Date: Fri, 14 Feb 2025 21:29:37 +0100 Message-ID: <20250214202944.69897-5-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL=1.623 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch handles QAPI struct types and generates the equivalent types in Go. The following patch adds extra logic when a member of the struct has a Type that can take JSON Null value (e.g: StrOrNull in QEMU) The highlights of this implementation are: 1. Generating a Go struct that requires a @base type, the @base type fields are copied over to the Go struct. The advantage of this approach is to not have embed structs in any of the QAPI types. Note that embedding a @base type is recursive, that is, if the @base type has a @base, all of those fields will be copied over. 2. About the Go struct's fields: i) They can be either by Value or Reference. ii) Every field that is marked as optional in the QAPI specification are translated to Reference fields in its Go structure. This design decision is the most straightforward way to check if a given field was set or not. Exception only for types that can take JSON Null value. iii) Mandatory fields are always by Value with the exception of QAPI arrays, which are handled by Reference (to a block of memory) by Go. iv) All the fields are named with Uppercase due Golang's export convention. Example: qapi: | ## | # @BlockdevCreateOptionsFile: | # | # Driver specific image creation options for file. | # | # @filename: Filename for the new image file | # | # @size: Size of the virtual disk in bytes | # | # @preallocation: Preallocation mode for the new image (default: off; | # allowed values: off, falloc (if CONFIG_POSIX_FALLOCATE), full | # (if CONFIG_POSIX)) | # | # @nocow: Turn off copy-on-write (valid only on btrfs; default: off) | # | # @extent-size-hint: Extent size hint to add to the image file; 0 for | # not adding an extent size hint (default: 1 MB, since 5.1) | # | # Since: 2.12 | ## | { 'struct': 'BlockdevCreateOptionsFile', | 'data': { 'filename': 'str', | 'size': 'size', | '*preallocation': 'PreallocMode', | '*nocow': 'bool', | '*extent-size-hint': 'size'} } go: | // Driver specific image creation options for file. | // | // Since: 2.12 | type BlockdevCreateOptionsFile struct { | // Filename for the new image file | Filename string `json:"filename"` | // Size of the virtual disk in bytes | Size uint64 `json:"size"` | // Preallocation mode for the new image (default: off; allowed | // values: off, falloc (if CONFIG_POSIX_FALLOCATE), full (if | // CONFIG_POSIX)) | Preallocation *PreallocMode `json:"preallocation,omitempty"` | // Turn off copy-on-write (valid only on btrfs; default: off) | Nocow *bool `json:"nocow,omitempty"` | // Extent size hint to add to the image file; 0 for not adding an | // extent size hint (default: 1 MB, since 5.1) | ExtentSizeHint *uint64 `json:"extent-size-hint,omitempty"` | } Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 193 +++++++++++++++++++++++++++++++++- 1 file changed, 192 insertions(+), 1 deletion(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index aa1a18a501..e8a47b4a1e 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -150,6 +150,14 @@ def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: vis.write(output_dir) +def qapi_name_is_base(name: str) -> bool: + return qapi_name_is_object(name) and name.endswith("-base") + + +def qapi_name_is_object(name: str) -> bool: + return name.startswith("q_obj_") + + def qapi_to_field_name(name: str) -> str: return name.title().replace("_", "").replace("-", "") @@ -158,6 +166,27 @@ def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") +def qapi_to_go_type_name(name: str) -> str: + # We want to keep CamelCase for Golang types. We want to avoid removing + # already set CameCase names while fixing uppercase ones, eg: + # 1) q_obj_SocketAddress_base -> SocketAddressBase + # 2) q_obj_WATCHDOG-arg -> WatchdogArg + + if qapi_name_is_object(name): + # Remove q_obj_ prefix + name = name[6:] + + # Handle CamelCase + words = list(name.replace("_", "-").split("-")) + name = words[0] + if name.islower() or name.isupper(): + name = name.title() + + name += "".join(word.title() for word in words[1:]) + + return name + + def qapi_schema_type_to_go_type(qapitype: str) -> str: schema_types_to_go = { "str": "string", @@ -325,6 +354,131 @@ def generate_struct_type( return f"""{type_doc}{with_type} struct{members}""" +def get_struct_field( + self: QAPISchemaGenGolangVisitor, + qapi_name: str, + qapi_type_name: str, + field_doc: str, + is_optional: bool, + is_variant: bool, +) -> dict[str:str]: + field = qapi_to_field_name(qapi_name) + member_type = qapi_schema_type_to_go_type(qapi_type_name) + + optional = "" + if is_optional: + if member_type not in self.accept_null_types: + optional = ",omitempty" + + # Use pointer to type when field is optional + isptr = "*" if is_optional and member_type[0] not in "*[" else "" + + fieldtag = ( + '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`' + ) + arg = { + "name": f"{field}", + "type": f"{isptr}{member_type}", + "tag": f"{fieldtag}", + } + if field_doc != "": + arg["doc"] = field_doc + + return arg + + +def recursive_base( + self: QAPISchemaGenGolangVisitor, + base: Optional[QAPISchemaObjectType], +) -> List[dict[str:str]]: + fields: List[dict[str:str]] = [] + + if not base: + return fields + + if base.base is not None: + embed_base = self.schema.lookup_entity(base.base.name) + fields = recursive_base(self, embed_base) + + doc = self.docmap.get(base.name, None) + _, docfields = qapi_to_golang_struct_docs(doc) + + for member in base.local_members: + field_doc = docfields.get(member.name, "") + field = get_struct_field( + self, + member.name, + member.type.name, + field_doc, + member.optional, + False, + ) + fields.append(field) + + return fields + + +# Helper function that is used for most of QAPI types +def qapi_to_golang_struct( + self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + __: QAPISchemaIfCond, + ___: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants], + indent: int = 0, + doc_enabled: bool = True, +) -> str: + fields = recursive_base(self, base) + + doc = self.docmap.get(name, None) + type_doc, docfields = qapi_to_golang_struct_docs(doc) + if not doc_enabled: + type_doc = "" + + if members: + for member in members: + field_doc = docfields.get(member.name, "") if doc_enabled else "" + field = get_struct_field( + self, + member.name, + member.type.name, + field_doc, + member.optional, + False, + ) + fields.append(field) + + exists = {} + if variants: + fields.append({"comment": "Variants fields"}) + for variant in variants.variants: + if variant.type.is_implicit(): + continue + + exists[variant.name] = True + field_doc = docfields.get(variant.name, "") if doc_enabled else "" + field = get_struct_field( + self, + variant.name, + variant.type.name, + field_doc, + True, + True, + ) + fields.append(field) + + type_name = qapi_to_go_type_name(name) + content = string_to_code( + generate_struct_type( + type_name, type_doc=type_doc, args=fields, indent=indent + ) + ) + return content + + def generate_template_alternate( self: QAPISchemaGenGolangVisitor, name: str, @@ -434,6 +588,7 @@ def __init__(self, _: str): types = { "alternate": ["encoding/json", "errors", "fmt"], "enum": [], + "struct": ["encoding/json"], } self.schema: QAPISchema @@ -441,6 +596,7 @@ def __init__(self, _: str): self.duplicate = list(gofiles) self.enums: dict[str, str] = {} self.alternates: dict[str, str] = {} + self.structs: dict[str, str] = {} self.accept_null_types = [] self.docmap = {} @@ -477,6 +633,7 @@ def visit_end(self) -> None: del self.schema self.types["enum"] += generate_content_from_dict(self.enums) self.types["alternate"] += generate_content_from_dict(self.alternates) + self.types["struct"] += generate_content_from_dict(self.structs) def visit_object_type( self, @@ -488,7 +645,41 @@ def visit_object_type( members: List[QAPISchemaObjectTypeMember], branches: Optional[QAPISchemaBranches], ) -> None: - pass + # Do not handle anything besides struct. + if ( + name == self.schema.the_empty_object_type.name + or not isinstance(name, str) + or info.defn_meta not in ["struct"] + ): + return + + # Base structs are embed + if qapi_name_is_base(name): + return + + # visit all inner objects as well, they are not going to be + # called by python's generator. + if branches: + for branch in branches.variants: + assert isinstance(branch.type, QAPISchemaObjectType) + self.visit_object_type( + self, + branch.type.name, + branch.type.info, + branch.type.ifcond, + branch.type.base, + branch.type.local_members, + branch.type.branches, + ) + + # Save generated Go code to be written later + if info.defn_meta == "struct": + assert name not in self.structs + self.structs[name] = string_to_code( + qapi_to_golang_struct( + self, name, info, ifcond, features, base, members, branches + ) + ) def visit_alternate_type( self, From patchwork Fri Feb 14 20:29:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA760C02198 for ; Fri, 14 Feb 2025 20:33:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2KM-00048U-BQ; Fri, 14 Feb 2025 15:30:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KH-00047m-Ux for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:18 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KF-00020S-Ag for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565013; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=obSlDwh3TsaJPeAE7rBADduh9KD7yNBRlbve9w1ILBk=; b=U1gOV8SEVUp276cncsRt7ujS0+jYyZbvFJuZHJ1n7bc6JZvsf8zbEXRCcG3/k3hO967s8m cxTTQ2/V+uwkpYa4cLtOZATiU8ey8nwkijchnGW32LjBFD8a7ZttUdKDmt32v1Z9BR2nZZ Wi4MJrSEPWiy0NGWvtlyusyok58yL+k= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-576-_gE6C0o_PeyC8lVHP2T91A-1; Fri, 14 Feb 2025 15:30:09 -0500 X-MC-Unique: _gE6C0o_PeyC8lVHP2T91A-1 X-Mimecast-MFC-AGG-ID: _gE6C0o_PeyC8lVHP2T91A_1739565009 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0DACE180087F for ; Fri, 14 Feb 2025 20:30:09 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8BDFA180087D; Fri, 14 Feb 2025 20:30:05 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 05/11] qapi: golang: structs: Address nullable members Date: Fri, 14 Feb 2025 21:29:38 +0100 Message-ID: <20250214202944.69897-6-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL=1.623 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Explaining why this is needed needs some context, so taking the example of StrOrNull alternate type and considering a simplified struct that has two fields: qapi: | { 'struct': 'MigrationExample', | 'data': { '*label': 'StrOrNull', | 'target': 'StrOrNull' } } We have an optional member 'label' which can have three JSON values: 1. A string: { "target": "a.host.com", "label": "happy" } 2. A null : { "target": "a.host.com", "label": null } 3. Absent : { "target": null} The member 'target' is not optional, hence it can't be absent. A Go struct that contains an optional type that can be JSON Null like 'label' in the example above, will need extra care when Marshaling and Unmarshaling from JSON. This patch handles this very specific case: - It implements the Marshaler interface for these structs to properly handle these values. - It adds the interface AbsentAlternate() and implement it for any Alternate that can be JSON Null. See its uses in map_and_set() Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 290 ++++++++++++++++++++++++++++++++-- 1 file changed, 279 insertions(+), 11 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index e8a47b4a1e..0637bb3e3e 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -59,6 +59,17 @@ ) """ +TEMPLATE_ALTERNATE = """ +// Only implemented on Alternate types that can take JSON NULL as value. +// +// This is a helper for the marshalling code. It should return true only when +// the Alternate is empty (no members are set), otherwise it returns false and +// the member set to be Marshalled. +type AbsentAlternate interface { + ToAnyOrAbsent() (any, bool) +} +""" + TEMPLATE_ALTERNATE_CHECK_INVALID_JSON_NULL = """ // Check for json-null first if string(data) == "null" {{ @@ -98,6 +109,19 @@ return nil }""" +TEMPLATE_ALTERNATE_NULLABLE = """ +func (s *{name}) ToAnyOrAbsent() (any, bool) {{ + if s != nil {{ + if s.IsNull {{ + return nil, false +{absent_check_fields} + }} + }} + + return nil, true +}} +""" + TEMPLATE_ALTERNATE_METHODS = """ func (s {name}) MarshalJSON() ([]byte, error) {{ {marshal_check_fields} @@ -111,6 +135,26 @@ """ +TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL = """ +func (s {type_name}) MarshalJSON() ([]byte, error) {{ + m := make(map[string]any) +{map_members}{map_special} + return json.Marshal(&m) +}} + +func (s *{type_name}) UnmarshalJSON(data []byte) error {{ + tmp := {struct}{{}} + + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + +{set_members}{set_special} + return nil +}} +""" + + # Takes the documentation object of a specific type and returns # that type's documentation and its member's docs. def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): @@ -359,20 +403,30 @@ def get_struct_field( qapi_name: str, qapi_type_name: str, field_doc: str, + within_nullable_struct: bool, is_optional: bool, is_variant: bool, -) -> dict[str:str]: +) -> Tuple[dict[str:str], bool]: field = qapi_to_field_name(qapi_name) member_type = qapi_schema_type_to_go_type(qapi_type_name) + is_nullable = False optional = "" if is_optional: - if member_type not in self.accept_null_types: + if member_type in self.accept_null_types: + is_nullable = True + else: optional = ",omitempty" # Use pointer to type when field is optional isptr = "*" if is_optional and member_type[0] not in "*[" else "" + if within_nullable_struct: + # Within a struct which has a field of type that can hold JSON NULL, + # we have to _not_ use a pointer, otherwise the Marshal methods are + # not called. + isptr = "" if member_type in self.accept_null_types else isptr + fieldtag = ( '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`' ) @@ -384,38 +438,228 @@ def get_struct_field( if field_doc != "": arg["doc"] = field_doc - return arg + return arg, is_nullable + + +# This helper is used whithin a struct that has members that accept JSON NULL. +def map_and_set( + is_nullable: bool, field: str, field_is_optional: bool, name: str +) -> Tuple[str, str]: + mapstr = "" + setstr = "" + if is_nullable: + mapstr = f""" + if val, absent := s.{field}.ToAnyOrAbsent(); !absent {{ + m["{name}"] = val + }} +""" + setstr += f""" + if _, absent := (&tmp.{field}).ToAnyOrAbsent(); !absent {{ + s.{field} = &tmp.{field} + }} +""" + elif field_is_optional: + mapstr = f""" + if s.{field} != nil {{ + m["{name}"] = s.{field} + }} +""" + setstr = f""" s.{field} = tmp.{field}\n""" + else: + mapstr = f""" m["{name}"] = s.{field}\n""" + setstr = f""" s.{field} = tmp.{field}\n""" + + return mapstr, setstr + + +def recursive_base_nullable( + self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType] +) -> Tuple[List[dict[str:str]], str, str, str, str]: + fields: List[dict[str:str]] = [] + map_members = "" + set_members = "" + map_special = "" + set_special = "" + + if not base: + return fields, map_members, set_members, map_special, set_special + + doc = self.docmap.get(base.name, None) + _, docfields = qapi_to_golang_struct_docs(doc) + + if base.base is not None: + embed_base = self.schema.lookup_entity(base.base.name) + ( + fields, + map_members, + set_members, + map_special, + set_special, + ) = recursive_base_nullable(self, embed_base) + + for member in base.local_members: + field_doc = docfields.get(member.name, "") + field, _ = get_struct_field( + self, + member.name, + member.type.name, + field_doc, + True, + member.optional, + False, + ) + fields.append(field) + + member_type = qapi_schema_type_to_go_type(member.type.name) + nullable = member_type in self.accept_null_types + field_name = qapi_to_field_name(member.name) + tomap, toset = map_and_set( + nullable, field_name, member.optional, member.name + ) + if nullable: + map_special += tomap + set_special += toset + else: + map_members += tomap + set_members += toset + + return fields, map_members, set_members, map_special, set_special + + +# Helper function. This is executed when the QAPI schema has members +# that could accept JSON NULL (e.g: StrOrNull in QEMU"s QAPI schema). +# This struct will need to be extended with Marshal/Unmarshal methods to +# properly handle such atypical members. +# +# Only the Marshallaing methods are generated but we do need to iterate over +# all the members to properly set/check them in those methods. +def struct_with_nullable_generate_marshal( + self: QAPISchemaGenGolangVisitor, + name: str, + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants], +) -> str: + ( + fields, + map_members, + set_members, + map_special, + set_special, + ) = recursive_base_nullable(self, base) + + doc = self.docmap.get(name, None) + _, docfields = qapi_to_golang_struct_docs(doc) + + if members: + for member in members: + field_doc = docfields.get(member.name, "") + field, _ = get_struct_field( + self, + member.name, + member.type.name, + field_doc, + True, + member.optional, + False, + ) + fields.append(field) + + member_type = qapi_schema_type_to_go_type(member.type.name) + nullable = member_type in self.accept_null_types + tomap, toset = map_and_set( + nullable, + qapi_to_field_name(member.name), + member.optional, + member.name, + ) + if nullable: + map_special += tomap + set_special += toset + else: + map_members += tomap + set_members += toset + + if variants: + for variant in variants.variants: + if variant.type.is_implicit(): + continue + + field, _ = get_struct_field( + self, + variant.name, + variant.type.name, + True, + variant.optional, + True, + ) + fields.append(field) + + member_type = qapi_schema_type_to_go_type(variant.type.name) + nullable = member_type in self.accept_null_types + tomap, toset = map_and_set( + nullable, + qapi_to_field_name(variant.name), + variant.optional, + variant.name, + ) + if nullable: + map_special += tomap + set_special += toset + else: + map_members += tomap + set_members += toset + + type_name = qapi_to_go_type_name(name) + struct = generate_struct_type("", args=fields, indent=1) + return string_to_code( + TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL.format( + struct=struct[1:-1], + type_name=type_name, + map_members=map_members, + map_special=map_special, + set_members=set_members, + set_special=set_special, + ) + ) def recursive_base( self: QAPISchemaGenGolangVisitor, base: Optional[QAPISchemaObjectType], -) -> List[dict[str:str]]: + discriminator: Optional[str] = None, +) -> Tuple[List[dict[str:str]], bool]: fields: List[dict[str:str]] = [] + with_nullable = False if not base: - return fields + return fields, with_nullable if base.base is not None: embed_base = self.schema.lookup_entity(base.base.name) - fields = recursive_base(self, embed_base) + fields, with_nullable = recursive_base(self, embed_base, discriminator) doc = self.docmap.get(base.name, None) _, docfields = qapi_to_golang_struct_docs(doc) for member in base.local_members: + if discriminator and member.name == discriminator: + continue + field_doc = docfields.get(member.name, "") - field = get_struct_field( + field, nullable = get_struct_field( self, member.name, member.type.name, field_doc, + False, member.optional, False, ) fields.append(field) + with_nullable = True if nullable else with_nullable - return fields + return fields, with_nullable # Helper function that is used for most of QAPI types @@ -431,7 +675,8 @@ def qapi_to_golang_struct( indent: int = 0, doc_enabled: bool = True, ) -> str: - fields = recursive_base(self, base) + discriminator = None if not variants else variants.tag_member.name + fields, with_nullable = recursive_base(self, base, discriminator) doc = self.docmap.get(name, None) type_doc, docfields = qapi_to_golang_struct_docs(doc) @@ -441,15 +686,17 @@ def qapi_to_golang_struct( if members: for member in members: field_doc = docfields.get(member.name, "") if doc_enabled else "" - field = get_struct_field( + field, nullable = get_struct_field( self, member.name, member.type.name, field_doc, + False, member.optional, False, ) fields.append(field) + with_nullable = True if nullable else with_nullable exists = {} if variants: @@ -460,15 +707,17 @@ def qapi_to_golang_struct( exists[variant.name] = True field_doc = docfields.get(variant.name, "") if doc_enabled else "" - field = get_struct_field( + field, nullable = get_struct_field( self, variant.name, variant.type.name, field_doc, + False, True, True, ) fields.append(field) + with_nullable = True if nullable else with_nullable type_name = qapi_to_go_type_name(name) content = string_to_code( @@ -476,6 +725,10 @@ def qapi_to_golang_struct( type_name, type_doc=type_doc, args=fields, indent=indent ) ) + if with_nullable: + content += struct_with_nullable_generate_marshal( + self, name, base, members, variants + ) return content @@ -484,6 +737,7 @@ def generate_template_alternate( name: str, variants: Optional[QAPISchemaVariants], ) -> str: + absent_check_fields = "" args: List[dict[str:str]] = [] nullable = name in self.accept_null_types if nullable: @@ -517,6 +771,12 @@ def generate_template_alternate( assert nullable continue + if nullable: + absent_check_fields += string_to_code( + TEMPLATE_ALTERNATE_NULLABLE_CHECK[1:].format( + var_name=var_name + ) + ) skip_indent = 1 + len(FOUR_SPACES) if marshal_check_fields == "": skip_indent = 1 @@ -528,6 +788,12 @@ def generate_template_alternate( ].format(var_name=var_name, var_type=var_type) content += string_to_code(generate_struct_type(name, args=args)) + if nullable: + content += string_to_code( + TEMPLATE_ALTERNATE_NULLABLE.format( + name=name, absent_check_fields=absent_check_fields[:-1] + ) + ) content += string_to_code( TEMPLATE_ALTERNATE_METHODS.format( name=name, @@ -629,6 +895,8 @@ def visit_begin(self, schema: QAPISchema) -> None: ) self.types[qapitype] += generate_template_imports(imports) + self.types["alternate"] += string_to_code(TEMPLATE_ALTERNATE) + def visit_end(self) -> None: del self.schema self.types["enum"] += generate_content_from_dict(self.enums) From patchwork Fri Feb 14 20:29:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D943CC02198 for ; Fri, 14 Feb 2025 20:33:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2Kr-000491-PC; Fri, 14 Feb 2025 15:31:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KJ-00047q-9q for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:19 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KH-00020z-0C for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565016; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c8tZQemqjYq4asfStaHd1xQomy+ne9g/9ss/GyZ19Cs=; b=OFZS+al52MhhnvNumTt8SftZOX1X/2FTUWWsCCASGRvOwCxfhkLqJ6zVsBJ/Vt4V5CneRd 2VAXoJZHavbBiMC8q8qdZDBguxJidDNB7pFdu0mvHD+Wti+VfTZp4bn0JEakiZcedyLYJu DojXgM0Bpj64fO4WlkzxX2nMDk5LKf4= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-479-CoED3PVoNES9-4pI28VZvA-1; Fri, 14 Feb 2025 15:30:12 -0500 X-MC-Unique: CoED3PVoNES9-4pI28VZvA-1 X-Mimecast-MFC-AGG-ID: CoED3PVoNES9-4pI28VZvA_1739565012 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1338A1800875 for ; Fri, 14 Feb 2025 20:30:12 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9E13D1800352; Fri, 14 Feb 2025 20:30:09 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 06/11] qapi: golang: Generate union type Date: Fri, 14 Feb 2025 21:29:39 +0100 Message-ID: <20250214202944.69897-7-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it. The QAPI union type has two types of fields: The @base and the @Variants members. The @base fields can be considered common members for the union while only one field maximum is set for the @Variants. In the QAPI specification, it defines a @discriminator field, which is an Enum type. The purpose of the @discriminator is to identify which @variant type is being used. For the @discriminator's enum that are not handled by the QAPI Union, we add in the Go struct a separate block as "Unbranched enum fields". The rationale for this extra block is to allow the user to pass that enum value under the discriminator, without extra payload. The union types implement the Marshaler and Unmarshaler interfaces to seamless decode from JSON objects to Golang structs and vice versa. qapi: | ## | # @SetPasswordOptions: | # | # Options for set_password. | # | # @protocol: | # - 'vnc' to modify the VNC server password | # - 'spice' to modify the Spice server password | # | # @password: the new password | # | # @connected: How to handle existing clients when changing the | # password. If nothing is specified, defaults to 'keep'. For | # VNC, only 'keep' is currently implemented. | # | # Since: 7.0 | ## | { 'union': 'SetPasswordOptions', | 'base': { 'protocol': 'DisplayProtocol', | 'password': 'str', | '*connected': 'SetPasswordAction' }, | 'discriminator': 'protocol', | 'data': { 'vnc': 'SetPasswordOptionsVnc' } } go: | // Options for set_password. | // | // Since: 7.0 | type SetPasswordOptions struct { | // the new password | Password string `json:"password"` | // How to handle existing clients when changing the password. If | // nothing is specified, defaults to 'keep'. For VNC, only 'keep' | // is currently implemented. | Connected *SetPasswordAction `json:"connected,omitempty"` | // Variants fields | Vnc *SetPasswordOptionsVnc `json:"-"` | // Unbranched enum fields | Spice bool `json:"-"` | } | | func (s SetPasswordOptions) MarshalJSON() ([]byte, error) { | ... | } | | func (s *SetPasswordOptions) UnmarshalJSON(data []byte) error { | ... | } Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 208 +++++++++++++++++++++++++++++++++- scripts/qapi/golang/utils.go | 12 ++ 2 files changed, 217 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index 0637bb3e3e..59e9b1f644 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -155,6 +155,81 @@ """ +TEMPLATE_UNION_CHECK_VARIANT_FIELD = """ + if s.{field} != nil && err == nil {{ + if len(bytes) != 0 {{ + err = errors.New(`multiple variant fields set`) + }} else if err = unwrapToMap(m, s.{field}); err == nil {{ + m["{discriminator}"] = {go_enum_value} + bytes, err = json.Marshal(m) + }} + }} +""" + +TEMPLATE_UNION_CHECK_UNBRANCHED_FIELD = """ + if s.{field} && err == nil {{ + if len(bytes) != 0 {{ + err = errors.New(`multiple variant fields set`) + }} else {{ + m["{discriminator}"] = {go_enum_value} + bytes, err = json.Marshal(m) + }} + }} +""" + +TEMPLATE_UNION_DRIVER_VARIANT_CASE = """ + case {go_enum_value}: + s.{field} = new({member_type}) + if err := json.Unmarshal(data, s.{field}); err != nil {{ + s.{field} = nil + return err + }}""" + +TEMPLATE_UNION_DRIVER_UNBRANCHED_CASE = """ + case {go_enum_value}: + s.{field} = true +""" + +TEMPLATE_UNION_METHODS = """ +func (s {type_name}) MarshalJSON() ([]byte, error) {{ + var bytes []byte + var err error + m := make(map[string]any) + {{ + type Alias {type_name} + v := Alias(s) + unwrapToMap(m, &v) + }} +{check_fields} + if err != nil {{ + return nil, fmt.Errorf("marshal {type_name} due:'%s' struct='%+v'", err, s) + }} else if len(bytes) == 0 {{ + return nil, fmt.Errorf("marshal {type_name} unsupported, struct='%+v'", s) + }} + return bytes, nil +}} + +func (s *{type_name}) UnmarshalJSON(data []byte) error {{ +{base_type_def} + tmp := struct {{ + {base_type_name} + }}{{}} + + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} +{base_type_assign_unmarshal} + switch tmp.{discriminator} {{ +{driver_cases} + default: + return fmt.Errorf("unmarshal {type_name} received unrecognized value '%s'", + tmp.{discriminator}) + }} + return nil +}} +""" + + # Takes the documentation object of a specific type and returns # that type's documentation and its member's docs. def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): @@ -202,6 +277,12 @@ def qapi_name_is_object(name: str) -> bool: return name.startswith("q_obj_") +def qapi_base_name_to_parent(name: str) -> str: + if qapi_name_is_base(name): + name = name[6:-5] + return name + + def qapi_to_field_name(name: str) -> str: return name.title().replace("_", "").replace("-", "") @@ -639,7 +720,7 @@ def recursive_base( embed_base = self.schema.lookup_entity(base.base.name) fields, with_nullable = recursive_base(self, embed_base, discriminator) - doc = self.docmap.get(base.name, None) + doc = self.docmap.get(qapi_base_name_to_parent(base.name), None) _, docfields = qapi_to_golang_struct_docs(doc) for member in base.local_members: @@ -719,6 +800,24 @@ def qapi_to_golang_struct( fields.append(field) with_nullable = True if nullable else with_nullable + if info.defn_meta == "union" and variants: + enum_name = variants.tag_member.type.name + enum_obj = self.schema.lookup_entity(enum_name) + if len(exists) != len(enum_obj.members): + fields.append({"comment": "Unbranched enum fields"}) + for member in enum_obj.members: + if member.name in exists: + continue + + field_doc = ( + docfields.get(member.name, "") if doc_enabled else "" + ) + field, nullable = get_struct_field( + self, member.name, "bool", field_doc, False, False, True + ) + fields.append(field) + with_nullable = True if nullable else with_nullable + type_name = qapi_to_go_type_name(name) content = string_to_code( generate_struct_type( @@ -732,6 +831,98 @@ def qapi_to_golang_struct( return content +def qapi_to_golang_methods_union( + self: QAPISchemaGenGolangVisitor, + name: str, + base: Optional[QAPISchemaObjectType], + variants: Optional[QAPISchemaVariants], +) -> str: + type_name = qapi_to_go_type_name(name) + + assert base + base_type_assign_unmarshal = "" + base_type_name = qapi_to_go_type_name(base.name) + base_type_def = qapi_to_golang_struct( + self, + base.name, + base.info, + base.ifcond, + base.features, + base.base, + base.members, + base.branches, + indent=1, + doc_enabled=False, + ) + + discriminator = qapi_to_field_name(variants.tag_member.name) + for member in base.local_members: + field = qapi_to_field_name(member.name) + if field == discriminator: + continue + base_type_assign_unmarshal += f""" + s.{field} = tmp.{field}""" + + driver_cases = "" + check_fields = "" + exists = {} + enum_name = variants.tag_member.type.name + if variants: + for var in variants.variants: + if var.type.is_implicit(): + continue + + field = qapi_to_field_name(var.name) + enum_value = qapi_to_field_name_enum(var.name) + member_type = qapi_schema_type_to_go_type(var.type.name) + go_enum_value = f"""{enum_name}{enum_value}""" + exists[go_enum_value] = True + + check_fields += TEMPLATE_UNION_CHECK_VARIANT_FIELD.format( + field=field, + discriminator=variants.tag_member.name, + go_enum_value=go_enum_value, + ) + driver_cases += TEMPLATE_UNION_DRIVER_VARIANT_CASE.format( + go_enum_value=go_enum_value, + field=field, + member_type=member_type, + ) + + enum_obj = self.schema.lookup_entity(enum_name) + if len(exists) != len(enum_obj.members): + for member in enum_obj.members: + value = qapi_to_field_name_enum(member.name) + go_enum_value = f"""{enum_name}{value}""" + + if go_enum_value in exists: + continue + + field = qapi_to_field_name(member.name) + + check_fields += TEMPLATE_UNION_CHECK_UNBRANCHED_FIELD.format( + field=field, + discriminator=variants.tag_member.name, + go_enum_value=go_enum_value, + ) + driver_cases += TEMPLATE_UNION_DRIVER_UNBRANCHED_CASE.format( + go_enum_value=go_enum_value, + field=field, + ) + + return string_to_code( + TEMPLATE_UNION_METHODS.format( + type_name=type_name, + check_fields=check_fields[1:], + base_type_def=base_type_def[1:], + base_type_name=base_type_name, + base_type_assign_unmarshal=base_type_assign_unmarshal, + discriminator=discriminator, + driver_cases=driver_cases[1:], + ) + ) + + def generate_template_alternate( self: QAPISchemaGenGolangVisitor, name: str, @@ -855,6 +1046,7 @@ def __init__(self, _: str): "alternate": ["encoding/json", "errors", "fmt"], "enum": [], "struct": ["encoding/json"], + "union": ["encoding/json", "errors", "fmt"], } self.schema: QAPISchema @@ -863,6 +1055,7 @@ def __init__(self, _: str): self.enums: dict[str, str] = {} self.alternates: dict[str, str] = {} self.structs: dict[str, str] = {} + self.unions: dict[str, str] = {} self.accept_null_types = [] self.docmap = {} @@ -902,6 +1095,7 @@ def visit_end(self) -> None: self.types["enum"] += generate_content_from_dict(self.enums) self.types["alternate"] += generate_content_from_dict(self.alternates) self.types["struct"] += generate_content_from_dict(self.structs) + self.types["union"] += generate_content_from_dict(self.unions) def visit_object_type( self, @@ -913,11 +1107,11 @@ def visit_object_type( members: List[QAPISchemaObjectTypeMember], branches: Optional[QAPISchemaBranches], ) -> None: - # Do not handle anything besides struct. + # Do not handle anything besides struct and unions. if ( name == self.schema.the_empty_object_type.name or not isinstance(name, str) - or info.defn_meta not in ["struct"] + or info.defn_meta not in ["struct", "union"] ): return @@ -948,6 +1142,14 @@ def visit_object_type( self, name, info, ifcond, features, base, members, branches ) ) + else: + assert name not in self.unions + self.unions[name] = qapi_to_golang_struct( + self, name, info, ifcond, features, base, members, branches + ) + self.unions[name] += qapi_to_golang_methods_union( + self, name, base, branches + ) def visit_alternate_type( self, diff --git a/scripts/qapi/golang/utils.go b/scripts/qapi/golang/utils.go index f00c0a5d83..193e0c53bb 100644 --- a/scripts/qapi/golang/utils.go +++ b/scripts/qapi/golang/utils.go @@ -9,6 +9,7 @@ package qapi import ( "encoding/json" + "fmt" "strings" ) @@ -24,3 +25,14 @@ func strictDecode(into interface{}, from []byte) error { } return nil } + +// This helper is used to move struct's fields into a map. +// This function is useful to merge JSON objects. +func unwrapToMap(m map[string]any, data any) error { + if bytes, err := json.Marshal(&data); err != nil { + return fmt.Errorf("unwrapToMap: %s", err) + } else if err := json.Unmarshal(bytes, &m); err != nil { + return fmt.Errorf("unwrapToMap: %s, data=%s", err, string(bytes)) + } + return nil +} From patchwork Fri Feb 14 20:29:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D696C02198 for ; Fri, 14 Feb 2025 20:31:52 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2LR-0004jI-3B; Fri, 14 Feb 2025 15:31:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KN-00048e-5C for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:23 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KL-00021K-8E for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BAf9GaMvB/OkUbr8TNOAgRbhsmMLbkbbbiRNM5/Hcu8=; b=VNvmETYXTIP/3s12Z4eafNr5APM1vYkvM9lrHU+PAmt2Ea0/yYpXuOevkNv1zIM3W3iff6 mq3ApOiugmt+bfkEVtl36was3S3n4wUrXAuieCpKqwA1iRkrSZ25WekTgmF/76F39HhU4e rZSXB9AJbWwT3L8wQrPb+P1O87g5Glg= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-297-O2JL-QDNODOpoqvkTEDLXw-1; Fri, 14 Feb 2025 15:30:17 -0500 X-MC-Unique: O2JL-QDNODOpoqvkTEDLXw-1 X-Mimecast-MFC-AGG-ID: O2JL-QDNODOpoqvkTEDLXw_1739565015 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5B18919039C1 for ; Fri, 14 Feb 2025 20:30:15 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BC85D1800358; Fri, 14 Feb 2025 20:30:12 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 07/11] qapi: golang: Generate event type Date: Fri, 14 Feb 2025 21:29:40 +0100 Message-ID: <20250214202944.69897-8-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch handles QAPI event types and generates data structures in Go that handles it. Note that the timestamp is part of the first layer of unmarshal, so it s a member of protocol.go's Message type. Example: qapi: | ## | # @MEMORY_DEVICE_SIZE_CHANGE: | # | # Emitted when the size of a memory device changes. Only emitted for | # memory devices that can actually change the size (e.g., virtio-mem | # due to guest action). | # | # @id: device's ID | # | # @size: the new size of memory that the device provides | # | # @qom-path: path to the device object in the QOM tree (since 6.2) | # | # .. note:: This event is rate-limited. | # | # Since: 5.1 | # | # .. qmp-example:: | # | # <- { "event": "MEMORY_DEVICE_SIZE_CHANGE", | # "data": { "id": "vm0", "size": 1073741824, | # "qom-path": "/machine/unattached/device[2]" }, | # "timestamp": { "seconds": 1588168529, "microseconds": 201316 } } | ## | { 'event': 'MEMORY_DEVICE_SIZE_CHANGE', | 'data': { '*id': 'str', 'size': 'size', 'qom-path' : 'str'} } go: | // Emitted when the size of a memory device changes. Only emitted for | // memory devices that can actually change the size (e.g., virtio-mem | // due to guest action). | // | // .. note:: This event is rate-limited. | // | // Since: 5.1 | // | // .. qmp-example:: <- { "event": "MEMORY_DEVICE_SIZE_CHANGE", | // "data": { "id": "vm0", "size": 1073741824, "qom-path": | // "/machine/unattached/device[2]" }, "timestamp": { "seconds": | // 1588168529, "microseconds": 201316 } } | type MemoryDeviceSizeChangeEvent struct { | // device's ID | Id *string `json:"id,omitempty"` | // the new size of memory that the device provides | Size uint64 `json:"size"` | // path to the device object in the QOM tree (since 6.2) | QomPath string `json:"qom-path"` | } Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 49 ++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index 59e9b1f644..63d55ca950 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -291,7 +291,7 @@ def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") -def qapi_to_go_type_name(name: str) -> str: +def qapi_to_go_type_name(name: str, meta: Optional[str] = None) -> str: # We want to keep CamelCase for Golang types. We want to avoid removing # already set CameCase names while fixing uppercase ones, eg: # 1) q_obj_SocketAddress_base -> SocketAddressBase @@ -309,6 +309,12 @@ def qapi_to_go_type_name(name: str) -> str: name += "".join(word.title() for word in words[1:]) + # Handle specific meta suffix + types = ["event"] + if meta in types: + name = name[:-3] if name.endswith("Arg") else name + name += meta.title().replace(" ", "") + return name @@ -818,7 +824,8 @@ def qapi_to_golang_struct( fields.append(field) with_nullable = True if nullable else with_nullable - type_name = qapi_to_go_type_name(name) + type_name = qapi_to_go_type_name(name, info.defn_meta) + content = string_to_code( generate_struct_type( type_name, type_doc=type_doc, args=fields, indent=indent @@ -996,6 +1003,15 @@ def generate_template_alternate( return "\n" + content +def generate_template_event(events: dict[str, Tuple[str, str]]) -> str: + content = "" + for name in sorted(events): + type_name, gocode = events[name] + content += gocode + + return content + + def generate_content_from_dict(data: dict[str, str]) -> str: content = "" @@ -1045,11 +1061,13 @@ def __init__(self, _: str): types = { "alternate": ["encoding/json", "errors", "fmt"], "enum": [], + "event": [], "struct": ["encoding/json"], "union": ["encoding/json", "errors", "fmt"], } self.schema: QAPISchema + self.events: dict[str, Tuple[str, str]] = {} self.golang_package_name = "qapi" self.duplicate = list(gofiles) self.enums: dict[str, str] = {} @@ -1096,6 +1114,7 @@ def visit_end(self) -> None: self.types["alternate"] += generate_content_from_dict(self.alternates) self.types["struct"] += generate_content_from_dict(self.structs) self.types["union"] += generate_content_from_dict(self.unions) + self.types["event"] += generate_template_event(self.events) def visit_object_type( self, @@ -1254,7 +1273,31 @@ def visit_event( arg_type: Optional[QAPISchemaObjectType], boxed: bool, ) -> None: - pass + assert name == info.defn_name + assert name not in self.events + type_name = qapi_to_go_type_name(name, info.defn_meta) + + if isinstance(arg_type, QAPISchemaObjectType): + content = string_to_code( + qapi_to_golang_struct( + self, + name, + info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.branches, + ) + ) + else: + doc = self.docmap.get(name, None) + type_doc, _ = qapi_to_golang_struct_docs(doc) + content = string_to_code( + generate_struct_type(type_name, type_doc=type_doc) + ) + + self.events[name] = (type_name, content) def write(self, outdir: str) -> None: godir = "go" From patchwork Fri Feb 14 20:29:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E341C021A4 for ; Fri, 14 Feb 2025 20:32:00 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2LX-0004w4-3p; Fri, 14 Feb 2025 15:31:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KN-00048g-Bd for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:23 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KL-00021Q-M3 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565021; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S7J/rt19u5i7ccmeMEi0N5tDk0udDVxbxjxL2xbH3Lg=; b=Uz3WJI5AG7qhqSxMD9TDHH1kwtHBdt7Sbxmif1atok+NnG1P8Cz4+rtC712WA9D5pUX1oN 0vnxMe+UEBeVR/WM0EIvk2kaB3w7afxLa79nTOi97LHBQvl1xoqaFwCHhjzVi6wDxL3fZO XL0JvhXjU5TkU3N8AcK0gejx24+c1K4= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-547-Gbdm0egUOEmEN3C0IURRuQ-1; Fri, 14 Feb 2025 15:30:19 -0500 X-MC-Unique: Gbdm0egUOEmEN3C0IURRuQ-1 X-Mimecast-MFC-AGG-ID: Gbdm0egUOEmEN3C0IURRuQ_1739565018 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 84EAF19373DC for ; Fri, 14 Feb 2025 20:30:18 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 0DA561800940; Fri, 14 Feb 2025 20:30:15 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 08/11] qapi: golang: Generate Event interface Date: Fri, 14 Feb 2025 21:29:41 +0100 Message-ID: <20250214202944.69897-9-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The Event interface is an abstraction that can be used by client and server to the manager the Event types albeit with a different implementation for sending and receiving. The implementation of client/server is not part of this series. Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 38 ++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index 63d55ca950..b9a2c47137 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -229,6 +229,12 @@ }} """ +TEMPLATE_EVENT = """ +type Event interface {{ +{methods} +}} +""" + # Takes the documentation object of a specific type and returns # that type's documentation and its member's docs. @@ -1003,13 +1009,16 @@ def generate_template_alternate( return "\n" + content -def generate_template_event(events: dict[str, Tuple[str, str]]) -> str: +def generate_template_event(events: dict[str, Tuple[str, str]]) -> (str, str): content = "" + methods = "" for name in sorted(events): type_name, gocode = events[name] + methods += f"\t{type_name}({type_name}, time.Time) error\n" content += gocode - return content + iface = string_to_code(TEMPLATE_EVENT.format(methods=methods[:-1])) + return content, iface def generate_content_from_dict(data: dict[str, str]) -> str: @@ -1065,6 +1074,9 @@ def __init__(self, _: str): "struct": ["encoding/json"], "union": ["encoding/json", "errors", "fmt"], } + interfaces = { + "event": ["time"], + } self.schema: QAPISchema self.events: dict[str, Tuple[str, str]] = {} @@ -1080,6 +1092,9 @@ def __init__(self, _: str): self.types = dict.fromkeys(types, "") self.types_import = types + self.interfaces = dict.fromkeys(interfaces, "") + self.interface_imports = interfaces + def visit_begin(self, schema: QAPISchema) -> None: self.schema = schema @@ -1100,6 +1115,12 @@ def visit_begin(self, schema: QAPISchema) -> None: continue self.docmap[doc.symbol] = doc + for qapitype, imports in self.interface_imports.items(): + self.interfaces[qapitype] = TEMPLATE_GENERATED_HEADER[1:].format( + package_name=self.golang_package_name + ) + self.interfaces[qapitype] += generate_template_imports(imports) + for qapitype, imports in self.types_import.items(): self.types[qapitype] = TEMPLATE_GENERATED_HEADER[1:].format( package_name=self.golang_package_name @@ -1114,7 +1135,10 @@ def visit_end(self) -> None: self.types["alternate"] += generate_content_from_dict(self.alternates) self.types["struct"] += generate_content_from_dict(self.structs) self.types["union"] += generate_content_from_dict(self.unions) - self.types["event"] += generate_template_event(self.events) + + evtype, eviface = generate_template_event(self.events) + self.types["event"] += evtype + self.interfaces["event"] += eviface def visit_object_type( self, @@ -1318,3 +1342,11 @@ def write(self, outdir: str) -> None: with open(pathname, "w", encoding="utf8") as outfile: outfile.write(content) + + # Interfaces to be generated + for qapitype, content in self.interfaces.items(): + gofile = f"gen_iface_{qapitype}.go" + pathname = os.path.join(targetpath, gofile) + + with open(pathname, "w", encoding="utf8") as outfile: + outfile.write(content) From patchwork Fri Feb 14 20:29:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6527DC02198 for ; Fri, 14 Feb 2025 20:37:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2Ls-0005Z7-79; Fri, 14 Feb 2025 15:31:57 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KT-00049H-Ez for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:50 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KP-000225-59 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N3SVrTqLq8tuMrJJ7lHZyUbVQuh8koYYjB5pQtXNihI=; b=alvB0Xohjj9M+/Q8tmvz+cq4D+UvetVULyws+shWYO/+CuEb4mWJh8fLr0GXRq7Aklg6DE FKv7Wq2lP/E+S8J3Fd2QrpeFQa2Q2CUdrvB2XtHQNVDXYNBSLpcqZzgCIKdvzby3gwBu7p F1avZ/S5tqXmS8wmpWJqMMOs9cz0+CI= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-141-rePO_smrPnONYfBgBjPnWA-1; Fri, 14 Feb 2025 15:30:22 -0500 X-MC-Unique: rePO_smrPnONYfBgBjPnWA-1 X-Mimecast-MFC-AGG-ID: rePO_smrPnONYfBgBjPnWA_1739565021 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9879118EB2C6 for ; Fri, 14 Feb 2025 20:30:21 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 513D91800358; Fri, 14 Feb 2025 20:30:18 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 09/11] qapi: golang: Generate command type Date: Fri, 14 Feb 2025 21:29:42 +0100 Message-ID: <20250214202944.69897-10-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org This patch handles QAPI command types and generates data structures in Go that handles it. Note that command's id is part of the first layer of unmarshal, so it is a member of protocol.go's Message type. qapi: | ## | # @add-fd: | # | # Add a file descriptor, that was passed via SCM rights, to an fd set. | # | # @fdset-id: The ID of the fd set to add the file descriptor to. | # | # @opaque: A free-form string that can be used to describe the fd. | # | # Returns: | # @AddfdInfo | # | # Errors: | # - If file descriptor was not received, GenericError | # - If @fdset-id is a negative value, GenericError | # | # .. note:: The list of fd sets is shared by all monitor connections. | # | # .. note:: If @fdset-id is not specified, a new fd set will be | # created. | # | # Since: 1.2 | # | # .. qmp-example:: | # | # -> { "execute": "add-fd", "arguments": { "fdset-id": 1 } } | # <- { "return": { "fdset-id": 1, "fd": 3 } } | ## | { 'command': 'add-fd', | 'data': { '*fdset-id': 'int', | '*opaque': 'str' }, | 'returns': 'AddfdInfo' } go: | // Add a file descriptor, that was passed via SCM rights, to an fd | // set. | // | // Returns: @AddfdInfo | // | // Errors: - If file descriptor was not received, GenericError - | // If @fdset-id is a negative value, GenericError | // | // .. note:: The list of fd sets is shared by all monitor connections. | // .. note:: If @fdset-id is not specified, a new fd set will be | // created. | // | // Since: 1.2 | // | // .. qmp-example:: -> { "execute": "add-fd", "arguments": { | // "fdset-id": 1 } } <- { "return": { "fdset-id": 1, "fd": 3 } } | type AddFdCommand struct { | // The ID of the fd set to add the file descriptor to. | FdsetId *int64 `json:"fdset-id,omitempty"` | // A free-form string that can be used to describe the fd. | Opaque *string `json:"opaque,omitempty"` | } Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 52 +++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index b9a2c47137..a14970fb1f 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -316,7 +316,7 @@ def qapi_to_go_type_name(name: str, meta: Optional[str] = None) -> str: name += "".join(word.title() for word in words[1:]) # Handle specific meta suffix - types = ["event"] + types = ["event", "command"] if meta in types: name = name[:-3] if name.endswith("Arg") else name name += meta.title().replace(" ", "") @@ -1009,6 +1009,15 @@ def generate_template_alternate( return "\n" + content +def generate_template_command(commands: dict[str, Tuple[str, str]]) -> str: + content = "" + for name in sorted(commands): + type_name, gocode = commands[name] + content += gocode + + return content + + def generate_template_event(events: dict[str, Tuple[str, str]]) -> (str, str): content = "" methods = "" @@ -1069,6 +1078,7 @@ def __init__(self, _: str): # Map each qapi type to the necessary Go imports types = { "alternate": ["encoding/json", "errors", "fmt"], + "command": [], "enum": [], "event": [], "struct": ["encoding/json"], @@ -1080,6 +1090,7 @@ def __init__(self, _: str): self.schema: QAPISchema self.events: dict[str, Tuple[str, str]] = {} + self.commands: dict[str, Tuple[str, str]] = {} self.golang_package_name = "qapi" self.duplicate = list(gofiles) self.enums: dict[str, str] = {} @@ -1140,6 +1151,8 @@ def visit_end(self) -> None: self.types["event"] += evtype self.interfaces["event"] += eviface + self.types["command"] += generate_template_command(self.commands) + def visit_object_type( self, name: str, @@ -1286,7 +1299,42 @@ def visit_command( allow_preconfig: bool, coroutine: bool, ) -> None: - pass + assert name == info.defn_name + assert name not in self.commands + + type_name = qapi_to_go_type_name(name, info.defn_meta) + + doc = self.docmap.get(name, None) + type_doc, _ = qapi_to_golang_struct_docs(doc) + + content = "" + if boxed or not arg_type or not qapi_name_is_object(arg_type.name): + args: List[dict[str:str]] = [] + if arg_type: + args.append( + { + "name": f"{arg_type.name}", + } + ) + content += string_to_code( + generate_struct_type(type_name, type_doc=type_doc, args=args) + ) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content += string_to_code( + qapi_to_golang_struct( + self, + name, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.branches, + ) + ) + + self.commands[name] = (type_name, content) def visit_event( self, From patchwork Fri Feb 14 20:29:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1F711C021A4 for ; Fri, 14 Feb 2025 20:34:37 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2LX-0004xK-Sc; Fri, 14 Feb 2025 15:31:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KU-00049S-RD for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:50 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KT-00022R-84 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565028; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OOU0wCWpbF8Ks7g7EwMzl8CZelbsnNtS6EUVGTCCun4=; b=BfI0R9sM69FQB2cES1uYnVm/t3ldUoV1d9HSpVgXJaJvjIM+yNbWBhz6gO78iz2toxoiV1 4kl0N5qsW+Chi0r/nR+P9jNjYaJeElz9Y7d75nPlfKZpeNcAK205ZpY7lWgSETDL2IJKri 2//bdRjq+4k8cex5/7X+Ylh5NqEp7do= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-609-X8iRNArFPJa5ArR-eIyVQg-1; Fri, 14 Feb 2025 15:30:26 -0500 X-MC-Unique: X8iRNArFPJa5ArR-eIyVQg-1 X-Mimecast-MFC-AGG-ID: X8iRNArFPJa5ArR-eIyVQg_1739565025 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 62E0019039C8 for ; Fri, 14 Feb 2025 20:30:25 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 49BD41800352; Fri, 14 Feb 2025 20:30:21 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 10/11] qapi: golang: Generate Command sync/async interfaces Date: Fri, 14 Feb 2025 21:29:43 +0100 Message-ID: <20250214202944.69897-11-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The Command interface is an abstraction that can be used by client and server to the manager the Command types albeit with a different implementation for sending and receiving. The proposal for Sync is defined as Command while the Async is named CommandAsync. The implementation of client/server is not part of this series. Signed-off-by: Victor Toso --- scripts/qapi/golang/golang.py | 56 +++++++++++++++++++++++++++++++---- 1 file changed, 50 insertions(+), 6 deletions(-) diff --git a/scripts/qapi/golang/golang.py b/scripts/qapi/golang/golang.py index a14970fb1f..0b7dadff41 100644 --- a/scripts/qapi/golang/golang.py +++ b/scripts/qapi/golang/golang.py @@ -235,6 +235,20 @@ }} """ +TEMPLATE_COMMAND = """ +// Synchronous interface of all available commands +type Commands interface {{ +{methods} +}} + +{callbacks} + +// Asynchronous interface of all available commands +type CommandsAsync interface {{ +{async_methods} +}} +""" + # Takes the documentation object of a specific type and returns # that type's documentation and its member's docs. @@ -1009,13 +1023,37 @@ def generate_template_alternate( return "\n" + content -def generate_template_command(commands: dict[str, Tuple[str, str]]) -> str: +def generate_template_command( + commands: dict[str, Tuple[str, str, str]] +) -> (str, str): content = "" + methods = "" + async_methods = "" + callbacks = "" + for name in sorted(commands): - type_name, gocode = commands[name] + type_name, gocode, retarg = commands[name] content += gocode - return content + name = type_name[:-7] + cbname = f"{name}Complete" + syncret = "error" + cbarg = "error" + if retarg != "": + cbarg = f"{retarg}, error" + syncret = f"({retarg}, error)" + methods += f"\t{name}({type_name}) {syncret}\n" + async_methods += f"\t{name}({type_name}, {cbname}) error\n" + callbacks += f"type {cbname} func({cbarg})\n" + + iface = string_to_code( + TEMPLATE_COMMAND.format( + methods=methods[:-1], + callbacks=callbacks[:-1], + async_methods=async_methods[:-1], + ) + ) + return content, iface def generate_template_event(events: dict[str, Tuple[str, str]]) -> (str, str): @@ -1085,12 +1123,13 @@ def __init__(self, _: str): "union": ["encoding/json", "errors", "fmt"], } interfaces = { + "command": [], "event": ["time"], } self.schema: QAPISchema self.events: dict[str, Tuple[str, str]] = {} - self.commands: dict[str, Tuple[str, str]] = {} + self.commands: dict[str, Tuple[str, str, str]] = {} self.golang_package_name = "qapi" self.duplicate = list(gofiles) self.enums: dict[str, str] = {} @@ -1151,7 +1190,9 @@ def visit_end(self) -> None: self.types["event"] += evtype self.interfaces["event"] += eviface - self.types["command"] += generate_template_command(self.commands) + cmdtype, cmdiface = generate_template_command(self.commands) + self.types["command"] += cmdtype + self.interfaces["command"] += cmdiface def visit_object_type( self, @@ -1334,7 +1375,10 @@ def visit_command( ) ) - self.commands[name] = (type_name, content) + retarg = "" + if ret_type: + retarg = qapi_schema_type_to_go_type(ret_type.name) + self.commands[name] = (type_name, content, retarg) def visit_event( self, From patchwork Fri Feb 14 20:29:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 13975599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55722C021A4 for ; Fri, 14 Feb 2025 20:33:50 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tj2Lv-0005lh-Na; Fri, 14 Feb 2025 15:32:01 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2Kc-00049t-27 for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tj2KX-00022v-Js for qemu-devel@nongnu.org; Fri, 14 Feb 2025 15:30:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1739565032; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JSUGq8fLUZu8+pIjnKvOSl0XeErVHPfIKfXGs8FYWoc=; b=BGSpwnxQRdHfQBsGyBYgYfVBPb+WHLkM0CJvWVQQj3GFMh+RRDXDqg2CWuMU2vtBAbXorB A1XPTTdapR3Ygw33BkBUahPhsn43jMVkMLPIHUknzj5hHtyMFXqAacjF9cgQU1DyGioDFv +dYIdDGuEE+DWwdKXjdDpbhvBoeym3c= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-463-hHUX8muSNG-VTszRJqRU2g-1; Fri, 14 Feb 2025 15:30:31 -0500 X-MC-Unique: hHUX8muSNG-VTszRJqRU2g-1 X-Mimecast-MFC-AGG-ID: hHUX8muSNG-VTszRJqRU2g_1739565030 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B951B196E078 for ; Fri, 14 Feb 2025 20:30:30 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.44.32.23]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 26147180056F; Fri, 14 Feb 2025 20:30:25 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= , Andrea Bolognani Subject: [PATCH v4 11/11] docs: add notes on Golang code generator Date: Fri, 14 Feb 2025 21:29:44 +0100 Message-ID: <20250214202944.69897-12-victortoso@redhat.com> In-Reply-To: <20250214202944.69897-1-victortoso@redhat.com> References: <20250214202944.69897-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -37 X-Spam_score: -3.8 X-Spam_bar: --- X-Spam_report: (-3.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.732, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org The goal of this patch is converge discussions into a documentation, to make it easy and explicit design decisions, known issues and what else might help a person interested in how the Go module is generated. Signed-off-by: Victor Toso --- docs/devel/index-build.rst | 1 + docs/devel/qapi-golang-code-gen.rst | 420 ++++++++++++++++++++++++++++ 2 files changed, 421 insertions(+) create mode 100644 docs/devel/qapi-golang-code-gen.rst diff --git a/docs/devel/index-build.rst b/docs/devel/index-build.rst index 0745c81a26..13cc0646c6 100644 --- a/docs/devel/index-build.rst +++ b/docs/devel/index-build.rst @@ -12,4 +12,5 @@ some of the basics if you are adding new files and targets to the build. kconfig docs qapi-code-gen + qapi-golang-code-gen control-flow-integrity diff --git a/docs/devel/qapi-golang-code-gen.rst b/docs/devel/qapi-golang-code-gen.rst new file mode 100644 index 0000000000..b39c14cfd5 --- /dev/null +++ b/docs/devel/qapi-golang-code-gen.rst @@ -0,0 +1,420 @@ +========================== +QAPI Golang code generator +========================== + +.. + Copyright (C) 2025 Red Hat, Inc. + + This work is licensed under the terms of the GNU GPL, version 2 or + later. See the COPYING file in the top-level directory. + + +Introduction +============ + +This document provides information of how the generated Go code maps +with the QAPI specification, clarifying design decisions when needed. + + +Scope of the generated Go code +============================== + +The scope is to provide data structures that can interpret and be used +to generate valid QMP messages. These data structures are generated +from a QAPI schema and should be able to handle QMP messages from the +same schema. + +We also provide interfaces for Commands and Events which allows an +abstraction for client and server applications with the possibility of +custom back end implantations. + +The generated Go code is a Go module with data structs that uses Go +standard library ``encoding/json``, implementing its field tags and +Marshal interface whenever needed. + + +QAPI Documentation +================== + +The documentation included in QAPI schema such as type and type's +fields information, comments, examples and more, they are converted +and embed in the Go generated source code. Metadata information that +might not be relevant to developers are excluded (e.g: TODOs) + + +QAPI types to Go structs +======================== + +Enum +---- + +Enums are mapped as strings in Go, using a specified string type per +Enum to help with type safety in the Go application. + +:: + + { 'enum': 'HostMemPolicy', + 'data': [ 'default', 'preferred', 'bind', 'interleave' ] } + +.. code-block:: go + + // Host memory policy types + // + // Since: 2.1 + type HostMemPolicy string + + const ( + // restore default policy, remove any nondefault policy + HostMemPolicyDefault HostMemPolicy = "default" + // set the preferred host nodes for allocation + HostMemPolicyPreferred HostMemPolicy = "preferred" + // a strict policy that restricts memory allocation to the host nodes specified + HostMemPolicyBind HostMemPolicy = "bind" + // memory allocations are interleaved across the set of host nodes specified + HostMemPolicyInterleave HostMemPolicy = "interleave" + ) + + +Struct +------ + +The mapping between a QAPI struct in Go struct is very straightforward. + - Each member of the QAPI struct has its own field in a Go struct. + - Optional members are pointers type with 'omitempty' field tag set + +One important design decision was to _not_ embed base struct, copying +the base members to the original struct. This reduces the complexity +for the Go application. + +:: + + { 'struct': 'BlockExportOptionsNbdBase', + 'data': { '*name': 'str', '*description': 'str' } } + + { 'struct': 'BlockExportOptionsNbd', + 'base': 'BlockExportOptionsNbdBase', + 'data': { '*bitmaps': ['BlockDirtyBitmapOrStr'], + '*allocation-depth': 'bool' } } + +.. code-block:: go + + // An NBD block export (distinct options used in the NBD branch of + // block-export-add). + // + // Since: 5.2 + type BlockExportOptionsNbd struct { + // Export name. If unspecified, the @device parameter is used as + // the export name. (Since 2.12) + Name *string `json:"name,omitempty"` + // Free-form description of the export, up to 4096 bytes. (Since + // 5.0) + Description *string `json:"description,omitempty"` + // Also export each of the named dirty bitmaps reachable from + // @device, so the NBD client can use NBD_OPT_SET_META_CONTEXT + // with the metadata context name "qemu:dirty-bitmap:BITMAP" to + // inspect each bitmap. Since 7.1 bitmap may be specified by + // node/name pair. + Bitmaps []BlockDirtyBitmapOrStr `json:"bitmaps,omitempty"` + // Also export the allocation depth map for @device, so the NBD + // client can use NBD_OPT_SET_META_CONTEXT with the metadata + // context name "qemu:allocation-depth" to inspect allocation + // details. (since 5.2) + AllocationDepth *bool `json:"allocation-depth,omitempty"` + } + + +Union +----- + +Unions in QAPI are bounded to a Enum type which provides all possible +branches of the union. The most important caveat here is that the Union +does not need to have a complex type implemented for all possible +branches of the Enum. Receiving a enum value of a empty branch is valid. + +The generated Go struct will then define a field for each +Enum value. The type for Enum values of empty branch is bool. Only one +field can be set at time. + +:: + + { 'union': 'ImageInfoSpecificQCow2Encryption', + 'base': 'ImageInfoSpecificQCow2EncryptionBase', + 'discriminator': 'format', + 'data': { 'luks': 'QCryptoBlockInfoLUKS' } } + + { 'struct': 'ImageInfoSpecificQCow2EncryptionBase', + 'data': { 'format': 'BlockdevQcow2EncryptionFormat'}} + + { 'enum': 'BlockdevQcow2EncryptionFormat', + 'data': [ 'aes', 'luks' ] } + +.. code-block:: go + + type ImageInfoSpecificQCow2Encryption struct { + // Variants fields + Luks *QCryptoBlockInfoLUKS `json:"-"` + // Empty branched enum fields + Aes bool `json:"-"` + } + + func (s ImageInfoSpecificQCow2Encryption) MarshalJSON() ([]byte, error) { + // ... + // Logic for branched Enum + if s.Luks != nil && err == nil { + if len(bytes) != 0 { + err = errors.New(`multiple variant fields set`) + } else if err = unwrapToMap(m, s.Luks); err == nil { + m["format"] = BlockdevQcow2EncryptionFormatLuks + bytes, err = json.Marshal(m) + } + } + + // Logic for unbranched Enum + if s.Aes && err == nil { + if len(bytes) != 0 { + err = errors.New(`multiple variant fields set`) + } else { + m["format"] = BlockdevQcow2EncryptionFormatAes + bytes, err = json.Marshal(m) + } + } + + // ... + // Handle errors + } + + + func (s *ImageInfoSpecificQCow2Encryption) UnmarshalJSON(data []byte) error { + // ... + + switch tmp.Format { + case BlockdevQcow2EncryptionFormatLuks: + s.Luks = new(QCryptoBlockInfoLUKS) + if err := json.Unmarshal(data, s.Luks); err != nil { + s.Luks = nil + return err + } + case BlockdevQcow2EncryptionFormatAes: + s.Aes = true + + default: + return fmt.Errorf("error: unmarshal: ImageInfoSpecificQCow2Encryption: received unrecognized value: '%s'", + tmp.Format) + } + return nil + } + + +Alternate +--------- + +Like Unions, alternates can have branches. Unlike Unions, they don't +have a discriminator field and each branch should be a different class +of Type entirely (e.g: You can't have two branches of type int in one +Alternate). + +While the marshalling is similar to Unions, the unmarshalling uses a +try-and-error approach, trying to fit the data payload in one of the +Alternate fields. + +The biggest caveat is handling Alternates that can take JSON Null as +value. The issue lies on ``encoding/json`` library limitation where +unmarshalling JSON Null data to a Go struct which has the 'omitempty' +field as it will bypass the Marshal interface. The same happens when +marshalling, if the field tag 'omitempty' is used, a nil pointer would +never be translated to null JSON value. The problem here is that we do +use pointer to type plus ``omitempty`` field to express a QAPI +optional member. + +In order to handle JSON Null, the generator needs to do the following: + - Read the QAPI schema prior to generate any code and cache + all alternate types that can take JSON Null + - For all Go structs that should be considered optional and they type + are one of those alternates, do not set ``omitempty`` and implement + Marshal interface for this Go struct, to properly handle JSON Null + - In the Alternate, uses a boolean 'IsNull' to express a JSON Null + and implement the AbsentAlternate interface, to help structs know + if a given Alternate type should be considered Absent (not set) or + any other possible Value, including JSON Null. + +:: + + { 'alternate': 'BlockdevRefOrNull', + 'data': { 'definition': 'BlockdevOptions', + 'reference': 'str', + 'null': 'null' } } + +.. code-block:: go + + // Reference to a block device. + // + // Since: 2.9 + type BlockdevRefOrNull struct { + // defines a new block device inline + Definition *BlockdevOptions + // references the ID of an existing block device. An empty string + // means that no block device should be referenced. Deprecated; + // use null instead. + Reference *string + // No block device should be referenced (since 2.10) + IsNull bool + } + + func (s *BlockdevRefOrNull) ToAnyOrAbsent() (any, bool) { + if s != nil { + if s.IsNull { + return nil, false + } else if s.Definition != nil { + return *s.Definition, false + } else if s.Reference != nil { + return *s.Reference, false + } + } + + return nil, true + } + + func (s BlockdevRefOrNull) MarshalJSON() ([]byte, error) { + if s.IsNull { + return []byte("null"), nil + } else if s.Definition != nil { + return json.Marshal(s.Definition) + } else if s.Reference != nil { + return json.Marshal(s.Reference) + } + return []byte("{}"), nil + } + + func (s *BlockdevRefOrNull) UnmarshalJSON(data []byte) error { + // Check for json-null first + if string(data) == "null" { + s.IsNull = true + return nil + } + // Check for BlockdevOptions + { + s.Definition = new(BlockdevOptions) + if err := StrictDecode(s.Definition, data); err == nil { + return nil + } + s.Definition = nil + } + + // Check for string + { + s.Reference = new(string) + if err := StrictDecode(s.Reference, data); err == nil { + return nil + } + s.Reference = nil + } + + return fmt.Errorf("Can't convert to BlockdevRefOrNull: %s", string(data)) + } + + +Event +----- + +Each event is mapped to its own struct with. + +:: + + { 'event': 'SHUTDOWN', + 'data': { 'guest': 'bool', + 'reason': 'ShutdownCause' } } + +.. code-block:: go + + // Emitted when the virtual machine has shut down, indicating that + // qemu is about to exit. + // + // .. note:: If the command-line option "-no-shutdown" has been + // specified, qemu will not exit, and a STOP event will eventually + // follow the SHUTDOWN event. + // + // Since: 0.12 + // + // .. qmp-example:: <- { "event": "SHUTDOWN", "data": { + // "guest": true, "reason": "guest-shutdown" }, "timestamp": { + // "seconds": 1267040730, "microseconds": 682951 } } + type ShutdownEvent struct { + // If true, the shutdown was triggered by a guest request (such as + // a guest-initiated ACPI shutdown request or other hardware- + // specific action) rather than a host request (such as sending + // qemu a SIGINT). (since 2.10) + Guest bool `json:"guest"` + // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0) + Reason ShutdownCause `json:"reason"` + } + + +Command +------- + +Each commands is mapped to its own struct. If the command has a boxed +data struct, the option struct will be embed in the command struct. + +The return value is always a well defined type and it is part of first +layer unmarshalling type, Message. + +:: + + { 'command': 'set_password', + 'boxed': true, + 'data': 'SetPasswordOptions' } + + { 'union': 'SetPasswordOptions', + 'base': { 'protocol': 'DisplayProtocol', + 'password': 'str', + '*connected': 'SetPasswordAction' }, + 'discriminator': 'protocol', + 'data': { 'vnc': 'SetPasswordOptionsVnc' } } + +.. code-block:: go + + // Set the password of a remote display server. + // Errors: - If Spice is not enabled, DeviceNotFound + // + // Since: 0.14 + // + // .. qmp-example:: -> { "execute": "set_password", "arguments": { + // "protocol": "vnc", "password": "secret" } + // } <- { "return": {} } + type SetPasswordCommand struct { + SetPasswordOptions + } + +Now an example of a command without boxed type. + +:: + + { 'command': 'set_link', + 'data': {'name': 'str', 'up': 'bool'} } + +.. code-block:: go + + // Sets the link status of a virtual network adapter. + // + // Errors: - If @name is not a valid network device, DeviceNotFound + // + // Since: 0.14 + // + // .. note:: Not all network adapters support setting link status. + // This command will succeed even if the network adapter does not + // support link status notification. .. qmp-example:: -> { + // "execute": "set_link", "arguments": { "name": "e1000.0", "up": + // false } } <- { "return": {} } + type SetLinkCommand struct { + // the device name of the virtual network adapter + Name string `json:"name"` + // true to set the link status to be up + Up bool `json:"up"` + } + +Known issues +============ + +- Type names might not follow proper Go convention. Andrea suggested an + annotation to the QAPI schema that could solve it. + https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html