From patchwork Thu Apr 22 03:06:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9307AC433ED for ; Thu, 22 Apr 2021 03:11:49 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 134CD61422 for ; Thu, 22 Apr 2021 03:11:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 134CD61422 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50280 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPkq-0003Fb-70 for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:11:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49020) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgr-0006LG-9R for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:34073) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgc-0006Pp-Un for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gEz07BQQAy3Wf56JTsirqqR3UI5WCp9HQvhee+cJaj0=; b=I6gjdt3pnOPtGxWvmK/KdSOpTRwDviVp1nKWI7VlegpBT26LvorkGlST3tgpuU+26mr2zk x2d4w6ZvZjxh8/CHx0ubVf8nzeo/uWYoYzetMYV4at1ztcV6BUlT7bq3PrQsMj5eouKaCI HigX9y+X3JGy7G//q4fXgj/WuMhG9kA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-339-xJ9f6RPdMZ-MTmr1Ly8KGw-1; Wed, 21 Apr 2021 23:07:23 -0400 X-MC-Unique: xJ9f6RPdMZ-MTmr1Ly8KGw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4567E8189C8; Thu, 22 Apr 2021 03:07:22 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BA422915B; Thu, 22 Apr 2021 03:07:21 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 01/22] qapi/parser: Don't try to handle file errors Date: Wed, 21 Apr 2021 23:06:59 -0400 Message-Id: <20210422030720.3685766-2-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The short-ish version of what motivates this patch is: - The parser initializer does not possess adequate context to write a good error message -- It tries to determine the caller's semantic context. - We don't want to allow QAPISourceInfo(None, None, None) to exist. - Errors made using such an object are currently incorrect. - It's not technically a semantic error if we cannot open the schema - There are various typing constraints that make mixing these two cases undesirable for a single special case. - The current open block in parser's initializer will leak file pointers, because it isn't using a with statement. Here's the details in why this got written the way it did, and why a few disparate issues are rolled into one commit. (They're hard to fix separately without writing really weird stuff that'd be harder to review.) The error message string here is incorrect: > python3 qapi-gen.py 'fake.json' qapi-gen.py: qapi-gen.py: can't read schema file 'fake.json': No such file or directory In pursuing it, we find that QAPISourceInfo has a special accommodation for when there's no filename. Meanwhile, we intend to type info.fname as str; something we always have. To remove this, we need to not have a "fake" QAPISourceInfo object. We also don't want to explicitly begin accommodating QAPISourceInfo being None, because we actually want to eventually prove that this can never happen -- We don't want to confuse "The file isn't open yet" with "This error stems from a definition that wasn't defined in any file". (An earlier series tried to create an official dummy object, but it was tough to prove in review that it worked correctly without creating new regressions. This patch avoids trying to re-litigate that discussion. We would like to first prove that we never raise QAPISemError for any built-in object before we relent and add "special" info objects. We aren't ready to do that yet, so crashing is preferred.) So, how to solve this mess? Here's one way: Don't try to handle errors at a level with "mixed" semantic levels; i.e. don't try to handle inclusion errors (should report a source line where the include was triggered) with command line errors (where we specified a file we couldn't read). Simply remove the error handling from the initializer of the parser. Pythonic! Now it's the caller's job to figure out what to do about it. Handle the error in QAPISchemaParser._include() instead, where we do have the correct semantic context to not need to play games with the error message generation. Next, to re-gain a nice error at the top level, add a new try/except into qapi/main.generate(). Now the error looks sensible: > python3 qapi-gen.py 'fake.json' qapi-gen.py: can't read schema file 'fake.json': No such file or directory Lastly, with this usage gone, we can remove the special type violation from QAPISourceInfo, and all is well with the world. Signed-off-by: John Snow --- scripts/qapi/main.py | 8 +++++++- scripts/qapi/parser.py | 18 +++++++++--------- scripts/qapi/source.py | 3 --- 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py index 703e7ed1ed5..70f8aa86f37 100644 --- a/scripts/qapi/main.py +++ b/scripts/qapi/main.py @@ -48,7 +48,13 @@ def generate(schema_file: str, """ assert invalid_prefix_char(prefix) is None - schema = QAPISchema(schema_file) + try: + schema = QAPISchema(schema_file) + except OSError as err: + raise QAPIError( + f"can't read schema file '{schema_file}': {err.strerror}" + ) from err + gen_types(schema, output_dir, prefix, builtins) gen_visit(schema, output_dir, prefix, builtins) gen_commands(schema, output_dir, prefix) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index ca5e8e18e00..b378fa33807 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -40,15 +40,9 @@ def __init__(self, fname, previously_included=None, incl_info=None): previously_included = previously_included or set() previously_included.add(os.path.abspath(fname)) - try: - fp = open(fname, 'r', encoding='utf-8') + # Allow the caller to catch this error. + with open(fname, 'r', encoding='utf-8') as fp: self.src = fp.read() - except IOError as e: - raise QAPISemError(incl_info or QAPISourceInfo(None, None, None), - "can't read %s file '%s': %s" - % ("include" if incl_info else "schema", - fname, - e.strerror)) if self.src == '' or self.src[-1] != '\n': self.src += '\n' @@ -129,7 +123,13 @@ def _include(self, include, info, incl_fname, previously_included): if incl_abs_fname in previously_included: return None - return QAPISchemaParser(incl_fname, previously_included, info) + try: + return QAPISchemaParser(incl_fname, previously_included, info) + except OSError as err: + raise QAPISemError( + info, + f"can't read include file '{incl_fname}': {err.strerror}" + ) from err def _check_pragma_list_of_str(self, name, value, info): if (not isinstance(value, list) diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py index 03b6ede0828..1ade864d7b9 100644 --- a/scripts/qapi/source.py +++ b/scripts/qapi/source.py @@ -10,7 +10,6 @@ # See the COPYING file in the top-level directory. import copy -import sys from typing import List, Optional, TypeVar @@ -53,8 +52,6 @@ def next_line(self: T) -> T: return info def loc(self) -> str: - if self.fname is None: - return sys.argv[0] ret = self.fname if self.line is not None: ret += ':%d' % self.line From patchwork Thu Apr 22 03:07:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86C20C433ED for ; Thu, 22 Apr 2021 03:15:25 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0197A6142F for ; Thu, 22 Apr 2021 03:15:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0197A6142F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPoJ-0006pf-O2 for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:15:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49094) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgw-0006PT-BJ for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:35075) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006Ru-Jv for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yO8sw/BnHE8AhkP8SqeYby/5a9XooTxfhl4rIp/cyk4=; b=fawAknjxqQTM9VwR7ZKQfrTW6npRFmrPF5og9kjRjM69PrLz6dgZO7Fg5H1EbWuW2d3Zuu 85S79kT8Ie4Sc/BMbyDx3eJzNDhIDyisRZ/KDuUn/v7YvQpMtlthVpTa5QBMZyt/jflxXB fseFhQ2bDCOPtZp8SoZS5MalB/nRNWA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-139-yNM_zkLzNy6YG3-fwq3x2w-1; Wed, 21 Apr 2021 23:07:24 -0400 X-MC-Unique: yNM_zkLzNy6YG3-fwq3x2w-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2E1481006C80; Thu, 22 Apr 2021 03:07:23 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6D60B607CB; Thu, 22 Apr 2021 03:07:22 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 02/22] qapi/source: [RFC] add "with_column" contextmanager Date: Wed, 21 Apr 2021 23:07:00 -0400 Message-Id: <20210422030720.3685766-3-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is a silly one, but... it's important to have fun. This patch isn't *needed*, it's here as an RFC. In trying to experiment with different ways to solve the problem addressed by the previous commit, I kept getting confused at how the "source location" string with line and column number was built across two different classes. (i.e. QAPISourceError appends the column, but QAPISourceInfo does not track column information natively.) I was afraid to try and fully implement column number directly in QAPISourceInfo on the chance that it might have undesirable effects, so I came up with a quick "hack" to centralize the 'location' information generation. It's a little goofy, but it works :') Signed-off-by: John Snow --- scripts/qapi/error.py | 8 +++----- scripts/qapi/source.py | 23 ++++++++++++++++++++++- 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/scripts/qapi/error.py b/scripts/qapi/error.py index e35e4ddb26a..6b04f56f8a2 100644 --- a/scripts/qapi/error.py +++ b/scripts/qapi/error.py @@ -39,11 +39,9 @@ def __init__(self, def __str__(self) -> str: assert self.info is not None - loc = str(self.info) - if self.col is not None: - assert self.info.line is not None - loc += ':%s' % self.col - return loc + ': ' + self.msg + with self.info.at_column(self.col): + loc = str(self.info) + return f"{loc}: {self.msg}" class QAPISemError(QAPISourceError): diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py index 1ade864d7b9..21090b9fe78 100644 --- a/scripts/qapi/source.py +++ b/scripts/qapi/source.py @@ -9,8 +9,14 @@ # This work is licensed under the terms of the GNU GPL, version 2. # See the COPYING file in the top-level directory. +from contextlib import contextmanager import copy -from typing import List, Optional, TypeVar +from typing import ( + Iterator, + List, + Optional, + TypeVar, +) class QAPISchemaPragma: @@ -35,6 +41,7 @@ def __init__(self, fname: str, line: int, parent: Optional['QAPISourceInfo']): self.fname = fname self.line = line + self._column: Optional[int] = None self.parent = parent self.pragma: QAPISchemaPragma = ( parent.pragma if parent else QAPISchemaPragma() @@ -52,9 +59,14 @@ def next_line(self: T) -> T: return info def loc(self) -> str: + # column cannot be provided meaningfully when line is absent. + assert self.line or self._column is None + ret = self.fname if self.line is not None: ret += ':%d' % self.line + if self._column is not None: + ret += ':%d' % self._column return ret def in_defn(self) -> str: @@ -71,5 +83,14 @@ def include_path(self) -> str: parent = parent.parent return ret + @contextmanager + def at_column(self, column: Optional[int]) -> Iterator[None]: + current_column = self._column + try: + self._column = column + yield + finally: + self._column = current_column + def __str__(self) -> str: return self.include_path() + self.in_defn() + self.loc() From patchwork Thu Apr 22 03:07:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDE6BC433ED for ; Thu, 22 Apr 2021 03:08:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 54DE261428 for ; Thu, 22 Apr 2021 03:08:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 54DE261428 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42044 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPi4-00082k-9o for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:08:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49022) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgr-0006LH-9U for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:36361) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006Ro-HV for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I7gEPdebCE6paPsNskhQIQjVSM3mJoBcyzJdr5zoIOc=; b=DuVdFvpevaKHXRJpXMbX7ovXNuGMpu+BAtZcYMnLI1PGn7IfbnLEsYgmpPc5xZzLAICjpO H6s1ZV9yNlViB5hnGRH4P2YwpWG2zLpWxC6qN7dijpkj9WDa3NgkAXGG5IcuRIZzmHWVax FY6vtKWVi7jNgEf7RMupEyIB6yqUeIo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-216-VT97YbiIMr-3bPzbwYNgqQ-1; Wed, 21 Apr 2021 23:07:24 -0400 X-MC-Unique: VT97YbiIMr-3bPzbwYNgqQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0EE3B801814; Thu, 22 Apr 2021 03:07:24 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 54364607CB; Thu, 22 Apr 2021 03:07:23 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 03/22] qapi/source: Remove line number from QAPISourceInfo initializer Date: Wed, 21 Apr 2021 23:07:01 -0400 Message-Id: <20210422030720.3685766-4-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" With the QAPISourceInfo(None, None, None) construct gone, there's not really any reason to have to specify that a file starts on the first line. Remove it from the initializer and have it default to 1. Remove the last vestiges where we check for 'line' being unset. That won't happen again, now! Signed-off-by: John Snow --- scripts/qapi/parser.py | 2 +- scripts/qapi/source.py | 12 +++--------- 2 files changed, 4 insertions(+), 10 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index b378fa33807..edd0af33ae0 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -47,7 +47,7 @@ def __init__(self, fname, previously_included=None, incl_info=None): if self.src == '' or self.src[-1] != '\n': self.src += '\n' self.cursor = 0 - self.info = QAPISourceInfo(fname, 1, incl_info) + self.info = QAPISourceInfo(fname, incl_info) self.line_pos = 0 self.exprs = [] self.docs = [] diff --git a/scripts/qapi/source.py b/scripts/qapi/source.py index 21090b9fe78..afa21518974 100644 --- a/scripts/qapi/source.py +++ b/scripts/qapi/source.py @@ -37,10 +37,9 @@ def __init__(self) -> None: class QAPISourceInfo: T = TypeVar('T', bound='QAPISourceInfo') - def __init__(self, fname: str, line: int, - parent: Optional['QAPISourceInfo']): + def __init__(self, fname: str, parent: Optional['QAPISourceInfo'] = None): self.fname = fname - self.line = line + self.line = 1 self._column: Optional[int] = None self.parent = parent self.pragma: QAPISchemaPragma = ( @@ -59,12 +58,7 @@ def next_line(self: T) -> T: return info def loc(self) -> str: - # column cannot be provided meaningfully when line is absent. - assert self.line or self._column is None - - ret = self.fname - if self.line is not None: - ret += ':%d' % self.line + ret = f"{self.fname}:{self.line}" if self._column is not None: ret += ':%d' % self._column return ret From patchwork Thu Apr 22 03:07:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 918FBC433B4 for ; Thu, 22 Apr 2021 03:15:26 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ED57E61430 for ; Thu, 22 Apr 2021 03:15:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED57E61430 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58972 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPoK-0006sD-SO for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:15:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49076) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgv-0006Mz-Gf for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:25146) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006Rv-KB for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RBSpPHs0AsRbpBEzwdSQYY7vEN3RjpPhj0epOcml6Oc=; b=DcvxABnlc8YMWHQVqidabPGDiAHoDTFpSVEOFRGbRUA2F2JEHfo8pFtMOzkjX+Sda/29Vv CR83AnUbZJdm3E4Xhx2OHjJLEHOQlhIUmkqNDqGMmotJMalORc0J9hXWKIBMl6A5gLfkHh raCAFtgZiBCULaTi2C5vpnakmj3U5Y0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-154-jbIavhwqMH-3DeRISyuhDQ-1; Wed, 21 Apr 2021 23:07:25 -0400 X-MC-Unique: jbIavhwqMH-3DeRISyuhDQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E580A107ACCD; Thu, 22 Apr 2021 03:07:24 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 347AC607CB; Thu, 22 Apr 2021 03:07:24 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 04/22] qapi/parser: factor parsing routine into method Date: Wed, 21 Apr 2021 23:07:02 -0400 Message-Id: <20210422030720.3685766-5-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" For the sake of keeping __init__ smaller (and treating it more like a gallery of what state variables we can expect to see), put the actual parsing action into a parse method. We can still invoke it from the init method to reduce churn. To accomplish this, 'previously_included' because the private data member '_included', and the filename is stashed as _fname. Add any missing declarations to the init method, and group them by function so they can be understood quickly at a glance. Signed-off-by: John Snow --- scripts/qapi/parser.py | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index edd0af33ae0..f519518075e 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -37,23 +37,38 @@ def __init__(self, parser, msg): class QAPISchemaParser: def __init__(self, fname, previously_included=None, incl_info=None): - previously_included = previously_included or set() - previously_included.add(os.path.abspath(fname)) + self._fname = fname + self._included = previously_included or set() + self._included.add(os.path.abspath(self._fname)) + self.src = '' - # Allow the caller to catch this error. - with open(fname, 'r', encoding='utf-8') as fp: - self.src = fp.read() - - if self.src == '' or self.src[-1] != '\n': - self.src += '\n' + # Lexer state (see `accept` for details): + self.info = QAPISourceInfo(self._fname, incl_info) + self.tok = None + self.pos = 0 self.cursor = 0 - self.info = QAPISourceInfo(fname, incl_info) + self.val = None self.line_pos = 0 + + # Parser output: self.exprs = [] self.docs = [] - self.accept() + + # Showtime! + self._parse() + + def _parse(self): cur_doc = None + with open(self._fname, 'r', encoding='utf-8') as fp: + self.src = fp.read() + if self.src == '' or self.src[-1] != '\n': + self.src += '\n' + + # Prime the lexer: + self.accept() + + # Parse until done: while self.tok is not None: info = self.info if self.tok == '#': @@ -71,12 +86,12 @@ def __init__(self, fname, previously_included=None, incl_info=None): if not isinstance(include, str): raise QAPISemError(info, "value of 'include' must be a string") - incl_fname = os.path.join(os.path.dirname(fname), + incl_fname = os.path.join(os.path.dirname(self._fname), include) self.exprs.append({'expr': {'include': incl_fname}, 'info': info}) exprs_include = self._include(include, info, incl_fname, - previously_included) + self._included) if exprs_include: self.exprs.extend(exprs_include.exprs) self.docs.extend(exprs_include.docs) From patchwork Thu Apr 22 03:07:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98E1BC433B4 for ; Thu, 22 Apr 2021 03:11:52 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2A9DD61422 for ; Thu, 22 Apr 2021 03:11:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2A9DD61422 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50594 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPkt-0003Op-8O for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:11:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49046) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgt-0006Lb-A8 for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:43 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:29030) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006SK-Jx for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pL1l/VqQlyd061vDNxj2U4genMFEVf+bwm5y/ryP3NM=; b=STeoWLmwxnuQZv+YqW0OkJyzvKv+L0nxrz6fSTN0mAajNq30tMhSQfk9x9At01lsPCWOly DrXN7MJGiLcPqRv3KBP3gOBDk/XJfmxidG2ZDbP3MgfISbx1HUL+/rOYMjdap/AQmBSq90 cfreysqxJYbnYsRBPCoOSP9elraxcng= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-167-Dwg_77vuNJ2vXGruatDb_w-1; Wed, 21 Apr 2021 23:07:27 -0400 X-MC-Unique: Dwg_77vuNJ2vXGruatDb_w-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CAD878030A1; Thu, 22 Apr 2021 03:07:25 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 19E29607CB; Thu, 22 Apr 2021 03:07:25 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 05/22] qapi/parser: Assert lexer value is a string Date: Wed, 21 Apr 2021 23:07:03 -0400 Message-Id: <20210422030720.3685766-6-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The type checker can't narrow the type of the token value to string, because it's only loosely correlated with the return token. We know that a token of '#' should always have a "str" value. Add an assertion. Signed-off-by: John Snow --- scripts/qapi/parser.py | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index f519518075e..c75434e75a5 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -303,6 +303,7 @@ def get_doc(self, info): cur_doc = QAPIDoc(self, info) self.accept(False) while self.tok == '#': + assert isinstance(self.val, str), "Expected str value" if self.val.startswith('##'): # End of doc comment if self.val != '##': From patchwork Thu Apr 22 03:07:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E978FC43460 for ; Thu, 22 Apr 2021 03:12:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B0E7E61435 for ; Thu, 22 Apr 2021 03:12:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B0E7E61435 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50966 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPl4-0003YE-LY for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:12:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49066) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgv-0006MG-5E for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20510) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006SP-K8 for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PFAT3UhFNT6M1PcQh3K2YyKNd9ww7GL+/z+LIKs2t1s=; b=Gr5rvrvjBdG0+boAWEZhGCflVLX5l/gOVaVUL+8a+3J+Rqa8A7WdKP6Zpuy88Ao+eFcWSi M8ph/xp2o76K49Q15hFPErVg/h+mtT3WIiubCL5ACzdTqos5fpb1v6Km8FMUqR5hLRDz/O 9ksUNDuzb/3X2KnxCg6TV5uks/9vApI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-491-xu3YGywuMiayKcmpK0iu9g-1; Wed, 21 Apr 2021 23:07:27 -0400 X-MC-Unique: xu3YGywuMiayKcmpK0iu9g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AFD8410054F6; Thu, 22 Apr 2021 03:07:26 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id F1C1D607CB; Thu, 22 Apr 2021 03:07:25 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 06/22] qapi/parser: assert get_expr returns object in outer loop Date: Wed, 21 Apr 2021 23:07:04 -0400 Message-Id: <20210422030720.3685766-7-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" get_expr can return many things, depending on where it is used. In the outer parsing loop, we expect and require it to return a dict. (It's (maybe) a bit involved to teach mypy that when nested is False, this is already always True. I'll look into it later, maybe.) Signed-off-by: John Snow --- scripts/qapi/parser.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index c75434e75a5..6b443b1247e 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -78,6 +78,8 @@ def _parse(self): continue expr = self.get_expr(False) + assert isinstance(expr, dict) # Guaranteed when nested=False + if 'include' in expr: self.reject_expr_doc(cur_doc) if len(expr) != 1: @@ -278,6 +280,7 @@ def get_values(self): self.accept() def get_expr(self, nested): + # TODO: Teach mypy that nested=False means the retval is a Dict. if self.tok != '{' and not nested: raise QAPIParseError(self, "expected '{'") if self.tok == '{': From patchwork Thu Apr 22 03:07:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 030A0C433ED for ; Thu, 22 Apr 2021 03:09:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7632161428 for ; Thu, 22 Apr 2021 03:09:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7632161428 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42336 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPiB-00089o-Du for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:09:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49092) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgw-0006P0-6r for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:55451) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006Sk-Jv for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060850; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2olikWbH/SV+OVQXcFptnescSY5UJmO6DKAftYO0vSY=; b=QWPfntVlAxrojhWEd8DPiw/6WnKK7mMxRIyne1+BLExzKg0gtGK74C/FcXnqqhLTAhlnVD qjuQ1AApPMRbAeUEVWuiAH7aNHSOEH7uQFlIzNDYp+zG4Bi1oiFLGGT9gqFMF8Ltp1aqCP f+hAqnyqXz9PT24BIl3aKD1o78CYfgI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-413-cP0mXC9cMye4T6LlGQiOlg-1; Wed, 21 Apr 2021 23:07:28 -0400 X-MC-Unique: cP0mXC9cMye4T6LlGQiOlg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 94159343A2; Thu, 22 Apr 2021 03:07:27 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id D848D2915B; Thu, 22 Apr 2021 03:07:26 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 07/22] qapi/parser: assert object keys are strings Date: Wed, 21 Apr 2021 23:07:05 -0400 Message-Id: <20210422030720.3685766-8-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The single quote token implies the value is a string. Assert this to be the case. Signed-off-by: John Snow --- scripts/qapi/parser.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 6b443b1247e..8d1fe0ddda5 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -246,6 +246,8 @@ def get_members(self): raise QAPIParseError(self, "expected string or '}'") while True: key = self.val + assert isinstance(key, str) # Guaranteed by tok == "'" + self.accept() if self.tok != ':': raise QAPIParseError(self, "expected ':'") From patchwork Thu Apr 22 03:07:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06AC2C433B4 for ; Thu, 22 Apr 2021 03:09:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 756BA61404 for ; Thu, 22 Apr 2021 03:09:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 756BA61404 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:42328 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPiB-00089g-AW for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:09:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49086) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgv-0006Oc-UY for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:39173) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgh-0006T6-K4 for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060851; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7taK/l9BVxBgkc2nBxqRCRx3NNXPpwjFOzRWukSv+eg=; b=ePbxG0Zw0quq6VOoc/V/d4VncLVabXMTfBcDTouzts84wh2rXuVEWCXe0DxGsqE4M/bIND i+uSSHAdxaBv4RocidmTp4oP6WnFtgBedobXgbqMBcLSfIZ6DQL0njlHbv+KA2mbQ6MZ8Z GfjyPvQaNBJrH0KxGySa4iE8wURjpzQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-528-GlSbvs-DPJKK4QcAeueV1A-1; Wed, 21 Apr 2021 23:07:29 -0400 X-MC-Unique: GlSbvs-DPJKK4QcAeueV1A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 74E9B8189C8; Thu, 22 Apr 2021 03:07:28 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA6B0607CB; Thu, 22 Apr 2021 03:07:27 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 08/22] qapi/parser: Use @staticmethod where appropriate Date: Wed, 21 Apr 2021 23:07:06 -0400 Message-Id: <20210422030720.3685766-9-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" No self, no thank you! Signed-off-by: John Snow --- scripts/qapi/parser.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 8d1fe0ddda5..f2425c0228a 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -127,7 +127,8 @@ def reject_expr_doc(doc): "documentation for '%s' is not followed by the definition" % doc.symbol) - def _include(self, include, info, incl_fname, previously_included): + @staticmethod + def _include(include, info, incl_fname, previously_included): incl_abs_fname = os.path.abspath(incl_fname) # catch inclusion cycle inf = info @@ -148,7 +149,8 @@ def _include(self, include, info, incl_fname, previously_included): f"can't read include file '{incl_fname}': {err.strerror}" ) from err - def _check_pragma_list_of_str(self, name, value, info): + @staticmethod + def _check_pragma_list_of_str(name, value, info): if (not isinstance(value, list) or any([not isinstance(elt, str) for elt in value])): raise QAPISemError( From patchwork Thu Apr 22 03:07:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 639D2C433B4 for ; Thu, 22 Apr 2021 03:18:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9B92A61437 for ; Thu, 22 Apr 2021 03:18:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B92A61437 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37372 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPqr-0001EA-O2 for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:18:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49146) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgx-0006Ri-IM for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:50234) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgk-0006TE-RM for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xpkhKvDHe9RkmkEsvTSqCffg5pJnPEUTyuWI8r+TjZk=; b=EPD7npMqIgiAdXFiYMgSph3HmM7Mnf1xsO/a74xhuIaHnqoOSQ/Tj0plbzTNgbq+ymHav3 0F2gMbjZAJVMJtmWWb5sGzco8TaqTAXm1zF5tZHFtmM1KV+MvvP/q+UooXThqtkui8pVoE BAod1zK5UUzctkF0vHrNXOAK/DqkDRQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-306-i6Kw0kCOOGGzp1sbBkOACg-1; Wed, 21 Apr 2021 23:07:30 -0400 X-MC-Unique: i6Kw0kCOOGGzp1sbBkOACg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7416F18397A3; Thu, 22 Apr 2021 03:07:29 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9D263607CB; Thu, 22 Apr 2021 03:07:28 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 09/22] qapi: add match_nofail helper Date: Wed, 21 Apr 2021 23:07:07 -0400 Message-Id: <20210422030720.3685766-10-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Mypy cannot generally understand that these regex functions cannot possibly fail. Add a _nofail helper that clarifies this for mypy. Signed-off-by: John Snow --- scripts/qapi/common.py | 8 +++++++- scripts/qapi/main.py | 6 ++---- scripts/qapi/parser.py | 13 +++++++------ 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py index cbd3fd81d36..d38c1746767 100644 --- a/scripts/qapi/common.py +++ b/scripts/qapi/common.py @@ -12,7 +12,7 @@ # See the COPYING file in the top-level directory. import re -from typing import Optional, Sequence +from typing import Match, Optional, Sequence #: Magic string that gets removed along with all space to its right. @@ -210,3 +210,9 @@ def gen_endif(ifcond: Sequence[str]) -> str: #endif /* %(cond)s */ ''', cond=ifc) return ret + + +def match_nofail(pattern: str, string: str) -> Match[str]: + match = re.match(pattern, string) + assert match is not None + return match diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py index 70f8aa86f37..e8d4ba4b389 100644 --- a/scripts/qapi/main.py +++ b/scripts/qapi/main.py @@ -8,11 +8,11 @@ """ import argparse -import re import sys from typing import Optional from .commands import gen_commands +from .common import match_nofail from .error import QAPIError from .events import gen_events from .introspect import gen_introspect @@ -22,9 +22,7 @@ def invalid_prefix_char(prefix: str) -> Optional[str]: - match = re.match(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) - # match cannot be None, but mypy cannot infer that. - assert match is not None + match = match_nofail(r'([A-Za-z_.-][A-Za-z0-9_.-]*)?', prefix) if match.end() != len(prefix): return prefix[match.end()] return None diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index f2425c0228a..7f3c009f64b 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -18,6 +18,7 @@ import os import re +from .common import match_nofail from .error import QAPISemError, QAPISourceError from .source import QAPISourceInfo @@ -235,8 +236,8 @@ def accept(self, skip_comment=True): elif not self.tok.isspace(): # Show up to next structural, whitespace or quote # character - match = re.match('[^[\\]{}:,\\s\'"]+', - self.src[self.cursor-1:]) + match = match_nofail('[^[\\]{}:,\\s\'"]+', + self.src[self.cursor-1:]) raise QAPIParseError(self, "stray '%s'" % match.group(0)) def get_members(self): @@ -369,7 +370,7 @@ def append(self, line): # Strip leading spaces corresponding to the expected indent level # Blank lines are always OK. if line: - indent = re.match(r'\s*', line).end() + indent = match_nofail(r'\s*', line).end() if indent < self._indent: raise QAPIParseError( self._parser, @@ -505,7 +506,7 @@ def _append_args_line(self, line): # from line and replace it with spaces so that 'f' has the # same index as it did in the original line and can be # handled the same way we will handle following lines. - indent = re.match(r'@\S*:\s*', line).end() + indent = match_nofail(r'@\S*:\s*', line).end() line = line[indent:] if not line: # Line was just the "@arg:" header; following lines @@ -540,7 +541,7 @@ def _append_features_line(self, line): # from line and replace it with spaces so that 'f' has the # same index as it did in the original line and can be # handled the same way we will handle following lines. - indent = re.match(r'@\S*:\s*', line).end() + indent = match_nofail(r'@\S*:\s*', line).end() line = line[indent:] if not line: # Line was just the "@arg:" header; following lines @@ -586,7 +587,7 @@ def _append_various_line(self, line): # from line and replace it with spaces so that 'f' has the # same index as it did in the original line and can be # handled the same way we will handle following lines. - indent = re.match(r'\S*:\s*', line).end() + indent = match_nofail(r'\S*:\s*', line).end() line = line[indent:] if not line: # Line was just the "Section:" header; following lines From patchwork Thu Apr 22 03:07:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C46AC433B4 for ; Thu, 22 Apr 2021 03:23:25 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A924360FE3 for ; Thu, 22 Apr 2021 03:23:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A924360FE3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47860 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPw3-0005hw-Qg for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:23:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49182) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgz-0006Uw-0b for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:40635) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgl-0006TK-Cx for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fu3F09nKvK06zGFS37EYCt26mMCvcXkCpmOT//N3gFU=; b=SPfV9q6nUxtxZxqNUIJZLssMWD1VIj/+8NqDgA1UtS8RUw1GTZkGTU/q0G81tEgrnuydhv pdoyzeNSGDOhyCsgrlFYpi4txYDfS6iX69OPy1UEqmj6EPkyuDmruk4Ku9oUFkHTcz2/6e HkquQZ+XTw1iKYWfA+J88klHhnJUWZE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-359-J4Xbgs2JPMKb7FQm9ekBLQ-1; Wed, 21 Apr 2021 23:07:31 -0400 X-MC-Unique: J4Xbgs2JPMKb7FQm9ekBLQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3C6D5107ACCA; Thu, 22 Apr 2021 03:07:30 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 84376607CB; Thu, 22 Apr 2021 03:07:29 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 10/22] qapi/parser: Fix typing of token membership tests Date: Wed, 21 Apr 2021 23:07:08 -0400 Message-Id: <20210422030720.3685766-11-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" When the token can be None, we can't use 'x in "abc"' style membership tests to group types of tokens together, because 'None in "abc"' is a TypeError. Easy enough to fix, if not a little ugly. Signed-off-by: John Snow --- scripts/qapi/parser.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 7f3c009f64b..16fd36f8391 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -272,7 +272,7 @@ def get_values(self): if self.tok == ']': self.accept() return expr - if self.tok not in "{['tf": + if self.tok is None or self.tok not in "{['tf": raise QAPIParseError( self, "expected '{', '[', ']', string, or boolean") while True: @@ -294,7 +294,8 @@ def get_expr(self, nested): elif self.tok == '[': self.accept() expr = self.get_values() - elif self.tok in "'tf": + elif self.tok and self.tok in "'tf": + assert isinstance(self.val, (str, bool)) expr = self.val self.accept() else: From patchwork Thu Apr 22 03:07:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73255C433B4 for ; Thu, 22 Apr 2021 03:20:51 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1493161437 for ; Thu, 22 Apr 2021 03:20:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1493161437 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43554 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPta-0003nY-0c for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:20:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49148) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgx-0006Rx-NI for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:59842) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgl-0006TV-CD for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KLPp9ExvWedGct+OVfi2fBU7wnsFwYmzoh81Q31Wt90=; b=f4X2/mLDjHyENeIIwgUKWKAf9JPNQo3Otx9O+VA5JxVf9AdIs96APzmB0pDbYTsXBPLoZf PBH+cjCZqq0aHGnvIgiU5F709lIbNwFvubrOpIUW/HXxiuYKz3O2h58/vEQrwF85e3AMID WvoVMjT5BqGZmyS68Am29i8nZYQFwEg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-5-wK7Eong2Pn2yPKFcZOhHcg-1; Wed, 21 Apr 2021 23:07:32 -0400 X-MC-Unique: wK7Eong2Pn2yPKFcZOhHcg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2444C8030A1; Thu, 22 Apr 2021 03:07:31 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 65F98607CB; Thu, 22 Apr 2021 03:07:30 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 11/22] qapi/parser: Rework _check_pragma_list_of_str as a TypeGuard Date: Wed, 21 Apr 2021 23:07:09 -0400 Message-Id: <20210422030720.3685766-12-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" TypeGuards wont exist in Python proper until 3.10. Ah well. We can hack up our own by declaring this function to return the type we claim it checks for and using this to safely downcast object -> List[str]. In so doing, I bring this function in-line under _pragma so it can use the 'info' object in its closure. Having done this, _pragma also now no longer needs to take a 'self' parameter, so drop it. Rename it to just _check(), to help us out with the line-length -- and now that it's contained within _pragma, it is contextually easier to see how it's used anyway -- especially with types. Signed-off-by: John Snow --- I left (name, value) as args to avoid creating a fully magic "macro", though, I thought this was too weird: info.pragma.foobar = _check() and it looked more reasonable as: info.pragma.foobar = _check(name, value) Signed-off-by: John Snow --- scripts/qapi/parser.py | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 16fd36f8391..d02a134aae9 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -17,6 +17,7 @@ from collections import OrderedDict import os import re +from typing import List from .common import match_nofail from .error import QAPISemError, QAPISourceError @@ -151,28 +152,27 @@ def _include(include, info, incl_fname, previously_included): ) from err @staticmethod - def _check_pragma_list_of_str(name, value, info): - if (not isinstance(value, list) - or any([not isinstance(elt, str) for elt in value])): - raise QAPISemError( - info, - "pragma %s must be a list of strings" % name) + def _pragma(name, value, info): + + def _check(name, value) -> List[str]: + if (not isinstance(value, list) or + any([not isinstance(elt, str) for elt in value])): + raise QAPISemError( + info, + "pragma %s must be a list of strings" % name) + return value - def _pragma(self, name, value, info): if name == 'doc-required': if not isinstance(value, bool): raise QAPISemError(info, "pragma 'doc-required' must be boolean") info.pragma.doc_required = value elif name == 'command-name-exceptions': - self._check_pragma_list_of_str(name, value, info) - info.pragma.command_name_exceptions = value + info.pragma.command_name_exceptions = _check(name, value) elif name == 'command-returns-exceptions': - self._check_pragma_list_of_str(name, value, info) - info.pragma.command_returns_exceptions = value + info.pragma.command_returns_exceptions = _check(name, value) elif name == 'member-name-exceptions': - self._check_pragma_list_of_str(name, value, info) - info.pragma.member_name_exceptions = value + info.pragma.member_name_exceptions = _check(name, value) else: raise QAPISemError(info, "unknown pragma '%s'" % name) From patchwork Thu Apr 22 03:07:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44F6FC433B4 for ; Thu, 22 Apr 2021 03:17:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8782661437 for ; Thu, 22 Apr 2021 03:17:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8782661437 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37298 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPqm-0001CT-LZ for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:17:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49192) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgz-0006WG-GV for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:49401) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgn-0006Te-Uj for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uE8n0dZPpXG5itRbKUVBTmuelPWVnwlpVJU/qF6x6zA=; b=HRxxPd5asDJUf+oIzZauEH6+RiMMWpyM6X9lTRUk7xrrubBRu5xZ0ABHR90xJnP1whQMpQ 88rfVasRMWM0BsYJaOLIux16ACyXb/EJyXdlpsofKc7jhamMYrWBUZ/dYfSXVT5JGoamDX VUu3AJjDL+fHud6MeiAEl7GPsjnt5IM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-519-_BIpVNkEMr6V2VGNBSGs7g-1; Wed, 21 Apr 2021 23:07:33 -0400 X-MC-Unique: _BIpVNkEMr6V2VGNBSGs7g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0B2968030BB; Thu, 22 Apr 2021 03:07:32 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4C6E62915B; Thu, 22 Apr 2021 03:07:31 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 12/22] qapi/parser: add type hint annotations Date: Wed, 21 Apr 2021 23:07:10 -0400 Message-Id: <20210422030720.3685766-13-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Annotations do not change runtime behavior. This commit *only* adds annotations. (Annotations for QAPIDoc are in a later commit.) Signed-off-by: John Snow --- scripts/qapi/parser.py | 61 ++++++++++++++++++++++++++++-------------- 1 file changed, 41 insertions(+), 20 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index d02a134aae9..f2b57d5642a 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -17,16 +17,29 @@ from collections import OrderedDict import os import re -from typing import List +from typing import ( + Dict, + List, + Optional, + Set, + Union, +) from .common import match_nofail from .error import QAPISemError, QAPISourceError from .source import QAPISourceInfo +#: Represents a parsed JSON object; semantically: one QAPI schema expression. +Expression = Dict[str, object] + +# Return value alias for get_expr(). +_ExprValue = Union[List[object], Dict[str, object], str, bool] + + class QAPIParseError(QAPISourceError): """Error class for all QAPI schema parsing errors.""" - def __init__(self, parser, msg): + def __init__(self, parser: 'QAPISchemaParser', msg: str): col = 1 for ch in parser.src[parser.line_pos:parser.pos]: if ch == '\t': @@ -38,7 +51,10 @@ def __init__(self, parser, msg): class QAPISchemaParser: - def __init__(self, fname, previously_included=None, incl_info=None): + def __init__(self, + fname: str, + previously_included: Optional[Set[str]] = None, + incl_info: Optional[QAPISourceInfo] = None): self._fname = fname self._included = previously_included or set() self._included.add(os.path.abspath(self._fname)) @@ -46,20 +62,20 @@ def __init__(self, fname, previously_included=None, incl_info=None): # Lexer state (see `accept` for details): self.info = QAPISourceInfo(self._fname, incl_info) - self.tok = None + self.tok: Optional[str] = None self.pos = 0 self.cursor = 0 - self.val = None + self.val: Optional[Union[bool, str]] = None self.line_pos = 0 # Parser output: - self.exprs = [] - self.docs = [] + self.exprs: List[Expression] = [] + self.docs: List[QAPIDoc] = [] # Showtime! self._parse() - def _parse(self): + def _parse(self) -> None: cur_doc = None with open(self._fname, 'r', encoding='utf-8') as fp: @@ -122,7 +138,7 @@ def _parse(self): self.reject_expr_doc(cur_doc) @staticmethod - def reject_expr_doc(doc): + def reject_expr_doc(doc: Optional['QAPIDoc']) -> None: if doc and doc.symbol: raise QAPISemError( doc.info, @@ -130,10 +146,14 @@ def reject_expr_doc(doc): % doc.symbol) @staticmethod - def _include(include, info, incl_fname, previously_included): + def _include(include: str, + info: QAPISourceInfo, + incl_fname: str, + previously_included: Set[str] + ) -> Optional['QAPISchemaParser']: incl_abs_fname = os.path.abspath(incl_fname) # catch inclusion cycle - inf = info + inf: Optional[QAPISourceInfo] = info while inf: if incl_abs_fname == os.path.abspath(inf.fname): raise QAPISemError(info, "inclusion loop for %s" % include) @@ -152,9 +172,9 @@ def _include(include, info, incl_fname, previously_included): ) from err @staticmethod - def _pragma(name, value, info): + def _pragma(name: str, value: object, info: QAPISourceInfo) -> None: - def _check(name, value) -> List[str]: + def _check(name: str, value: object) -> List[str]: if (not isinstance(value, list) or any([not isinstance(elt, str) for elt in value])): raise QAPISemError( @@ -176,7 +196,7 @@ def _check(name, value) -> List[str]: else: raise QAPISemError(info, "unknown pragma '%s'" % name) - def accept(self, skip_comment=True): + def accept(self, skip_comment: bool = True) -> None: while True: self.tok = self.src[self.cursor] self.pos = self.cursor @@ -240,8 +260,8 @@ def accept(self, skip_comment=True): self.src[self.cursor-1:]) raise QAPIParseError(self, "stray '%s'" % match.group(0)) - def get_members(self): - expr = OrderedDict() + def get_members(self) -> 'OrderedDict[str, object]': + expr: 'OrderedDict[str, object]' = OrderedDict() if self.tok == '}': self.accept() return expr @@ -267,8 +287,8 @@ def get_members(self): if self.tok != "'": raise QAPIParseError(self, "expected string") - def get_values(self): - expr = [] + def get_values(self) -> List[object]: + expr: List[object] = [] if self.tok == ']': self.accept() return expr @@ -284,8 +304,9 @@ def get_values(self): raise QAPIParseError(self, "expected ',' or ']'") self.accept() - def get_expr(self, nested): + def get_expr(self, nested: bool = False) -> _ExprValue: # TODO: Teach mypy that nested=False means the retval is a Dict. + expr: _ExprValue if self.tok != '{' and not nested: raise QAPIParseError(self, "expected '{'") if self.tok == '{': @@ -303,7 +324,7 @@ def get_expr(self, nested): self, "expected '{', '[', string, or boolean") return expr - def get_doc(self, info): + def get_doc(self, info: QAPISourceInfo) -> List['QAPIDoc']: if self.val != '##': raise QAPIParseError( self, "junk after '##' at start of documentation comment") From patchwork Thu Apr 22 03:07:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9696FC433ED for ; Thu, 22 Apr 2021 03:23:24 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1418160FE3 for ; Thu, 22 Apr 2021 03:23:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1418160FE3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47818 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPw3-0005gw-5B for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:23:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49262) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh2-0006dG-4X for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:50661) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgr-0006VC-3d for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B4jp8cAAn1x1QPx55eEH2P8Gg4s33gFRJr6p0MqE0aw=; b=UswRAuH3XOq5nOqRyDX+EBe0QdiT4gIKLaVwybvSiQbTzLOey5GYQUZdx+TyDkTsC2ePB9 B/etzfjjPCBiRUanBmnQdQAKoRfk0ppmIidjV0Smv7B8QaNKVgELD8anThdqs4/wyG/PSg cO9yD+p2EDFql7t+f3saaHjH9qBhlPM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-392-6mmcbhxOMi28tE5k_tukAg-1; Wed, 21 Apr 2021 23:07:33 -0400 X-MC-Unique: 6mmcbhxOMi28tE5k_tukAg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0AF4F107ACC7; Thu, 22 Apr 2021 03:07:33 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 334EC607CB; Thu, 22 Apr 2021 03:07:32 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 13/22] qapi/parser: [RFC] overload the return type of get_expr Date: Wed, 21 Apr 2021 23:07:11 -0400 Message-Id: <20210422030720.3685766-14-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Teach mypy that there are two possible return types here: either an Expression, or ... something else. Not a SLOC reduction, but it does remove an assertion. It also isn't much safer than a cast, mypy has no insight into if overloads are true or not. It's on the honor system. I thought I'd demonstrate its use, though. Signed-off-by: John Snow --- scripts/qapi/parser.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index f2b57d5642a..cbdddc344e7 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -23,6 +23,7 @@ Optional, Set, Union, + overload, ) from .common import match_nofail @@ -95,8 +96,7 @@ def _parse(self) -> None: self.docs.append(cur_doc) continue - expr = self.get_expr(False) - assert isinstance(expr, dict) # Guaranteed when nested=False + expr = self.get_expr() if 'include' in expr: self.reject_expr_doc(cur_doc) @@ -304,8 +304,15 @@ def get_values(self) -> List[object]: raise QAPIParseError(self, "expected ',' or ']'") self.accept() + @overload + # No nesting, must be an Expression. + def get_expr(self) -> Expression: ... + + @overload + # Possibly nested, might be anything. + def get_expr(self, nested: bool) -> _ExprValue: ... + def get_expr(self, nested: bool = False) -> _ExprValue: - # TODO: Teach mypy that nested=False means the retval is a Dict. expr: _ExprValue if self.tok != '{' and not nested: raise QAPIParseError(self, "expected '{'") From patchwork Thu Apr 22 03:07:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51904C433B4 for ; Thu, 22 Apr 2021 03:25:35 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B08B060FE3 for ; Thu, 22 Apr 2021 03:25:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B08B060FE3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52044 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPy9-0007RG-NH for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:25:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49196) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgz-0006Wc-Ks for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:60590) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgp-0006V4-6N for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060856; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EzUzxtDSlbCWdaLikKE4YDQ+smTaOhe5KxLbLEepuMI=; b=UUOwiydT6So1INrXVNjEsgJT5X8xezF30YqkxvBkf9uvSc3Raxk6SQUXoI0bB/FmT2bix9 h38kEoY+v5K6oHpxgf0CW8IB3TCHaTe5fnq0xZFj3tS/HUfVQTZoTUevFitcCPPkaQ0XQq Rv4KKw7ZhdX0fqyRUkxLUeMldGYYE6Q= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-91-9SvBiCegP2SDJIQliFUtRA-1; Wed, 21 Apr 2021 23:07:34 -0400 X-MC-Unique: 9SvBiCegP2SDJIQliFUtRA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E5B1F8189C6; Thu, 22 Apr 2021 03:07:33 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 324CF607CB; Thu, 22 Apr 2021 03:07:33 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 14/22] qapi/parser: Remove superfluous list constructor Date: Wed, 21 Apr 2021 23:07:12 -0400 Message-Id: <20210422030720.3685766-15-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" A generator suffices (and quiets a pylint warning.) Signed-off-by: John Snow --- scripts/qapi/parser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index cbdddc344e7..dbbd0fcbc2f 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -176,7 +176,7 @@ def _pragma(name: str, value: object, info: QAPISourceInfo) -> None: def _check(name: str, value: object) -> List[str]: if (not isinstance(value, list) or - any([not isinstance(elt, str) for elt in value])): + any(not isinstance(elt, str) for elt in value)): raise QAPISemError( info, "pragma %s must be a list of strings" % name) From patchwork Thu Apr 22 03:07:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAB1FC433B4 for ; Thu, 22 Apr 2021 03:27:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3BB2661445 for ; Thu, 22 Apr 2021 03:27:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3BB2661445 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:56172 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZQ07-0000ny-9O for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:27:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49240) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh1-0006bH-CH for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:51 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:43582) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgq-0006V9-IL for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060857; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4LJWk89waSYgNXJrIb8CYDjLyDrOKSQYel7tcT4x+Z4=; b=Ydn6wTjxLWigtdb6trsUhrOaLH1rfAkiH3IM6zwJz3yp4yNhe4rtCKZE/qzCEVbBWnjH5R ErNo5uvZ3doQiCPj3/BP9EbD5K+S7Y82oUCWHFKdGOgdyy3vKvKGfLCM8RYnR2VzSryuGu z16UOxom6kKTSWTcNM6UGG57UZv3Dr8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-452-cNhFq4N_PgyQ-sTYrItykw-1; Wed, 21 Apr 2021 23:07:35 -0400 X-MC-Unique: cNhFq4N_PgyQ-sTYrItykw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CD6B81006C81; Thu, 22 Apr 2021 03:07:34 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1CBC9607CB; Thu, 22 Apr 2021 03:07:34 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 15/22] qapi/parser: allow 'ch' variable name Date: Wed, 21 Apr 2021 23:07:13 -0400 Message-Id: <20210422030720.3685766-16-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" We can have a two-letter variable name, as a treat. Signed-off-by: John Snow --- scripts/qapi/pylintrc | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc index 88efbf71cb2..c5275d5f59b 100644 --- a/scripts/qapi/pylintrc +++ b/scripts/qapi/pylintrc @@ -43,6 +43,7 @@ good-names=i, _, fp, # fp = open(...) fd, # fd = os.open(...) + ch, [VARIABLES] From patchwork Thu Apr 22 03:07:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9617CC433ED for ; Thu, 22 Apr 2021 03:12:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 52B966142F for ; Thu, 22 Apr 2021 03:12:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 52B966142F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:50904 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPl4-0003Wm-DR for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:12:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49230) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh1-0006aZ-2o for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:51 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:34346) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgr-0006VJ-2Y for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060858; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rt7XxriI9T4iNSOpP5SwupV0N8/US6g4GlbT3Kf+rA=; b=W9VjtHOWZxkt8q4SC/Xx2ZAAZmqav5mzknslLTNY9JmVa3ewYImLgmPRvuhGob35/7k4Gw aMU0GDcoFQXUpHpIg00PoTQv1nljFnGo4ul3aHfZGrlprLvQOZCMpWGhmjDsmWXjtacajn oLWhh24CdDPaGBukIdp6YZxUyvrvl+8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-527-tjK8pptTOXKDne445NoxRg-1; Wed, 21 Apr 2021 23:07:36 -0400 X-MC-Unique: tjK8pptTOXKDne445NoxRg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B4D0A107ACCA; Thu, 22 Apr 2021 03:07:35 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3E852915B; Thu, 22 Apr 2021 03:07:34 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 16/22] qapi/parser: add docstrings Date: Wed, 21 Apr 2021 23:07:14 -0400 Message-Id: <20210422030720.3685766-17-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John Snow --- My hubris is infinite. OK, I only added a few -- to help me remember how the parser works at a glance. Signed-off-by: John Snow --- scripts/qapi/parser.py | 66 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index dbbd0fcbc2f..8fc77808ace 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -51,7 +51,24 @@ def __init__(self, parser: 'QAPISchemaParser', msg: str): class QAPISchemaParser: + """ + Performs parsing of a QAPI schema source file. + :param fname: Path to the source file. + :param previously_included: + The absolute paths of previously included source files. + Only used by recursive calls to avoid re-parsing files. + :param incl_info: + `QAPISourceInfo` for the parent document. + This may be None if this is the root schema document. + + :ivar exprs: Resulting parsed expressions. + :ivar docs: Resulting parsed documentation blocks. + + :raise OSError: For problems opening the root schema document. + :raise QAPIParseError: For JSON or QAPIDoc syntax problems. + :raise QAPISemError: For various semantic issues with the schema. + """ def __init__(self, fname: str, previously_included: Optional[Set[str]] = None, @@ -77,6 +94,11 @@ def __init__(self, self._parse() def _parse(self) -> None: + """ + Parse the QAPI schema document. + + :return: None; results are stored in ``exprs`` and ``docs``. + """ cur_doc = None with open(self._fname, 'r', encoding='utf-8') as fp: @@ -197,6 +219,50 @@ def _check(name: str, value: object) -> List[str]: raise QAPISemError(info, "unknown pragma '%s'" % name) def accept(self, skip_comment: bool = True) -> None: + """ + Read the next lexeme and process it into a token. + + :Object state: + :tok: represents the token type. See below for values. + :pos: is the position of the first character in the lexeme. + :cursor: is the position of the next character. + :val: is the variable value of the token, if any. + + Single-character tokens: + + These include ``LBRACE``, ``RBRACE``, ``COLON``, ``COMMA``, + ``LSQB``, and ``RSQB``. ``tok`` holds the single character + lexeme. ``val`` is ``None``. + + Multi-character tokens: + + - ``COMMENT``: + + - This token is not normally yielded by the lexer, but it + can be when ``skip_comment`` is False. + - ``tok`` is the value ``"#"``. + - ``val`` is a string including all chars until end-of-line. + + - ``STRING``: + + - ``tok`` is the ``"'"``, the single quote. + - ``value`` is the string, *excluding* the quotes. + + - ``TRUE`` and ``FALSE``: + + - ``tok`` is either ``"t"`` or ``"f"`` accordingly. + - ``val`` is either ``True`` or ``False`` accordingly. + + - ``NEWLINE`` and ``SPACE``: + + - These are consumed by the lexer directly. ``line_pos`` and + ``info`` are advanced when ``NEWLINE`` is encountered. + ``tok`` is set to ``None`` upon reaching EOF. + + :param skip_comment: + When false, return ``COMMENT`` tokens. + This is used when reading documentation blocks. + """ while True: self.tok = self.src[self.cursor] self.pos = self.cursor From patchwork Thu Apr 22 03:07:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,SUBJ_ALL_CAPS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B191C433ED for ; Thu, 22 Apr 2021 03:20:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EA6C861437 for ; Thu, 22 Apr 2021 03:20:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EA6C861437 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:43484 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPtZ-0003ls-14 for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:20:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49224) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh0-0006Zv-RI for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:28849) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgr-0006VO-2s for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZEo10Eg0ZqkzmOrOO561Y1EeBbIUSFERGFxAWWmJjtY=; b=eVDOxELZ4qfSN7A0pyjkpwaMZBNJFcVFD6NCVufSprePaHzPgkjZceqtmAPff+pDOngDAj 7H0BW2vcr2kk8pWQE/fqqpHHCLamqn+07lk7kDsoP7dvc1QFcnZV0YF43fCDKbSUNDbD/1 IpHxU+oNgglgtdiqp8vDLVBa2p1FL8c= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-602-jaUGUhBSNkuUQu56kpgbJw-1; Wed, 21 Apr 2021 23:07:37 -0400 X-MC-Unique: jaUGUhBSNkuUQu56kpgbJw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id AC6288189C7; Thu, 22 Apr 2021 03:07:36 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id DCB88607CB; Thu, 22 Apr 2021 03:07:35 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 17/22] CHECKPOINT Date: Wed, 21 Apr 2021 23:07:15 -0400 Message-Id: <20210422030720.3685766-18-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, SUBJ_ALL_CAPS=0.5 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" As of here, parser is actually fully typed, and QAPIDoc is not. Below, there are a few extra patches that "prove" this, but they are not necessarily meant for inclusion. They could theoretically be included anyway, but a few of them would need to be squashed together to ensure our "no intermediate breakages" rule, and a few things would need to be re-ordered. Consider them [RFC], but optional and completely safe to drop. Signed-off-by: John Snow --- scripts/qapi/mypy.ini | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini index 54ca4483d6d..d7bbb2dc9c7 100644 --- a/scripts/qapi/mypy.ini +++ b/scripts/qapi/mypy.ini @@ -4,6 +4,7 @@ disallow_untyped_calls = False python_version = 3.6 [mypy-qapi.parser] +# QAPISchemaParser is done, I promise! disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False From patchwork Thu Apr 22 03:07:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54B22C433B4 for ; Thu, 22 Apr 2021 03:25:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 841286144A for ; Thu, 22 Apr 2021 03:25:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 841286144A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:52214 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPyE-0007VR-Ip for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:25:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49386) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPhF-0007AP-1M for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:08:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:32200) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPhA-0006h9-IW for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:08:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ij/r9W8EWngxhlvqlpPP25xFsRTlle1Fhhs1bNLys74=; b=S6mvbagMzEnrpbSagKFl7j+rBP/bGJSecty/8zS6XQOubwNJYvhW3BNpiU+016PpkaSHW/ 02azIQyxoN8nOTHuOJC2JiBpbrUwFWXDqTf8BsKnEn206dk6p5idhpPyrJeusHfFB0prRm q+dEsBMCGyvN7f9hMh2GjEVUi7SqQYg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-26-5RuK6MuzPym4Lry1j-FfBA-1; Wed, 21 Apr 2021 23:07:38 -0400 X-MC-Unique: 5RuK6MuzPym4Lry1j-FfBA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B2D9F8030A1; Thu, 22 Apr 2021 03:07:37 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id D33382915B; Thu, 22 Apr 2021 03:07:36 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 18/22] qapi: [WIP] Rip QAPIDoc out of parser.py Date: Wed, 21 Apr 2021 23:07:16 -0400 Message-Id: <20210422030720.3685766-19-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This (rather unglamorously) rips QAPIDoc out of parser.py. It does not leave a working solution in its place, opting instead just for code movement. Signed-off-by: John Snow --- scripts/qapi/parser.py | 342 ------------------------------------- scripts/qapi/qapidoc.py | 362 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 362 insertions(+), 342 deletions(-) create mode 100644 scripts/qapi/qapidoc.py diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 8fc77808ace..6fed742124d 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -16,7 +16,6 @@ from collections import OrderedDict import os -import re from typing import ( Dict, List, @@ -430,344 +429,3 @@ def get_doc(self, info: QAPISourceInfo) -> List['QAPIDoc']: self.accept(False) raise QAPIParseError(self, "documentation comment must end with '##'") - - -class QAPIDoc: - """ - A documentation comment block, either definition or free-form - - Definition documentation blocks consist of - - * a body section: one line naming the definition, followed by an - overview (any number of lines) - - * argument sections: a description of each argument (for commands - and events) or member (for structs, unions and alternates) - - * features sections: a description of each feature flag - - * additional (non-argument) sections, possibly tagged - - Free-form documentation blocks consist only of a body section. - """ - - class Section: - def __init__(self, parser, name=None, indent=0): - # parser, for error messages about indentation - self._parser = parser - # optional section name (argument/member or section name) - self.name = name - self.text = '' - # the expected indent level of the text of this section - self._indent = indent - - def append(self, line): - # Strip leading spaces corresponding to the expected indent level - # Blank lines are always OK. - if line: - indent = match_nofail(r'\s*', line).end() - if indent < self._indent: - raise QAPIParseError( - self._parser, - "unexpected de-indent (expected at least %d spaces)" % - self._indent) - line = line[self._indent:] - - self.text += line.rstrip() + '\n' - - class ArgSection(Section): - def __init__(self, parser, name, indent=0): - super().__init__(parser, name, indent) - self.member = None - - def connect(self, member): - self.member = member - - def __init__(self, parser, info): - # self._parser is used to report errors with QAPIParseError. The - # resulting error position depends on the state of the parser. - # It happens to be the beginning of the comment. More or less - # servicable, but action at a distance. - self._parser = parser - self.info = info - self.symbol = None - self.body = QAPIDoc.Section(parser) - # dict mapping parameter name to ArgSection - self.args = OrderedDict() - self.features = OrderedDict() - # a list of Section - self.sections = [] - # the current section - self._section = self.body - self._append_line = self._append_body_line - - def has_section(self, name): - """Return True if we have a section with this name.""" - for i in self.sections: - if i.name == name: - return True - return False - - def append(self, line): - """ - Parse a comment line and add it to the documentation. - - The way that the line is dealt with depends on which part of - the documentation we're parsing right now: - * The body section: ._append_line is ._append_body_line - * An argument section: ._append_line is ._append_args_line - * A features section: ._append_line is ._append_features_line - * An additional section: ._append_line is ._append_various_line - """ - line = line[1:] - if not line: - self._append_freeform(line) - return - - if line[0] != ' ': - raise QAPIParseError(self._parser, "missing space after #") - line = line[1:] - self._append_line(line) - - def end_comment(self): - self._end_section() - - @staticmethod - def _is_section_tag(name): - return name in ('Returns:', 'Since:', - # those are often singular or plural - 'Note:', 'Notes:', - 'Example:', 'Examples:', - 'TODO:') - - def _append_body_line(self, line): - """ - Process a line of documentation text in the body section. - - If this a symbol line and it is the section's first line, this - is a definition documentation block for that symbol. - - If it's a definition documentation block, another symbol line - begins the argument section for the argument named by it, and - a section tag begins an additional section. Start that - section and append the line to it. - - Else, append the line to the current section. - """ - name = line.split(' ', 1)[0] - # FIXME not nice: things like '# @foo:' and '# @foo: ' aren't - # recognized, and get silently treated as ordinary text - if not self.symbol and not self.body.text and line.startswith('@'): - if not line.endswith(':'): - raise QAPIParseError(self._parser, "line should end with ':'") - self.symbol = line[1:-1] - # FIXME invalid names other than the empty string aren't flagged - if not self.symbol: - raise QAPIParseError(self._parser, "invalid name") - elif self.symbol: - # This is a definition documentation block - if name.startswith('@') and name.endswith(':'): - self._append_line = self._append_args_line - self._append_args_line(line) - elif line == 'Features:': - self._append_line = self._append_features_line - elif self._is_section_tag(name): - self._append_line = self._append_various_line - self._append_various_line(line) - else: - self._append_freeform(line) - else: - # This is a free-form documentation block - self._append_freeform(line) - - def _append_args_line(self, line): - """ - Process a line of documentation text in an argument section. - - A symbol line begins the next argument section, a section tag - section or a non-indented line after a blank line begins an - additional section. Start that section and append the line to - it. - - Else, append the line to the current section. - - """ - name = line.split(' ', 1)[0] - - if name.startswith('@') and name.endswith(':'): - # If line is "@arg: first line of description", find - # the index of 'f', which is the indent we expect for any - # following lines. We then remove the leading "@arg:" - # from line and replace it with spaces so that 'f' has the - # same index as it did in the original line and can be - # handled the same way we will handle following lines. - indent = match_nofail(r'@\S*:\s*', line).end() - line = line[indent:] - if not line: - # Line was just the "@arg:" header; following lines - # are not indented - indent = 0 - else: - line = ' ' * indent + line - self._start_args_section(name[1:-1], indent) - elif self._is_section_tag(name): - self._append_line = self._append_various_line - self._append_various_line(line) - return - elif (self._section.text.endswith('\n\n') - and line and not line[0].isspace()): - if line == 'Features:': - self._append_line = self._append_features_line - else: - self._start_section() - self._append_line = self._append_various_line - self._append_various_line(line) - return - - self._append_freeform(line) - - def _append_features_line(self, line): - name = line.split(' ', 1)[0] - - if name.startswith('@') and name.endswith(':'): - # If line is "@arg: first line of description", find - # the index of 'f', which is the indent we expect for any - # following lines. We then remove the leading "@arg:" - # from line and replace it with spaces so that 'f' has the - # same index as it did in the original line and can be - # handled the same way we will handle following lines. - indent = match_nofail(r'@\S*:\s*', line).end() - line = line[indent:] - if not line: - # Line was just the "@arg:" header; following lines - # are not indented - indent = 0 - else: - line = ' ' * indent + line - self._start_features_section(name[1:-1], indent) - elif self._is_section_tag(name): - self._append_line = self._append_various_line - self._append_various_line(line) - return - elif (self._section.text.endswith('\n\n') - and line and not line[0].isspace()): - self._start_section() - self._append_line = self._append_various_line - self._append_various_line(line) - return - - self._append_freeform(line) - - def _append_various_line(self, line): - """ - Process a line of documentation text in an additional section. - - A symbol line is an error. - - A section tag begins an additional section. Start that - section and append the line to it. - - Else, append the line to the current section. - """ - name = line.split(' ', 1)[0] - - if name.startswith('@') and name.endswith(':'): - raise QAPIParseError(self._parser, - "'%s' can't follow '%s' section" - % (name, self.sections[0].name)) - if self._is_section_tag(name): - # If line is "Section: first line of description", find - # the index of 'f', which is the indent we expect for any - # following lines. We then remove the leading "Section:" - # from line and replace it with spaces so that 'f' has the - # same index as it did in the original line and can be - # handled the same way we will handle following lines. - indent = match_nofail(r'\S*:\s*', line).end() - line = line[indent:] - if not line: - # Line was just the "Section:" header; following lines - # are not indented - indent = 0 - else: - line = ' ' * indent + line - self._start_section(name[:-1], indent) - - self._append_freeform(line) - - def _start_symbol_section(self, symbols_dict, name, indent): - # FIXME invalid names other than the empty string aren't flagged - if not name: - raise QAPIParseError(self._parser, "invalid parameter name") - if name in symbols_dict: - raise QAPIParseError(self._parser, - "'%s' parameter name duplicated" % name) - assert not self.sections - self._end_section() - self._section = QAPIDoc.ArgSection(self._parser, name, indent) - symbols_dict[name] = self._section - - def _start_args_section(self, name, indent): - self._start_symbol_section(self.args, name, indent) - - def _start_features_section(self, name, indent): - self._start_symbol_section(self.features, name, indent) - - def _start_section(self, name=None, indent=0): - if name in ('Returns', 'Since') and self.has_section(name): - raise QAPIParseError(self._parser, - "duplicated '%s' section" % name) - self._end_section() - self._section = QAPIDoc.Section(self._parser, name, indent) - self.sections.append(self._section) - - def _end_section(self): - if self._section: - text = self._section.text = self._section.text.strip() - if self._section.name and (not text or text.isspace()): - raise QAPIParseError( - self._parser, - "empty doc section '%s'" % self._section.name) - self._section = None - - def _append_freeform(self, line): - match = re.match(r'(@\S+:)', line) - if match: - raise QAPIParseError(self._parser, - "'%s' not allowed in free-form documentation" - % match.group(1)) - self._section.append(line) - - def connect_member(self, member): - if member.name not in self.args: - # Undocumented TODO outlaw - self.args[member.name] = QAPIDoc.ArgSection(self._parser, - member.name) - self.args[member.name].connect(member) - - def connect_feature(self, feature): - if feature.name not in self.features: - raise QAPISemError(feature.info, - "feature '%s' lacks documentation" - % feature.name) - self.features[feature.name].connect(feature) - - def check_expr(self, expr): - if self.has_section('Returns') and 'command' not in expr: - raise QAPISemError(self.info, - "'Returns:' is only valid for commands") - - def check(self): - - def check_args_section(args, info, what): - bogus = [name for name, section in args.items() - if not section.member] - if bogus: - raise QAPISemError( - self.info, - "documented member%s '%s' %s not exist" - % ("s" if len(bogus) > 1 else "", - "', '".join(bogus), - "do" if len(bogus) > 1 else "does")) - - check_args_section(self.args, self.info, 'members') - check_args_section(self.features, self.info, 'features') diff --git a/scripts/qapi/qapidoc.py b/scripts/qapi/qapidoc.py new file mode 100644 index 00000000000..eb24ea12a06 --- /dev/null +++ b/scripts/qapi/qapidoc.py @@ -0,0 +1,362 @@ +# -*- coding: utf-8 -*- +# +# QAPI schema (doc) parser +# +# Copyright IBM, Corp. 2011 +# Copyright (c) 2013-2019 Red Hat Inc. +# +# Authors: +# Anthony Liguori +# Markus Armbruster +# Marc-André Lureau +# Kevin Wolf +# +# This work is licensed under the terms of the GNU GPL, version 2. +# See the COPYING file in the top-level directory. + +from collections import OrderedDict +import re + +from .common import match_nofail +from .error import QAPISemError + + +class QAPIDoc: + """ + A documentation comment block, either definition or free-form + + Definition documentation blocks consist of + + * a body section: one line naming the definition, followed by an + overview (any number of lines) + + * argument sections: a description of each argument (for commands + and events) or member (for structs, unions and alternates) + + * features sections: a description of each feature flag + + * additional (non-argument) sections, possibly tagged + + Free-form documentation blocks consist only of a body section. + """ + + class Section: + def __init__(self, parser, name=None, indent=0): + # parser, for error messages about indentation + self._parser = parser + # optional section name (argument/member or section name) + self.name = name + self.text = '' + # the expected indent level of the text of this section + self._indent = indent + + def append(self, line): + # Strip leading spaces corresponding to the expected indent level + # Blank lines are always OK. + if line: + indent = match_nofail(r'\s*', line).end() + if indent < self._indent: + raise QAPIParseError( + self._parser, + "unexpected de-indent (expected at least %d spaces)" % + self._indent) + line = line[self._indent:] + + self.text += line.rstrip() + '\n' + + class ArgSection(Section): + def __init__(self, parser, name, indent=0): + super().__init__(parser, name, indent) + self.member = None + + def connect(self, member): + self.member = member + + def __init__(self, parser, info): + # self._parser is used to report errors with QAPIParseError. The + # resulting error position depends on the state of the parser. + # It happens to be the beginning of the comment. More or less + # servicable, but action at a distance. + self._parser = parser + self.info = info + self.symbol = None + self.body = QAPIDoc.Section(parser) + # dict mapping parameter name to ArgSection + self.args = OrderedDict() + self.features = OrderedDict() + # a list of Section + self.sections = [] + # the current section + self._section = self.body + self._append_line = self._append_body_line + + def has_section(self, name): + """Return True if we have a section with this name.""" + for i in self.sections: + if i.name == name: + return True + return False + + def append(self, line): + """ + Parse a comment line and add it to the documentation. + + The way that the line is dealt with depends on which part of + the documentation we're parsing right now: + * The body section: ._append_line is ._append_body_line + * An argument section: ._append_line is ._append_args_line + * A features section: ._append_line is ._append_features_line + * An additional section: ._append_line is ._append_various_line + """ + line = line[1:] + if not line: + self._append_freeform(line) + return + + if line[0] != ' ': + raise QAPIParseError(self._parser, "missing space after #") + line = line[1:] + self._append_line(line) + + def end_comment(self): + self._end_section() + + @staticmethod + def _is_section_tag(name): + return name in ('Returns:', 'Since:', + # those are often singular or plural + 'Note:', 'Notes:', + 'Example:', 'Examples:', + 'TODO:') + + def _append_body_line(self, line): + """ + Process a line of documentation text in the body section. + + If this a symbol line and it is the section's first line, this + is a definition documentation block for that symbol. + + If it's a definition documentation block, another symbol line + begins the argument section for the argument named by it, and + a section tag begins an additional section. Start that + section and append the line to it. + + Else, append the line to the current section. + """ + name = line.split(' ', 1)[0] + # FIXME not nice: things like '# @foo:' and '# @foo: ' aren't + # recognized, and get silently treated as ordinary text + if not self.symbol and not self.body.text and line.startswith('@'): + if not line.endswith(':'): + raise QAPIParseError(self._parser, "line should end with ':'") + self.symbol = line[1:-1] + # FIXME invalid names other than the empty string aren't flagged + if not self.symbol: + raise QAPIParseError(self._parser, "invalid name") + elif self.symbol: + # This is a definition documentation block + if name.startswith('@') and name.endswith(':'): + self._append_line = self._append_args_line + self._append_args_line(line) + elif line == 'Features:': + self._append_line = self._append_features_line + elif self._is_section_tag(name): + self._append_line = self._append_various_line + self._append_various_line(line) + else: + self._append_freeform(line) + else: + # This is a free-form documentation block + self._append_freeform(line) + + def _append_args_line(self, line): + """ + Process a line of documentation text in an argument section. + + A symbol line begins the next argument section, a section tag + section or a non-indented line after a blank line begins an + additional section. Start that section and append the line to + it. + + Else, append the line to the current section. + + """ + name = line.split(' ', 1)[0] + + if name.startswith('@') and name.endswith(':'): + # If line is "@arg: first line of description", find + # the index of 'f', which is the indent we expect for any + # following lines. We then remove the leading "@arg:" + # from line and replace it with spaces so that 'f' has the + # same index as it did in the original line and can be + # handled the same way we will handle following lines. + indent = match_nofail(r'@\S*:\s*', line).end() + line = line[indent:] + if not line: + # Line was just the "@arg:" header; following lines + # are not indented + indent = 0 + else: + line = ' ' * indent + line + self._start_args_section(name[1:-1], indent) + elif self._is_section_tag(name): + self._append_line = self._append_various_line + self._append_various_line(line) + return + elif (self._section.text.endswith('\n\n') + and line and not line[0].isspace()): + if line == 'Features:': + self._append_line = self._append_features_line + else: + self._start_section() + self._append_line = self._append_various_line + self._append_various_line(line) + return + + self._append_freeform(line) + + def _append_features_line(self, line): + name = line.split(' ', 1)[0] + + if name.startswith('@') and name.endswith(':'): + # If line is "@arg: first line of description", find + # the index of 'f', which is the indent we expect for any + # following lines. We then remove the leading "@arg:" + # from line and replace it with spaces so that 'f' has the + # same index as it did in the original line and can be + # handled the same way we will handle following lines. + indent = match_nofail(r'@\S*:\s*', line).end() + line = line[indent:] + if not line: + # Line was just the "@arg:" header; following lines + # are not indented + indent = 0 + else: + line = ' ' * indent + line + self._start_features_section(name[1:-1], indent) + elif self._is_section_tag(name): + self._append_line = self._append_various_line + self._append_various_line(line) + return + elif (self._section.text.endswith('\n\n') + and line and not line[0].isspace()): + self._start_section() + self._append_line = self._append_various_line + self._append_various_line(line) + return + + self._append_freeform(line) + + def _append_various_line(self, line): + """ + Process a line of documentation text in an additional section. + + A symbol line is an error. + + A section tag begins an additional section. Start that + section and append the line to it. + + Else, append the line to the current section. + """ + name = line.split(' ', 1)[0] + + if name.startswith('@') and name.endswith(':'): + raise QAPIParseError(self._parser, + "'%s' can't follow '%s' section" + % (name, self.sections[0].name)) + if self._is_section_tag(name): + # If line is "Section: first line of description", find + # the index of 'f', which is the indent we expect for any + # following lines. We then remove the leading "Section:" + # from line and replace it with spaces so that 'f' has the + # same index as it did in the original line and can be + # handled the same way we will handle following lines. + indent = match_nofail(r'\S*:\s*', line).end() + line = line[indent:] + if not line: + # Line was just the "Section:" header; following lines + # are not indented + indent = 0 + else: + line = ' ' * indent + line + self._start_section(name[:-1], indent) + + self._append_freeform(line) + + def _start_symbol_section(self, symbols_dict, name, indent): + # FIXME invalid names other than the empty string aren't flagged + if not name: + raise QAPIParseError(self._parser, "invalid parameter name") + if name in symbols_dict: + raise QAPIParseError(self._parser, + "'%s' parameter name duplicated" % name) + assert not self.sections + self._end_section() + self._section = QAPIDoc.ArgSection(self._parser, name, indent) + symbols_dict[name] = self._section + + def _start_args_section(self, name, indent): + self._start_symbol_section(self.args, name, indent) + + def _start_features_section(self, name, indent): + self._start_symbol_section(self.features, name, indent) + + def _start_section(self, name=None, indent=0): + if name in ('Returns', 'Since') and self.has_section(name): + raise QAPIParseError(self._parser, + "duplicated '%s' section" % name) + self._end_section() + self._section = QAPIDoc.Section(self._parser, name, indent) + self.sections.append(self._section) + + def _end_section(self): + if self._section: + text = self._section.text = self._section.text.strip() + if self._section.name and (not text or text.isspace()): + raise QAPIParseError( + self._parser, + "empty doc section '%s'" % self._section.name) + self._section = None + + def _append_freeform(self, line): + match = re.match(r'(@\S+:)', line) + if match: + raise QAPIParseError(self._parser, + "'%s' not allowed in free-form documentation" + % match.group(1)) + self._section.append(line) + + def connect_member(self, member): + if member.name not in self.args: + # Undocumented TODO outlaw + self.args[member.name] = QAPIDoc.ArgSection(self._parser, + member.name) + self.args[member.name].connect(member) + + def connect_feature(self, feature): + if feature.name not in self.features: + raise QAPISemError(feature.info, + "feature '%s' lacks documentation" + % feature.name) + self.features[feature.name].connect(feature) + + def check_expr(self, expr): + if self.has_section('Returns') and 'command' not in expr: + raise QAPISemError(self.info, + "'Returns:' is only valid for commands") + + def check(self): + + def check_args_section(args, info, what): + bogus = [name for name, section in args.items() + if not section.member] + if bogus: + raise QAPISemError( + self.info, + "documented member%s '%s' %s not exist" + % ("s" if len(bogus) > 1 else "", + "', '".join(bogus), + "do" if len(bogus) > 1 else "does")) + + check_args_section(self.args, self.info, 'members') + check_args_section(self.features, self.info, 'features') From patchwork Thu Apr 22 03:07:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E4D4C433B4 for ; Thu, 22 Apr 2021 03:29:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DD635613BA for ; Thu, 22 Apr 2021 03:29:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DD635613BA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:58342 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZQ1k-0001ii-ON for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:29:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49270) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh2-0006dW-7E for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:49921) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgt-0006Vm-17 for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060862; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HtIlONdYBHxGa8pNAI0MHyifBRlo8+Wum1TSrjIcQSQ=; b=KoHd2U9hmH32iDwWU5qjQjPwsgrhEBBgCKlXYtEA9xeW/VUqm6bQf0F/cZgU6KOr5ZN6mg eu/8bvD0wUl0JxrExO+AHJ0syfPfIGU18lDzXOC3jDhpsD4H8QaP/j8UhrGxahXrDM61vs Wk9ilX3zsNmJhdCk4asN281MybGp42E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-537-_-f-08O4MmG85YsB1iMajg-1; Wed, 21 Apr 2021 23:07:39 -0400 X-MC-Unique: _-f-08O4MmG85YsB1iMajg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 967BB801814; Thu, 22 Apr 2021 03:07:38 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id D870C607CB; Thu, 22 Apr 2021 03:07:37 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 19/22] qapi: [WIP] Add type ignores for qapidoc.py Date: Wed, 21 Apr 2021 23:07:17 -0400 Message-Id: <20210422030720.3685766-20-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John Snow --- scripts/qapi/mypy.ini | 5 +++++ scripts/qapi/pylintrc | 1 + 2 files changed, 6 insertions(+) diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini index d7bbb2dc9c7..1a72be2c788 100644 --- a/scripts/qapi/mypy.ini +++ b/scripts/qapi/mypy.ini @@ -9,6 +9,11 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False +[mypy-qapi.qapidoc] +disallow_untyped_defs = False +disallow_incomplete_defs = False +check_untyped_defs = False + [mypy-qapi.schema] disallow_untyped_defs = False disallow_incomplete_defs = False diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc index c5275d5f59b..ec7605edade 100644 --- a/scripts/qapi/pylintrc +++ b/scripts/qapi/pylintrc @@ -3,6 +3,7 @@ # Add files or directories matching the regex patterns to the ignore list. # The regex matches against base names, not paths. ignore-patterns=parser.py, + qapidoc.py, schema.py, From patchwork Thu Apr 22 03:07:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B12BCC433ED for ; Thu, 22 Apr 2021 03:15:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3F6556142F for ; Thu, 22 Apr 2021 03:15:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F6556142F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59396 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPoO-00072q-70 for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:15:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh2-0006dv-Da for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:56811) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgt-0006Vp-1b for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060862; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ckLTAzfordUWJsd4OylshUHYPDpvi9uqov2U8NS2hS0=; b=LRM5yeOJEXkmToH6lkyWX8VRn12r45gv194O5cDJ3gEw81tOUd02xtIUytOwianaMUovP+ tayeUbBkm6di+K6LJHK5R4zpOFMYWSFuB81fhtD/HdawG6rlumYxqyx7mQuafSWm23wgiJ 0BaIw0Z/2NsDJY5UJ4Bo7iAxQA9cJgg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-118-e63svZz8MBOX31KIzKHtvQ-1; Wed, 21 Apr 2021 23:07:40 -0400 X-MC-Unique: e63svZz8MBOX31KIzKHtvQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7BAEA343A0; Thu, 22 Apr 2021 03:07:39 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id C0C2A607CB; Thu, 22 Apr 2021 03:07:38 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 20/22] qapi: [WIP] Import QAPIDoc from qapidoc Signed-off-by: John Snow Date: Wed, 21 Apr 2021 23:07:18 -0400 Message-Id: <20210422030720.3685766-21-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John Snow --- scripts/qapi/expr.py | 2 +- scripts/qapi/parser.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/qapi/expr.py b/scripts/qapi/expr.py index 496f7e0333e..7616646e43d 100644 --- a/scripts/qapi/expr.py +++ b/scripts/qapi/expr.py @@ -44,7 +44,7 @@ from .common import c_name from .error import QAPISemError -from .parser import QAPIDoc +from .qapidoc import QAPIDoc from .source import QAPISourceInfo diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 6fed742124d..3932f05d015 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -27,6 +27,7 @@ from .common import match_nofail from .error import QAPISemError, QAPISourceError +from .qapidoc import QAPIDoc from .source import QAPISourceInfo From patchwork Thu Apr 22 03:07:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40B0BC433ED for ; Thu, 22 Apr 2021 03:18:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C55886143B for ; Thu, 22 Apr 2021 03:18:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C55886143B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37700 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZPqv-0001Mq-On for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:18:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49342) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh4-0006j5-GN for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:54 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:55453) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgu-0006Vy-Ue for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BZwa/2fjIVd2JaFTaYCH7hZ/ihsls7RCKllzX1LYnFs=; b=BSkUCiJPYbbAiXbsziM0k5PxFtF0gN+irzCHRTsqc46AR2jwvpYKq0yzlAuwcDM2kIOooM 2Dz/5IPhEXTzl/iZTTfUYHbpYG7EYf/j3veKidtN29ZOAU1Hrb1CikudqwcHS+jvwyKR2t XNev38VL2uOtwhD1qvEKUJkM5sE2cuM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-366-J4zH4Y8gNyiGFajk9Sc4zQ-1; Wed, 21 Apr 2021 23:07:41 -0400 X-MC-Unique: J4zH4Y8gNyiGFajk9Sc4zQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5F9CD107ACC7; Thu, 22 Apr 2021 03:07:40 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id A2A22607CB; Thu, 22 Apr 2021 03:07:39 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 21/22] qapi: [WIP] Add QAPIDocError Date: Wed, 21 Apr 2021 23:07:19 -0400 Message-Id: <20210422030720.3685766-22-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Raise this error instead of QAPIParseError and delegate the context up to the parent parser. In a chat off-list, we discussed how this design forces us to continue having less accurate error context information. Still, it's useful for an extremely simple split without a lot of fuss. Signed-off-by: John Snow --- scripts/qapi/parser.py | 12 ++++++++++-- scripts/qapi/qapidoc.py | 36 +++++++++++++++++------------------- 2 files changed, 27 insertions(+), 21 deletions(-) diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py index 3932f05d015..5832bd54eb1 100644 --- a/scripts/qapi/parser.py +++ b/scripts/qapi/parser.py @@ -27,7 +27,7 @@ from .common import match_nofail from .error import QAPISemError, QAPISourceError -from .qapidoc import QAPIDoc +from .qapidoc import QAPIDoc, QAPIDocError from .source import QAPISourceInfo @@ -397,7 +397,7 @@ def get_expr(self, nested: bool = False) -> _ExprValue: self, "expected '{', '[', string, or boolean") return expr - def get_doc(self, info: QAPISourceInfo) -> List['QAPIDoc']: + def _get_doc(self, info: QAPISourceInfo) -> List['QAPIDoc']: if self.val != '##': raise QAPIParseError( self, "junk after '##' at start of documentation comment") @@ -430,3 +430,11 @@ def get_doc(self, info: QAPISourceInfo) -> List['QAPIDoc']: self.accept(False) raise QAPIParseError(self, "documentation comment must end with '##'") + + def get_doc(self, info: QAPISourceInfo) -> List['QAPIDoc']: + # Tie QAPIDocError exceptions to the current parser state, + # re-raise as QAPIParseError. + try: + return self._get_doc(info) + except QAPIDocError as err: + raise QAPIParseError(self, str(err)) from err diff --git a/scripts/qapi/qapidoc.py b/scripts/qapi/qapidoc.py index eb24ea12a06..393af93323f 100644 --- a/scripts/qapi/qapidoc.py +++ b/scripts/qapi/qapidoc.py @@ -18,7 +18,11 @@ import re from .common import match_nofail -from .error import QAPISemError +from .error import QAPIError, QAPISemError + + +class QAPIDocError(QAPIError): + """QAPIDoc parsing errors.""" class QAPIDoc: @@ -56,8 +60,7 @@ def append(self, line): if line: indent = match_nofail(r'\s*', line).end() if indent < self._indent: - raise QAPIParseError( - self._parser, + raise QAPIDocError( "unexpected de-indent (expected at least %d spaces)" % self._indent) line = line[self._indent:] @@ -114,7 +117,7 @@ def append(self, line): return if line[0] != ' ': - raise QAPIParseError(self._parser, "missing space after #") + raise QAPIDocError("missing space after #") line = line[1:] self._append_line(line) @@ -148,11 +151,11 @@ def _append_body_line(self, line): # recognized, and get silently treated as ordinary text if not self.symbol and not self.body.text and line.startswith('@'): if not line.endswith(':'): - raise QAPIParseError(self._parser, "line should end with ':'") + raise QAPIDocError("line should end with ':'") self.symbol = line[1:-1] # FIXME invalid names other than the empty string aren't flagged if not self.symbol: - raise QAPIParseError(self._parser, "invalid name") + raise QAPIDocError("invalid name") elif self.symbol: # This is a definition documentation block if name.startswith('@') and name.endswith(':'): @@ -261,9 +264,8 @@ def _append_various_line(self, line): name = line.split(' ', 1)[0] if name.startswith('@') and name.endswith(':'): - raise QAPIParseError(self._parser, - "'%s' can't follow '%s' section" - % (name, self.sections[0].name)) + raise QAPIDocError("'%s' can't follow '%s' section" + % (name, self.sections[0].name)) if self._is_section_tag(name): # If line is "Section: first line of description", find # the index of 'f', which is the indent we expect for any @@ -286,10 +288,9 @@ def _append_various_line(self, line): def _start_symbol_section(self, symbols_dict, name, indent): # FIXME invalid names other than the empty string aren't flagged if not name: - raise QAPIParseError(self._parser, "invalid parameter name") + raise QAPIDocError("invalid parameter name") if name in symbols_dict: - raise QAPIParseError(self._parser, - "'%s' parameter name duplicated" % name) + raise QAPIDocError("'%s' parameter name duplicated" % name) assert not self.sections self._end_section() self._section = QAPIDoc.ArgSection(self._parser, name, indent) @@ -303,8 +304,7 @@ def _start_features_section(self, name, indent): def _start_section(self, name=None, indent=0): if name in ('Returns', 'Since') and self.has_section(name): - raise QAPIParseError(self._parser, - "duplicated '%s' section" % name) + raise QAPIDocError("duplicated '%s' section" % name) self._end_section() self._section = QAPIDoc.Section(self._parser, name, indent) self.sections.append(self._section) @@ -313,17 +313,15 @@ def _end_section(self): if self._section: text = self._section.text = self._section.text.strip() if self._section.name and (not text or text.isspace()): - raise QAPIParseError( - self._parser, + raise QAPIDocError( "empty doc section '%s'" % self._section.name) self._section = None def _append_freeform(self, line): match = re.match(r'(@\S+:)', line) if match: - raise QAPIParseError(self._parser, - "'%s' not allowed in free-form documentation" - % match.group(1)) + raise QAPIDocError("'%s' not allowed in free-form documentation" + % match.group(1)) self._section.append(line) def connect_member(self, member): From patchwork Thu Apr 22 03:07:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12217499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDEEBC433ED for ; Thu, 22 Apr 2021 03:31:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 73796613BA for ; Thu, 22 Apr 2021 03:31:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73796613BA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:60514 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lZQ3g-0002mh-EO for qemu-devel@archiver.kernel.org; Wed, 21 Apr 2021 23:31:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49300) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPh3-0006fv-6w for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52538) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lZPgv-0006Wn-03 for qemu-devel@nongnu.org; Wed, 21 Apr 2021 23:07:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619060864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9rkUphiFWgxIY4ugyKiKF34Jvg+Dnahj0kYjwUJ7AMI=; b=Ig64/wyUg5BKsWnTAdrJO4Xm5iTXOA85IruM7iaQrFkFK04xJ+V95hOWf50jrU7FYQWBSi uxPK+hVNG3Xzt0IuYFxiSY1efxC13bRX2BC6Ro8oRjT06HzGh7SSa25AVpS8AK5mf76XmB r8eNqmGzuymZ1s7/tgIrPV6ylmFeu1w= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-182-vkrokaM1PfOXLYw40wjM-A-1; Wed, 21 Apr 2021 23:07:42 -0400 X-MC-Unique: vkrokaM1PfOXLYw40wjM-A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4B43E343A0; Thu, 22 Apr 2021 03:07:41 +0000 (UTC) Received: from scv.redhat.com (ovpn-118-152.rdu2.redhat.com [10.10.118.152]) by smtp.corp.redhat.com (Postfix) with ESMTP id 87B182915B; Thu, 22 Apr 2021 03:07:40 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PATCH 22/22] qapi: [WIP] Enable linters on parser.py Date: Wed, 21 Apr 2021 23:07:20 -0400 Message-Id: <20210422030720.3685766-23-jsnow@redhat.com> In-Reply-To: <20210422030720.3685766-1-jsnow@redhat.com> References: <20210422030720.3685766-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Michael Roth , Cleber Rosa , John Snow , Markus Armbruster , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" (Only works after we excise QAPIDoc, of course.) Signed-off-by: John Snow --- scripts/qapi/mypy.ini | 6 ------ scripts/qapi/pylintrc | 3 +-- 2 files changed, 1 insertion(+), 8 deletions(-) diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini index 1a72be2c788..56c0f306c5e 100644 --- a/scripts/qapi/mypy.ini +++ b/scripts/qapi/mypy.ini @@ -3,12 +3,6 @@ strict = True disallow_untyped_calls = False python_version = 3.6 -[mypy-qapi.parser] -# QAPISchemaParser is done, I promise! -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.qapidoc] disallow_untyped_defs = False disallow_incomplete_defs = False diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc index ec7605edade..b41a26c1ceb 100644 --- a/scripts/qapi/pylintrc +++ b/scripts/qapi/pylintrc @@ -2,8 +2,7 @@ # Add files or directories matching the regex patterns to the ignore list. # The regex matches against base names, not paths. -ignore-patterns=parser.py, - qapidoc.py, +ignore-patterns=qapidoc.py, schema.py,