From patchwork Mon Sep 27 19:24:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB91CC433EF for ; Mon, 27 Sep 2021 19:28:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4D236600EF for ; Mon, 27 Sep 2021 19:28:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4D236600EF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:37022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwIU-0004QG-1t for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:28:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57456) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwGn-0001la-Gx for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:49960) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwGm-0006VY-1A for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=peMJnu/+prQzMTXxvxUm3lXCpJle8C5xm4Im2CFdynU=; b=EMHWH1vlTlOeO+JExq1UyntsVp9AulGg5Zk4yEGx73IH8CCNsuFjpNz/cJfGjU+a9Q+nql 7M6oHA2ulASd5eaNPPWGwSXuL1/ALj25qmQX4i/LwQWdpVOutrBkQgiWTg7rbmD7qPJsXd P7DK/pL9dCjmgvG3V/xyIMkh89m1WQM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-351-a7m-Dj9EPiysSSlvdBL-nA-1; Mon, 27 Sep 2021 15:26:29 -0400 X-MC-Unique: a7m-Dj9EPiysSSlvdBL-nA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E4112835DE2; Mon, 27 Sep 2021 19:26:27 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5921460871; Mon, 27 Sep 2021 19:26:13 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 01/32] python/aqmp: add asynchronous QMP (AQMP) subpackage Date: Mon, 27 Sep 2021 15:24:42 -0400 Message-Id: <20210927192513.744199-2-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" For now, it's empty! Soon, it won't be. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-2-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 27 +++++++++++++++++++++++++++ python/qemu/aqmp/py.typed | 0 python/setup.cfg | 1 + 3 files changed, 28 insertions(+) create mode 100644 python/qemu/aqmp/__init__.py create mode 100644 python/qemu/aqmp/py.typed diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py new file mode 100644 index 00000000000..391141c9484 --- /dev/null +++ b/python/qemu/aqmp/__init__.py @@ -0,0 +1,27 @@ +""" +QEMU Monitor Protocol (QMP) development library & tooling. + +This package provides a fairly low-level class for communicating +asynchronously with QMP protocol servers, as implemented by QEMU, the +QEMU Guest Agent, and the QEMU Storage Daemon. + +`QMPClient` provides the main functionality of this package. All errors +raised by this library dervive from `AQMPError`, see `aqmp.error` for +additional detail. See `aqmp.events` for an in-depth tutorial on +managing QMP events. +""" + +# Copyright (C) 2020, 2021 John Snow for Red Hat, Inc. +# +# Authors: +# John Snow +# +# Based on earlier work by Luiz Capitulino . +# +# This work is licensed under the terms of the GNU GPL, version 2. See +# the COPYING file in the top-level directory. + + +# The order of these fields impact the Sphinx documentation order. +__all__ = ( +) diff --git a/python/qemu/aqmp/py.typed b/python/qemu/aqmp/py.typed new file mode 100644 index 00000000000..e69de29bb2d diff --git a/python/setup.cfg b/python/setup.cfg index fdca265fec0..d1da9853a42 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -27,6 +27,7 @@ packages = qemu.qmp qemu.machine qemu.utils + qemu.aqmp [options.package_data] * = py.typed From patchwork Mon Sep 27 19:24:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6E13C433EF for ; Mon, 27 Sep 2021 19:32:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 716B760F94 for ; Mon, 27 Sep 2021 19:32:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 716B760F94 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:45648 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwMi-0001zJ-LB for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:32:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwGt-0001vQ-0a for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36332) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwGr-0006Zj-Ec for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770796; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x8QSKZr/hBDwRAmjIKOvyCJ9OqFQmRmhv6KdaRVQLgg=; b=P0Rv78JqYdR1z9cw7q1i83bROGwHdKB3g0iZvJNteKnk+Bj1BCtvPbep+hBrkjTlQvUYt3 c6Ckm36nIW6FArMLcB7i6G5zDORElnK46DuzZdHznCjY+m/GIelDmdM0/13a2ZMhumdJCH /dbGqeNy6yx0ZMyKRyWUpxIq+UJdq1o= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-185-44rueu2kOFOhHsi75UxDcw-1; Mon, 27 Sep 2021 15:26:35 -0400 X-MC-Unique: 44rueu2kOFOhHsi75UxDcw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2CAA219200C4; Mon, 27 Sep 2021 19:26:34 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 36C8A60871; Mon, 27 Sep 2021 19:26:28 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 02/32] python/aqmp: add error classes Date: Mon, 27 Sep 2021 15:24:43 -0400 Message-Id: <20210927192513.744199-3-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-3-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 4 +++ python/qemu/aqmp/error.py | 50 ++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 python/qemu/aqmp/error.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 391141c9484..c97be950bf4 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -21,7 +21,11 @@ # This work is licensed under the terms of the GNU GPL, version 2. See # the COPYING file in the top-level directory. +from .error import AQMPError + # The order of these fields impact the Sphinx documentation order. __all__ = ( + # Exceptions + 'AQMPError', ) diff --git a/python/qemu/aqmp/error.py b/python/qemu/aqmp/error.py new file mode 100644 index 00000000000..781f49b0087 --- /dev/null +++ b/python/qemu/aqmp/error.py @@ -0,0 +1,50 @@ +""" +AQMP Error Classes + +This package seeks to provide semantic error classes that are intended +to be used directly by clients when they would like to handle particular +semantic failures (e.g. "failed to connect") without needing to know the +enumeration of possible reasons for that failure. + +AQMPError serves as the ancestor for all exceptions raised by this +package, and is suitable for use in handling semantic errors from this +library. In most cases, individual public methods will attempt to catch +and re-encapsulate various exceptions to provide a semantic +error-handling interface. + +.. admonition:: AQMP Exception Hierarchy Reference + + | `Exception` + | +-- `AQMPError` + | +-- `ConnectError` + | +-- `StateError` + | +-- `ExecInterruptedError` + | +-- `ExecuteError` + | +-- `ListenerError` + | +-- `ProtocolError` + | +-- `DeserializationError` + | +-- `UnexpectedTypeError` + | +-- `ServerParseError` + | +-- `BadReplyError` + | +-- `GreetingError` + | +-- `NegotiationError` +""" + + +class AQMPError(Exception): + """Abstract error class for all errors originating from this package.""" + + +class ProtocolError(AQMPError): + """ + Abstract error class for protocol failures. + + Semantically, these errors are generally the fault of either the + protocol server or as a result of a bug in this library. + + :param error_message: Human-readable string describing the error. + """ + def __init__(self, error_message: str): + super().__init__(error_message) + #: Human-readable error message, without any prefix. + self.error_message: str = error_message From patchwork Mon Sep 27 19:24:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19E3FC433EF for ; Mon, 27 Sep 2021 19:36:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9719B60F94 for ; Mon, 27 Sep 2021 19:36:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9719B60F94 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:54142 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwQE-0007xA-P9 for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:36:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57538) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwH9-00028m-Dw for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:32220) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwH6-0006ld-ER for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PaDrUMXqrk6ssMzNvpIsTA8HwO2XNxgyFVWpnq9cKjE=; b=RymKc/sPvcq+cpbVoMJU5CDeRt5oi6JZFL5yUBqig7qCcjA9DyWiuYncrG1Tkg9Rhy+wTT NCaGCtH7iuF0TmqWmNTgX7AelIsu9XzL/LknF7fJ8PoxdZ9kKu31bn6FbmcPtPoTeOGM+d E6fra2rg5nqG7HPkeB/pDM2g1EE0vl8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-24-obtf592sNY6XTO3eZiM1-A-1; Mon, 27 Sep 2021 15:26:48 -0400 X-MC-Unique: obtf592sNY6XTO3eZiM1-A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2CF751006AA5; Mon, 27 Sep 2021 19:26:47 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 577B760871; Mon, 27 Sep 2021 19:26:34 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 03/32] python/pylint: Add exception for TypeVar names ('T') Date: Mon, 27 Sep 2021 15:24:44 -0400 Message-Id: <20210927192513.744199-4-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" 'T' is a common TypeVar name, allow its use. See also https://github.com/PyCQA/pylint/issues/3401 -- In the future, we might be able to have a separate list of acceptable names for TypeVars exclusively. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-4-jsnow@redhat.com Signed-off-by: John Snow --- python/setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/python/setup.cfg b/python/setup.cfg index d1da9853a42..9b67afd33e2 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -101,6 +101,7 @@ good-names=i, fh, # fh = open(...) fd, # fd = os.open(...) c, # for c in string: ... + T, # for TypeVars. See pylint#3401 [pylint.similarities] # Ignore imports when computing similarities. From patchwork Mon Sep 27 19:24:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0263AC433FE for ; Mon, 27 Sep 2021 19:32:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A294460FED for ; Mon, 27 Sep 2021 19:32:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A294460FED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:45604 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwMi-0001xW-MD for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:32:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57534) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwH8-00028Q-Lb for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:54 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:31020) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwH6-0006lu-G7 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:26:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770811; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8mWniNIMmCePwjlVa0JY+qDWL1wn2A8WJ40ToWhtwqo=; b=AEn0bgp8oKCwYdw6u6Z5vtzysM0neYxmGzheppg5kTsryy4AgVvQSQZU3AhSI6zqhwy8F1 zb2l2GaIFBWkmMUE0rKIrn0ZMz4EEYVUg/1F1wQXooJ8HTGL61u+pgWIDKbdku+q4QQaqI Rrv7YIMJRD+9F6khiHNcGUKtJbZLOXg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-402-83S6mfcvPKmqsJt8S73Nqg-1; Mon, 27 Sep 2021 15:26:50 -0400 X-MC-Unique: 83S6mfcvPKmqsJt8S73Nqg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 062F41084681; Mon, 27 Sep 2021 19:26:49 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5A38A60871; Mon, 27 Sep 2021 19:26:47 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 04/32] python/aqmp: add asyncio compatibility wrappers Date: Mon, 27 Sep 2021 15:24:45 -0400 Message-Id: <20210927192513.744199-5-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Python 3.6 does not have all of the goodies that Python 3.7 does, and we need to support both. Add some compatibility wrappers needed for this purpose. (Note: Python 3.6 is EOL December 2021.) Signed-off-by: John Snow Message-id: 20210915162955.333025-5-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/util.py | 89 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 python/qemu/aqmp/util.py diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py new file mode 100644 index 00000000000..28acd995dbf --- /dev/null +++ b/python/qemu/aqmp/util.py @@ -0,0 +1,89 @@ +""" +Miscellaneous Utilities + +This module provides asyncio utilities and compatibility wrappers for +Python 3.6 to provide some features that otherwise become available in +Python 3.7+. +""" + +import asyncio +import sys +from typing import ( + Any, + Coroutine, + Optional, + TypeVar, +) + + +T = TypeVar('T') + + +# ------------------------------- +# Section: Compatibility Wrappers +# ------------------------------- + + +def create_task(coro: Coroutine[Any, Any, T], + loop: Optional[asyncio.AbstractEventLoop] = None + ) -> 'asyncio.Future[T]': + """ + Python 3.6-compatible `asyncio.create_task` wrapper. + + :param coro: The coroutine to execute in a task. + :param loop: Optionally, the loop to create the task in. + + :return: An `asyncio.Future` object. + """ + if sys.version_info >= (3, 7): + if loop is not None: + return loop.create_task(coro) + return asyncio.create_task(coro) # pylint: disable=no-member + + # Python 3.6: + return asyncio.ensure_future(coro, loop=loop) + + +def is_closing(writer: asyncio.StreamWriter) -> bool: + """ + Python 3.6-compatible `asyncio.StreamWriter.is_closing` wrapper. + + :param writer: The `asyncio.StreamWriter` object. + :return: `True` if the writer is closing, or closed. + """ + if sys.version_info >= (3, 7): + return writer.is_closing() + + # Python 3.6: + transport = writer.transport + assert isinstance(transport, asyncio.WriteTransport) + return transport.is_closing() + + +async def wait_closed(writer: asyncio.StreamWriter) -> None: + """ + Python 3.6-compatible `asyncio.StreamWriter.wait_closed` wrapper. + + :param writer: The `asyncio.StreamWriter` to wait on. + """ + if sys.version_info >= (3, 7): + await writer.wait_closed() + return + + # Python 3.6 + transport = writer.transport + assert isinstance(transport, asyncio.WriteTransport) + + while not transport.is_closing(): + await asyncio.sleep(0) + + # This is an ugly workaround, but it's the best I can come up with. + sock = transport.get_extra_info('socket') + + if sock is None: + # Our transport doesn't have a socket? ... + # Nothing we can reasonably do. + return + + while sock.fileno() != -1: + await asyncio.sleep(0) From patchwork Mon Sep 27 19:24:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F3C5C433F5 for ; Mon, 27 Sep 2021 19:38:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 82CBD60F58 for ; Mon, 27 Sep 2021 19:38:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 82CBD60F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34464 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwSl-0005HI-Nc for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:38:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57592) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHJ-0002EM-F7 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:32800) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHD-0006rh-Ro for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JGIrlkKFP3nrDR09GpW/MbczaUbWjWohWNnZj6zeGO4=; b=gi5jwEDkv5dOcnd3QBXwtyM/1C2hGlN/C/QHZVxt8JYSqanHcPiNa/29QRJTUF32dP0JIm t77BgU0pAaPLHsZX43xyu0l95r/mGkmTfm4bXY6MA0wLU2KWwxEe/Epy1d7aNJpAi+ygm6 m7W6oZ4qHETzDmI7uu6O9nNVlv56Ryo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-604-qANMpn8KNOuo5iQ05hA-zw-1; Mon, 27 Sep 2021 15:26:55 -0400 X-MC-Unique: qANMpn8KNOuo5iQ05hA-zw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 873661006AA2; Mon, 27 Sep 2021 19:26:54 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 327EC60877; Mon, 27 Sep 2021 19:26:49 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 05/32] python/aqmp: add generic async message-based protocol support Date: Mon, 27 Sep 2021 15:24:46 -0400 Message-Id: <20210927192513.744199-6-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is the bare minimum that you need to establish a full-duplex async message-based protocol with Python's asyncio. The features to be added in forthcoming commits are: - Runstate tracking - Logging - Support for incoming connections via accept() - _cb_outbound, _cb_inbound message hooks - _readline() method Signed-off-by: John Snow Message-id: 20210915162955.333025-6-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 4 +- python/qemu/aqmp/protocol.py | 521 +++++++++++++++++++++++++++++++++++ python/qemu/aqmp/util.py | 53 ++++ 3 files changed, 577 insertions(+), 1 deletion(-) create mode 100644 python/qemu/aqmp/protocol.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index c97be950bf4..5c0de72672d 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,10 +22,12 @@ # the COPYING file in the top-level directory. from .error import AQMPError +from .protocol import ConnectError # The order of these fields impact the Sphinx documentation order. __all__ = ( - # Exceptions + # Exceptions, most generic to most explicit 'AQMPError', + 'ConnectError', ) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py new file mode 100644 index 00000000000..2a93da791e2 --- /dev/null +++ b/python/qemu/aqmp/protocol.py @@ -0,0 +1,521 @@ +""" +Generic Asynchronous Message-based Protocol Support + +This module provides a generic framework for sending and receiving +messages over an asyncio stream. `AsyncProtocol` is an abstract class +that implements the core mechanisms of a simple send/receive protocol, +and is designed to be extended. + +In this package, it is used as the implementation for the `QMPClient` +class. +""" + +import asyncio +from asyncio import StreamReader, StreamWriter +from ssl import SSLContext +# import exceptions will be removed in a forthcoming commit. +# The problem stems from pylint/flake8 believing that 'Any' +# is unused because of its only use in a string-quoted type. +from typing import ( # pylint: disable=unused-import # noqa + Any, + Awaitable, + Callable, + Generic, + List, + Optional, + Tuple, + TypeVar, + Union, +) + +from .error import AQMPError +from .util import ( + bottom_half, + create_task, + flush, + is_closing, + upper_half, + wait_closed, +) + + +T = TypeVar('T') +_TaskFN = Callable[[], Awaitable[None]] # aka ``async def func() -> None`` +_FutureT = TypeVar('_FutureT', bound=Optional['asyncio.Future[Any]']) + + +class ConnectError(AQMPError): + """ + Raised when the initial connection process has failed. + + This Exception always wraps a "root cause" exception that can be + interrogated for additional information. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + def __init__(self, error_message: str, exc: Exception): + super().__init__(error_message) + #: Human-readable error string + self.error_message: str = error_message + #: Wrapped root cause exception + self.exc: Exception = exc + + def __str__(self) -> str: + return f"{self.error_message}: {self.exc!s}" + + +class AsyncProtocol(Generic[T]): + """ + AsyncProtocol implements a generic async message-based protocol. + + This protocol assumes the basic unit of information transfer between + client and server is a "message", the details of which are left up + to the implementation. It assumes the sending and receiving of these + messages is full-duplex and not necessarily correlated; i.e. it + supports asynchronous inbound messages. + + It is designed to be extended by a specific protocol which provides + the implementations for how to read and send messages. These must be + defined in `_do_recv()` and `_do_send()`, respectively. + + Other callbacks have a default implementation, but are intended to be + either extended or overridden: + + - `_establish_session`: + The base implementation starts the reader/writer tasks. + A protocol implementation can override this call, inserting + actions to be taken prior to starting the reader/writer tasks + before the super() call; actions needing to occur afterwards + can be written after the super() call. + - `_on_message`: + Actions to be performed when a message is received. + """ + # pylint: disable=too-many-instance-attributes + + # ------------------------- + # Section: Public interface + # ------------------------- + + def __init__(self) -> None: + # stream I/O + self._reader: Optional[StreamReader] = None + self._writer: Optional[StreamWriter] = None + + # Outbound Message queue + self._outgoing: asyncio.Queue[T] + + # Special, long-running tasks: + self._reader_task: Optional[asyncio.Future[None]] = None + self._writer_task: Optional[asyncio.Future[None]] = None + + # Aggregate of the above two tasks, used for Exception management. + self._bh_tasks: Optional[asyncio.Future[Tuple[None, None]]] = None + + #: Disconnect task. The disconnect implementation runs in a task + #: so that asynchronous disconnects (initiated by the + #: reader/writer) are allowed to wait for the reader/writers to + #: exit. + self._dc_task: Optional[asyncio.Future[None]] = None + + @upper_half + async def connect(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Connect to the server and begin processing message queues. + + If this call fails, `runstate` is guaranteed to be set back to `IDLE`. + + :param address: + Address to connect to; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise StateError: When the `Runstate` is not `IDLE`. + :raise ConnectError: If a connection cannot be made to the server. + """ + await self._new_session(address, ssl) + + @upper_half + async def disconnect(self) -> None: + """ + Disconnect and wait for all tasks to fully stop. + + If there was an exception that caused the reader/writers to + terminate prematurely, it will be raised here. + + :raise Exception: When the reader or writer terminate unexpectedly. + """ + self._schedule_disconnect() + await self._wait_disconnect() + + # -------------------------- + # Section: Session machinery + # -------------------------- + + @upper_half + async def _new_session(self, + address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Establish a new connection and initialize the session. + + Connect or accept a new connection, then begin the protocol + session machinery. If this call fails, `runstate` is guaranteed + to be set back to `IDLE`. + + :param address: + Address to connect to; + UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise ConnectError: + When a connection or session cannot be established. + + This exception will wrap a more concrete one. In most cases, + the wrapped exception will be `OSError` or `EOFError`. If a + protocol-level failure occurs while establishing a new + session, the wrapped error may also be an `AQMPError`. + """ + try: + phase = "connection" + await self._establish_connection(address, ssl) + + phase = "session" + await self._establish_session() + + except BaseException as err: + emsg = f"Failed to establish {phase}" + await self.disconnect() + + # NB: CancelledError is not a BaseException before Python 3.8 + if isinstance(err, asyncio.CancelledError): + raise + + if isinstance(err, Exception): + raise ConnectError(emsg, err) from err + + # Raise BaseExceptions un-wrapped, they're more important. + raise + + @upper_half + async def _establish_connection( + self, + address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None, + ) -> None: + """ + Establish a new connection. + + :param address: + Address to connect to/listen on; + UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + """ + await self._do_connect(address, ssl) + + @upper_half + async def _do_connect(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Acting as the transport client, initiate a connection to a server. + + :param address: + Address to connect to; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise OSError: For stream-related errors. + """ + if isinstance(address, tuple): + connect = asyncio.open_connection(address[0], address[1], ssl=ssl) + else: + connect = asyncio.open_unix_connection(path=address, ssl=ssl) + self._reader, self._writer = await connect + + @upper_half + async def _establish_session(self) -> None: + """ + Establish a new session. + + Starts the readers/writer tasks; subclasses may perform their + own negotiations here. The Runstate will be RUNNING upon + successful conclusion. + """ + self._outgoing = asyncio.Queue() + + reader_coro = self._bh_loop_forever(self._bh_recv_message) + writer_coro = self._bh_loop_forever(self._bh_send_message) + + self._reader_task = create_task(reader_coro) + self._writer_task = create_task(writer_coro) + + self._bh_tasks = asyncio.gather( + self._reader_task, + self._writer_task, + ) + + @upper_half + @bottom_half + def _schedule_disconnect(self) -> None: + """ + Initiate a disconnect; idempotent. + + This method is used both in the upper-half as a direct + consequence of `disconnect()`, and in the bottom-half in the + case of unhandled exceptions in the reader/writer tasks. + + It can be invoked no matter what the `runstate` is. + """ + if not self._dc_task: + self._dc_task = create_task(self._bh_disconnect()) + + @upper_half + async def _wait_disconnect(self) -> None: + """ + Waits for a previously scheduled disconnect to finish. + + This method will gather any bottom half exceptions and re-raise + the one that occurred first; presuming it to be the root cause + of any subsequent Exceptions. It is intended to be used in the + upper half of the call chain. + + :raise Exception: + Arbitrary exception re-raised on behalf of the reader/writer. + """ + assert self._dc_task + + aws: List[Awaitable[object]] = [self._dc_task] + if self._bh_tasks: + aws.insert(0, self._bh_tasks) + all_defined_tasks = asyncio.gather(*aws) + + # Ensure disconnect is done; Exception (if any) is not raised here: + await asyncio.wait((self._dc_task,)) + + try: + await all_defined_tasks # Raise Exceptions from the bottom half. + finally: + self._cleanup() + + @upper_half + def _cleanup(self) -> None: + """ + Fully reset this object to a clean state and return to `IDLE`. + """ + def _paranoid_task_erase(task: _FutureT) -> Optional[_FutureT]: + # Help to erase a task, ENSURING it is fully quiesced first. + assert (task is None) or task.done() + return None if (task and task.done()) else task + + self._dc_task = _paranoid_task_erase(self._dc_task) + self._reader_task = _paranoid_task_erase(self._reader_task) + self._writer_task = _paranoid_task_erase(self._writer_task) + self._bh_tasks = _paranoid_task_erase(self._bh_tasks) + + self._reader = None + self._writer = None + + # ---------------------------- + # Section: Bottom Half methods + # ---------------------------- + + @bottom_half + async def _bh_disconnect(self) -> None: + """ + Disconnect and cancel all outstanding tasks. + + It is designed to be called from its task context, + :py:obj:`~AsyncProtocol._dc_task`. By running in its own task, + it is free to wait on any pending actions that may still need to + occur in either the reader or writer tasks. + """ + + def _done(task: Optional['asyncio.Future[Any]']) -> bool: + return task is not None and task.done() + + # NB: We can't rely on _bh_tasks being done() here, it may not + # yet have had a chance to run and gather itself. + tasks = tuple(filter(None, (self._writer_task, self._reader_task))) + error_pathway = _done(self._reader_task) or _done(self._writer_task) + + try: + # Try to flush the writer, if possible: + if not error_pathway: + await self._bh_flush_writer() + except: + error_pathway = True + raise + finally: + # Cancel any still-running tasks: + if self._writer_task is not None and not self._writer_task.done(): + self._writer_task.cancel() + if self._reader_task is not None and not self._reader_task.done(): + self._reader_task.cancel() + + # Close out the tasks entirely (Won't raise): + if tasks: + await asyncio.wait(tasks) + + # Lastly, close the stream itself. (May raise): + await self._bh_close_stream(error_pathway) + + @bottom_half + async def _bh_flush_writer(self) -> None: + if not self._writer_task: + return + + await self._outgoing.join() + if self._writer is not None: + await flush(self._writer) + + @bottom_half + async def _bh_close_stream(self, error_pathway: bool = False) -> None: + # NB: Closing the writer also implcitly closes the reader. + if not self._writer: + return + + if not is_closing(self._writer): + self._writer.close() + + try: + await wait_closed(self._writer) + except Exception: # pylint: disable=broad-except + # It's hard to tell if the Stream is already closed or + # not. Even if one of the tasks has failed, it may have + # failed for a higher-layered protocol reason. The + # stream could still be open and perfectly fine. + # I don't know how to discern its health here. + + if error_pathway: + # We already know that *something* went wrong. Let's + # just trust that the Exception we already have is the + # better one to present to the user, even if we don't + # genuinely *know* the relationship between the two. + pass + else: + # Oops, this is a brand-new error! + raise + + @bottom_half + async def _bh_loop_forever(self, async_fn: _TaskFN) -> None: + """ + Run one of the bottom-half methods in a loop forever. + + If the bottom half ever raises any exception, schedule a + disconnect that will terminate the entire loop. + + :param async_fn: The bottom-half method to run in a loop. + """ + try: + while True: + await async_fn() + except asyncio.CancelledError: + # We have been cancelled by _bh_disconnect, exit gracefully. + return + except BaseException: + self._schedule_disconnect() + raise + + @bottom_half + async def _bh_send_message(self) -> None: + """ + Wait for an outgoing message, then send it. + + Designed to be run in `_bh_loop_forever()`. + """ + msg = await self._outgoing.get() + try: + await self._send(msg) + finally: + self._outgoing.task_done() + + @bottom_half + async def _bh_recv_message(self) -> None: + """ + Wait for an incoming message and call `_on_message` to route it. + + Designed to be run in `_bh_loop_forever()`. + """ + msg = await self._recv() + await self._on_message(msg) + + # -------------------- + # Section: Message I/O + # -------------------- + + @upper_half + @bottom_half + async def _do_recv(self) -> T: + """ + Abstract: Read from the stream and return a message. + + Very low-level; intended to only be called by `_recv()`. + """ + raise NotImplementedError + + @upper_half + @bottom_half + async def _recv(self) -> T: + """ + Read an arbitrary protocol message. + + .. warning:: + This method is intended primarily for `_bh_recv_message()` + to use in an asynchronous task loop. Using it outside of + this loop will "steal" messages from the normal routing + mechanism. It is safe to use prior to `_establish_session()`, + but should not be used otherwise. + + This method uses `_do_recv()` to retrieve the raw message, and + then transforms it using `_cb_inbound()`. + + :return: A single (filtered, processed) protocol message. + """ + # A forthcoming commit makes this method less trivial. + return await self._do_recv() + + @upper_half + @bottom_half + def _do_send(self, msg: T) -> None: + """ + Abstract: Write a message to the stream. + + Very low-level; intended to only be called by `_send()`. + """ + raise NotImplementedError + + @upper_half + @bottom_half + async def _send(self, msg: T) -> None: + """ + Send an arbitrary protocol message. + + This method will transform any outgoing messages according to + `_cb_outbound()`. + + .. warning:: + Like `_recv()`, this method is intended to be called by + the writer task loop that processes outgoing + messages. Calling it directly may circumvent logic + implemented by the caller meant to correlate outgoing and + incoming messages. + + :raise OSError: For problems with the underlying stream. + """ + # A forthcoming commit makes this method less trivial. + self._do_send(msg) + + @bottom_half + async def _on_message(self, msg: T) -> None: + """ + Called to handle the receipt of a new message. + + .. caution:: + This is executed from within the reader loop, so be advised + that waiting on either the reader or writer task will lead + to deadlock. Additionally, any unhandled exceptions will + directly cause the loop to halt, so logic may be best-kept + to a minimum if at all possible. + + :param msg: The incoming message + """ + # Nothing to do in the abstract case. diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py index 28acd995dbf..5b8f968969d 100644 --- a/python/qemu/aqmp/util.py +++ b/python/qemu/aqmp/util.py @@ -13,12 +13,65 @@ Coroutine, Optional, TypeVar, + cast, ) T = TypeVar('T') +# -------------------------- +# Section: Utility Functions +# -------------------------- + + +async def flush(writer: asyncio.StreamWriter) -> None: + """ + Utility function to ensure a StreamWriter is *fully* drained. + + `asyncio.StreamWriter.drain` only promises we will return to below + the "high-water mark". This function ensures we flush the entire + buffer -- by setting the high water mark to 0 and then calling + drain. The flow control limits are restored after the call is + completed. + """ + transport = cast(asyncio.WriteTransport, writer.transport) + + # https://github.com/python/typeshed/issues/5779 + low, high = transport.get_write_buffer_limits() # type: ignore + transport.set_write_buffer_limits(0, 0) + try: + await writer.drain() + finally: + transport.set_write_buffer_limits(high, low) + + +def upper_half(func: T) -> T: + """ + Do-nothing decorator that annotates a method as an "upper-half" method. + + These methods must not call bottom-half functions directly, but can + schedule them to run. + """ + return func + + +def bottom_half(func: T) -> T: + """ + Do-nothing decorator that annotates a method as a "bottom-half" method. + + These methods must take great care to handle their own exceptions whenever + possible. If they go unhandled, they will cause termination of the loop. + + These methods do not, in general, have the ability to directly + report information to a caller’s context and will usually be + collected as a Task result instead. + + They must not call upper-half functions directly. + """ + return func + + # ------------------------------- # Section: Compatibility Wrappers # ------------------------------- From patchwork Mon Sep 27 19:24:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 286F3C433F5 for ; Mon, 27 Sep 2021 19:29:28 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C2EAD60F4F for ; Mon, 27 Sep 2021 19:29:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org C2EAD60F4F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:40296 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwJb-0006g4-03 for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:29:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57638) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHN-0002Hs-81 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20452) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHK-0006xV-5C for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TjHpvWYBX2RUvSs+g30HcGZRMIOqRc/J2QzZNjy8HT0=; b=DZRzsKjTiafs8j4nv4utiNR9n74M8/tsJvK90jwgL2MhF5pkI7uIMD3hgF0kXdANUe+Diz nKnb+rTs7SMwSlK5h7563TN9JKFwb5Jh9Tgs+rjReCGoruUaXB7DMcHt0brRUCsbdWyXwW DQMzAWlv555bBxeVz6dVqD6RlfZ9AHU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-349-5AUv08gkMCK7q1Qc-zgWTA-1; Mon, 27 Sep 2021 15:26:59 -0400 X-MC-Unique: 5AUv08gkMCK7q1Qc-zgWTA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9E993835DE0; Mon, 27 Sep 2021 19:26:58 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id BCBF060871; Mon, 27 Sep 2021 19:26:54 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 06/32] python/aqmp: add runstate state machine to AsyncProtocol Date: Mon, 27 Sep 2021 15:24:47 -0400 Message-Id: <20210927192513.744199-7-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This serves a few purposes: 1. Protect interfaces when it's not safe to call them (via @require) 2. Add an interface by which an async client can determine if the state has changed, for the purposes of connection management. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-7-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 6 +- python/qemu/aqmp/protocol.py | 159 ++++++++++++++++++++++++++++++++++- 2 files changed, 160 insertions(+), 5 deletions(-) diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 5c0de72672d..88ead4c0238 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,12 +22,16 @@ # the COPYING file in the top-level directory. from .error import AQMPError -from .protocol import ConnectError +from .protocol import ConnectError, Runstate, StateError # The order of these fields impact the Sphinx documentation order. __all__ = ( + # Classes + 'Runstate', + # Exceptions, most generic to most explicit 'AQMPError', + 'StateError', 'ConnectError', ) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 2a93da791e2..19460857bd6 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -12,11 +12,10 @@ import asyncio from asyncio import StreamReader, StreamWriter +from enum import Enum +from functools import wraps from ssl import SSLContext -# import exceptions will be removed in a forthcoming commit. -# The problem stems from pylint/flake8 believing that 'Any' -# is unused because of its only use in a string-quoted type. -from typing import ( # pylint: disable=unused-import # noqa +from typing import ( Any, Awaitable, Callable, @@ -26,6 +25,7 @@ Tuple, TypeVar, Union, + cast, ) from .error import AQMPError @@ -44,6 +44,20 @@ _FutureT = TypeVar('_FutureT', bound=Optional['asyncio.Future[Any]']) +class Runstate(Enum): + """Protocol session runstate.""" + + #: Fully quiesced and disconnected. + IDLE = 0 + #: In the process of connecting or establishing a session. + CONNECTING = 1 + #: Fully connected and active session. + RUNNING = 2 + #: In the process of disconnecting. + #: Runstate may be returned to `IDLE` by calling `disconnect()`. + DISCONNECTING = 3 + + class ConnectError(AQMPError): """ Raised when the initial connection process has failed. @@ -65,6 +79,76 @@ def __str__(self) -> str: return f"{self.error_message}: {self.exc!s}" +class StateError(AQMPError): + """ + An API command (connect, execute, etc) was issued at an inappropriate time. + + This error is raised when a command like + :py:meth:`~AsyncProtocol.connect()` is issued at an inappropriate + time. + + :param error_message: Human-readable string describing the state violation. + :param state: The actual `Runstate` seen at the time of the violation. + :param required: The `Runstate` required to process this command. + """ + def __init__(self, error_message: str, + state: Runstate, required: Runstate): + super().__init__(error_message) + self.error_message = error_message + self.state = state + self.required = required + + +F = TypeVar('F', bound=Callable[..., Any]) # pylint: disable=invalid-name + + +# Don't Panic. +def require(required_state: Runstate) -> Callable[[F], F]: + """ + Decorator: protect a method so it can only be run in a certain `Runstate`. + + :param required_state: The `Runstate` required to invoke this method. + :raise StateError: When the required `Runstate` is not met. + """ + def _decorator(func: F) -> F: + # _decorator is the decorator that is built by calling the + # require() decorator factory; e.g.: + # + # @require(Runstate.IDLE) def foo(): ... + # will replace 'foo' with the result of '_decorator(foo)'. + + @wraps(func) + def _wrapper(proto: 'AsyncProtocol[Any]', + *args: Any, **kwargs: Any) -> Any: + # _wrapper is the function that gets executed prior to the + # decorated method. + + name = type(proto).__name__ + + if proto.runstate != required_state: + if proto.runstate == Runstate.CONNECTING: + emsg = f"{name} is currently connecting." + elif proto.runstate == Runstate.DISCONNECTING: + emsg = (f"{name} is disconnecting." + " Call disconnect() to return to IDLE state.") + elif proto.runstate == Runstate.RUNNING: + emsg = f"{name} is already connected and running." + elif proto.runstate == Runstate.IDLE: + emsg = f"{name} is disconnected and idle." + else: + assert False + raise StateError(emsg, proto.runstate, required_state) + # No StateError, so call the wrapped method. + return func(proto, *args, **kwargs) + + # Return the decorated method; + # Transforming Func to Decorated[Func]. + return cast(F, _wrapper) + + # Return the decorator instance from the decorator factory. Phew! + return _decorator + + class AsyncProtocol(Generic[T]): """ AsyncProtocol implements a generic async message-based protocol. @@ -118,7 +202,24 @@ def __init__(self) -> None: #: exit. self._dc_task: Optional[asyncio.Future[None]] = None + self._runstate = Runstate.IDLE + self._runstate_changed: Optional[asyncio.Event] = None + + @property # @upper_half + def runstate(self) -> Runstate: + """The current `Runstate` of the connection.""" + return self._runstate + @upper_half + async def runstate_changed(self) -> Runstate: + """ + Wait for the `runstate` to change, then return that runstate. + """ + await self._runstate_event.wait() + return self.runstate + + @upper_half + @require(Runstate.IDLE) async def connect(self, address: Union[str, Tuple[str, int]], ssl: Optional[SSLContext] = None) -> None: """ @@ -152,6 +253,30 @@ async def disconnect(self) -> None: # Section: Session machinery # -------------------------- + @property + def _runstate_event(self) -> asyncio.Event: + # asyncio.Event() objects should not be created prior to entrance into + # an event loop, so we can ensure we create it in the correct context. + # Create it on-demand *only* at the behest of an 'async def' method. + if not self._runstate_changed: + self._runstate_changed = asyncio.Event() + return self._runstate_changed + + @upper_half + @bottom_half + def _set_state(self, state: Runstate) -> None: + """ + Change the `Runstate` of the protocol connection. + + Signals the `runstate_changed` event. + """ + if state == self._runstate: + return + + self._runstate = state + self._runstate_event.set() + self._runstate_event.clear() + @upper_half async def _new_session(self, address: Union[str, Tuple[str, int]], @@ -176,6 +301,8 @@ async def _new_session(self, protocol-level failure occurs while establishing a new session, the wrapped error may also be an `AQMPError`. """ + assert self.runstate == Runstate.IDLE + try: phase = "connection" await self._establish_connection(address, ssl) @@ -185,6 +312,7 @@ async def _new_session(self, except BaseException as err: emsg = f"Failed to establish {phase}" + # Reset from CONNECTING back to IDLE. await self.disconnect() # NB: CancelledError is not a BaseException before Python 3.8 @@ -197,6 +325,8 @@ async def _new_session(self, # Raise BaseExceptions un-wrapped, they're more important. raise + assert self.runstate == Runstate.RUNNING + @upper_half async def _establish_connection( self, @@ -211,6 +341,14 @@ async def _establish_connection( UNIX socket path or TCP address/port. :param ssl: SSL context to use, if any. """ + assert self.runstate == Runstate.IDLE + self._set_state(Runstate.CONNECTING) + + # Allow runstate watchers to witness 'CONNECTING' state; some + # failures in the streaming layer are synchronous and will not + # otherwise yield. + await asyncio.sleep(0) + await self._do_connect(address, ssl) @upper_half @@ -240,6 +378,8 @@ async def _establish_session(self) -> None: own negotiations here. The Runstate will be RUNNING upon successful conclusion. """ + assert self.runstate == Runstate.CONNECTING + self._outgoing = asyncio.Queue() reader_coro = self._bh_loop_forever(self._bh_recv_message) @@ -253,6 +393,9 @@ async def _establish_session(self) -> None: self._writer_task, ) + self._set_state(Runstate.RUNNING) + await asyncio.sleep(0) # Allow runstate_event to process + @upper_half @bottom_half def _schedule_disconnect(self) -> None: @@ -266,6 +409,7 @@ def _schedule_disconnect(self) -> None: It can be invoked no matter what the `runstate` is. """ if not self._dc_task: + self._set_state(Runstate.DISCONNECTING) self._dc_task = create_task(self._bh_disconnect()) @upper_half @@ -281,6 +425,7 @@ async def _wait_disconnect(self) -> None: :raise Exception: Arbitrary exception re-raised on behalf of the reader/writer. """ + assert self.runstate == Runstate.DISCONNECTING assert self._dc_task aws: List[Awaitable[object]] = [self._dc_task] @@ -295,6 +440,7 @@ async def _wait_disconnect(self) -> None: await all_defined_tasks # Raise Exceptions from the bottom half. finally: self._cleanup() + self._set_state(Runstate.IDLE) @upper_half def _cleanup(self) -> None: @@ -306,6 +452,7 @@ def _paranoid_task_erase(task: _FutureT) -> Optional[_FutureT]: assert (task is None) or task.done() return None if (task and task.done()) else task + assert self.runstate == Runstate.DISCONNECTING self._dc_task = _paranoid_task_erase(self._dc_task) self._reader_task = _paranoid_task_erase(self._reader_task) self._writer_task = _paranoid_task_erase(self._writer_task) @@ -314,6 +461,9 @@ def _paranoid_task_erase(task: _FutureT) -> Optional[_FutureT]: self._reader = None self._writer = None + # NB: _runstate_changed cannot be cleared because we still need it to + # send the final runstate changed event ...! + # ---------------------------- # Section: Bottom Half methods # ---------------------------- @@ -328,6 +478,7 @@ async def _bh_disconnect(self) -> None: it is free to wait on any pending actions that may still need to occur in either the reader or writer tasks. """ + assert self.runstate == Runstate.DISCONNECTING def _done(task: Optional['asyncio.Future[Any]']) -> bool: return task is not None and task.done() From patchwork Mon Sep 27 19:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FD60C433EF for ; Mon, 27 Sep 2021 19:38:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 67E1F60F58 for ; Mon, 27 Sep 2021 19:38:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 67E1F60F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34178 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwSf-00055m-Iq for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:38:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57594) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHJ-0002EN-Tb for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:51055) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHI-0006w9-BV for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770823; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/bDySIZ0fZU0hEGQwgvD5JQ+kpi9AzbgjN24ILMs72I=; b=FpOFfG6SMztmI8gx1PB4HwlSgy+XeXstslG2ZxGPPd5nzwrRJhvwvTHqcb5hXykqpeVixZ NcaTb++FwVgVcBNNcz+XuFHPDUm6bANHHGJ8d0KWzZaX6JPumDTJYqJj5FZw+OT9chZvHm PXNzwO6YmZmAXlgzK94cakUrcDjuRM0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-422-x1jvzJKMMba4INZFBNteSg-1; Mon, 27 Sep 2021 15:27:01 -0400 X-MC-Unique: x1jvzJKMMba4INZFBNteSg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8E6601084681; Mon, 27 Sep 2021 19:27:00 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id CB98660871; Mon, 27 Sep 2021 19:26:58 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 07/32] python/aqmp: Add logging utility helpers Date: Mon, 27 Sep 2021 15:24:48 -0400 Message-Id: <20210927192513.744199-8-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-8-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/util.py | 56 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py index 5b8f968969d..52a15321889 100644 --- a/python/qemu/aqmp/util.py +++ b/python/qemu/aqmp/util.py @@ -4,10 +4,15 @@ This module provides asyncio utilities and compatibility wrappers for Python 3.6 to provide some features that otherwise become available in Python 3.7+. + +Various logging and debugging utilities are also provided, such as +`exception_summary()` and `pretty_traceback()`, used primarily for +adding information into the logging stream. """ import asyncio import sys +import traceback from typing import ( Any, Coroutine, @@ -140,3 +145,54 @@ async def wait_closed(writer: asyncio.StreamWriter) -> None: while sock.fileno() != -1: await asyncio.sleep(0) + + +# ---------------------------- +# Section: Logging & Debugging +# ---------------------------- + + +def exception_summary(exc: BaseException) -> str: + """ + Return a summary string of an arbitrary exception. + + It will be of the form "ExceptionType: Error Message", if the error + string is non-empty, and just "ExceptionType" otherwise. + """ + name = type(exc).__qualname__ + smod = type(exc).__module__ + if smod not in ("__main__", "builtins"): + name = smod + '.' + name + + error = str(exc) + if error: + return f"{name}: {error}" + return name + + +def pretty_traceback(prefix: str = " | ") -> str: + """ + Formats the current traceback, indented to provide visual distinction. + + This is useful for printing a traceback within a traceback for + debugging purposes when encapsulating errors to deliver them up the + stack; when those errors are printed, this helps provide a nice + visual grouping to quickly identify the parts of the error that + belong to the inner exception. + + :param prefix: The prefix to append to each line of the traceback. + :return: A string, formatted something like the following:: + + | Traceback (most recent call last): + | File "foobar.py", line 42, in arbitrary_example + | foo.baz() + | ArbitraryError: [Errno 42] Something bad happened! + """ + output = "".join(traceback.format_exception(*sys.exc_info())) + + exc_lines = [] + for line in output.split('\n'): + exc_lines.append(prefix + line) + + # The last line is always empty, omit it + return "\n".join(exc_lines[:-1]) From patchwork Mon Sep 27 19:24:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C306DC433F5 for ; Mon, 27 Sep 2021 19:34:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 469E8608FB for ; Mon, 27 Sep 2021 19:34:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 469E8608FB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:49084 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwO4-0004Pe-7U for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:34:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57724) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHa-0002dG-MR for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22426) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHY-00079W-DE for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770839; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jv3ygWN7baaYko3xWhaZHlbf0a5zFoDzyb6Qw6YZ6GY=; b=bOBzi1ZnBUXxXklUoxlSeYNS7W0y6ToWmC6r6a+BPE41jxZhO7eglfG8+Ryw3jX4NDwrJn u6SnoKzN2b9Q4UY8DoHrjfPES+2Xi4i8dk00lWsne3UoHmEq8f8ZsATPREe8yCdiJSabhZ N3XqC6DbcuR1KPj4xY5jPWiOboMzNgw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-410-QxYi7ukKMr6dENT5z3SmJw-1; Mon, 27 Sep 2021 15:27:17 -0400 X-MC-Unique: QxYi7ukKMr6dENT5z3SmJw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6F54A802947; Mon, 27 Sep 2021 19:27:16 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id BDB5360871; Mon, 27 Sep 2021 19:27:00 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 08/32] python/aqmp: add logging to AsyncProtocol Date: Mon, 27 Sep 2021 15:24:49 -0400 Message-Id: <20210927192513.744199-9-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Give the connection and the reader/writer tasks nicknames, and add logging statements throughout. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-9-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 82 ++++++++++++++++++++++++++++++++---- 1 file changed, 73 insertions(+), 9 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 19460857bd6..1dfd12895dc 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -14,6 +14,7 @@ from asyncio import StreamReader, StreamWriter from enum import Enum from functools import wraps +import logging from ssl import SSLContext from typing import ( Any, @@ -32,8 +33,10 @@ from .util import ( bottom_half, create_task, + exception_summary, flush, is_closing, + pretty_traceback, upper_half, wait_closed, ) @@ -174,14 +177,28 @@ class AsyncProtocol(Generic[T]): can be written after the super() call. - `_on_message`: Actions to be performed when a message is received. + + :param name: + Name used for logging messages, if any. By default, messages + will log to 'qemu.aqmp.protocol', but each individual connection + can be given its own logger by giving it a name; messages will + then log to 'qemu.aqmp.protocol.${name}'. """ # pylint: disable=too-many-instance-attributes + #: Logger object for debugging messages from this connection. + logger = logging.getLogger(__name__) + # ------------------------- # Section: Public interface # ------------------------- - def __init__(self) -> None: + def __init__(self, name: Optional[str] = None) -> None: + #: The nickname for this connection, if any. + self.name: Optional[str] = name + if self.name is not None: + self.logger = self.logger.getChild(self.name) + # stream I/O self._reader: Optional[StreamReader] = None self._writer: Optional[StreamWriter] = None @@ -205,6 +222,14 @@ def __init__(self) -> None: self._runstate = Runstate.IDLE self._runstate_changed: Optional[asyncio.Event] = None + def __repr__(self) -> str: + cls_name = type(self).__name__ + tokens = [] + if self.name is not None: + tokens.append(f"name={self.name!r}") + tokens.append(f"runstate={self.runstate.name}") + return f"<{cls_name} {' '.join(tokens)}>" + @property # @upper_half def runstate(self) -> Runstate: """The current `Runstate` of the connection.""" @@ -246,6 +271,7 @@ async def disconnect(self) -> None: :raise Exception: When the reader or writer terminate unexpectedly. """ + self.logger.debug("disconnect() called.") self._schedule_disconnect() await self._wait_disconnect() @@ -273,6 +299,8 @@ def _set_state(self, state: Runstate) -> None: if state == self._runstate: return + self.logger.debug("Transitioning from '%s' to '%s'.", + str(self._runstate), str(state)) self._runstate = state self._runstate_event.set() self._runstate_event.clear() @@ -312,8 +340,15 @@ async def _new_session(self, except BaseException as err: emsg = f"Failed to establish {phase}" - # Reset from CONNECTING back to IDLE. - await self.disconnect() + self.logger.error("%s: %s", emsg, exception_summary(err)) + self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) + try: + # Reset from CONNECTING back to IDLE. + await self.disconnect() + except: + emsg = "Unexpected bottom half exception" + self.logger.critical("%s:\n%s\n", emsg, pretty_traceback()) + raise # NB: CancelledError is not a BaseException before Python 3.8 if isinstance(err, asyncio.CancelledError): @@ -363,12 +398,16 @@ async def _do_connect(self, address: Union[str, Tuple[str, int]], :raise OSError: For stream-related errors. """ + self.logger.debug("Connecting to %s ...", address) + if isinstance(address, tuple): connect = asyncio.open_connection(address[0], address[1], ssl=ssl) else: connect = asyncio.open_unix_connection(path=address, ssl=ssl) self._reader, self._writer = await connect + self.logger.debug("Connected.") + @upper_half async def _establish_session(self) -> None: """ @@ -382,8 +421,8 @@ async def _establish_session(self) -> None: self._outgoing = asyncio.Queue() - reader_coro = self._bh_loop_forever(self._bh_recv_message) - writer_coro = self._bh_loop_forever(self._bh_send_message) + reader_coro = self._bh_loop_forever(self._bh_recv_message, 'Reader') + writer_coro = self._bh_loop_forever(self._bh_send_message, 'Writer') self._reader_task = create_task(reader_coro) self._writer_task = create_task(writer_coro) @@ -410,6 +449,7 @@ def _schedule_disconnect(self) -> None: """ if not self._dc_task: self._set_state(Runstate.DISCONNECTING) + self.logger.debug("Scheduling disconnect.") self._dc_task = create_task(self._bh_disconnect()) @upper_half @@ -492,30 +532,39 @@ def _done(task: Optional['asyncio.Future[Any]']) -> bool: # Try to flush the writer, if possible: if not error_pathway: await self._bh_flush_writer() - except: + except BaseException as err: error_pathway = True + emsg = "Failed to flush the writer" + self.logger.error("%s: %s", emsg, exception_summary(err)) + self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) raise finally: # Cancel any still-running tasks: if self._writer_task is not None and not self._writer_task.done(): + self.logger.debug("Cancelling writer task.") self._writer_task.cancel() if self._reader_task is not None and not self._reader_task.done(): + self.logger.debug("Cancelling reader task.") self._reader_task.cancel() # Close out the tasks entirely (Won't raise): if tasks: + self.logger.debug("Waiting for tasks to complete ...") await asyncio.wait(tasks) # Lastly, close the stream itself. (May raise): await self._bh_close_stream(error_pathway) + self.logger.debug("Disconnected.") @bottom_half async def _bh_flush_writer(self) -> None: if not self._writer_task: return + self.logger.debug("Draining the outbound queue ...") await self._outgoing.join() if self._writer is not None: + self.logger.debug("Flushing the StreamWriter ...") await flush(self._writer) @bottom_half @@ -525,8 +574,10 @@ async def _bh_close_stream(self, error_pathway: bool = False) -> None: return if not is_closing(self._writer): + self.logger.debug("Closing StreamWriter.") self._writer.close() + self.logger.debug("Waiting for StreamWriter to close ...") try: await wait_closed(self._writer) except Exception: # pylint: disable=broad-except @@ -541,13 +592,18 @@ async def _bh_close_stream(self, error_pathway: bool = False) -> None: # just trust that the Exception we already have is the # better one to present to the user, even if we don't # genuinely *know* the relationship between the two. - pass + self.logger.debug( + "Discarding Exception from wait_closed:\n%s\n", + pretty_traceback(), + ) else: # Oops, this is a brand-new error! raise + finally: + self.logger.debug("StreamWriter closed.") @bottom_half - async def _bh_loop_forever(self, async_fn: _TaskFN) -> None: + async def _bh_loop_forever(self, async_fn: _TaskFN, name: str) -> None: """ Run one of the bottom-half methods in a loop forever. @@ -555,16 +611,24 @@ async def _bh_loop_forever(self, async_fn: _TaskFN) -> None: disconnect that will terminate the entire loop. :param async_fn: The bottom-half method to run in a loop. + :param name: The name of this task, used for logging. """ try: while True: await async_fn() except asyncio.CancelledError: # We have been cancelled by _bh_disconnect, exit gracefully. + self.logger.debug("Task.%s: cancelled.", name) return - except BaseException: + except BaseException as err: + self.logger.error("Task.%s: %s", + name, exception_summary(err)) + self.logger.debug("Task.%s: failure:\n%s\n", + name, pretty_traceback()) self._schedule_disconnect() raise + finally: + self.logger.debug("Task.%s: exiting.", name) @bottom_half async def _bh_send_message(self) -> None: From patchwork Mon Sep 27 19:24:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EDF7C433EF for ; Mon, 27 Sep 2021 20:03:34 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1561660FC0 for ; Mon, 27 Sep 2021 20:03:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1561660FC0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:33330 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwqZ-00025z-Vc for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 16:03:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58240) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJK-0006s3-0R for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:43074) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJI-00008q-3d for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770947; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sZARz5A0DIqncp+ZPsW4uIWN0nGnS9TKdHbIxHp1hak=; b=Py91RsMhgbV0aXjZVaHYvaKnnbcZYnL2IEXjO3NASRHusNAkdGcOhOozAZaf75pfSZs98u ugcnKsce9EX1PEJgTpQoTczQB2cfZniQIuBL81EhfdUN/ZEobpC8z+82/q8+Ggv+3oFSKS mvv0JxyhvE8t30B732w84P23/5OZYVY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-487-aatzkTnbPTmEv6seZIUlbg-1; Mon, 27 Sep 2021 15:27:22 -0400 X-MC-Unique: aatzkTnbPTmEv6seZIUlbg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2B4A05074D; Mon, 27 Sep 2021 19:27:21 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id A0ABF60871; Mon, 27 Sep 2021 19:27:16 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 09/32] python/aqmp: add AsyncProtocol.accept() method Date: Mon, 27 Sep 2021 15:24:50 -0400 Message-Id: <20210927192513.744199-10-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" It's a little messier than connect, because it wasn't designed to accept *precisely one* connection. Such is life. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-10-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 89 ++++++++++++++++++++++++++++++++++-- 1 file changed, 85 insertions(+), 4 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 1dfd12895dc..62c26ede5a4 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -243,6 +243,24 @@ async def runstate_changed(self) -> Runstate: await self._runstate_event.wait() return self.runstate + @upper_half + @require(Runstate.IDLE) + async def accept(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Accept a connection and begin processing message queues. + + If this call fails, `runstate` is guaranteed to be set back to `IDLE`. + + :param address: + Address to listen to; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise StateError: When the `Runstate` is not `IDLE`. + :raise ConnectError: If a connection could not be accepted. + """ + await self._new_session(address, ssl, accept=True) + @upper_half @require(Runstate.IDLE) async def connect(self, address: Union[str, Tuple[str, int]], @@ -308,7 +326,8 @@ def _set_state(self, state: Runstate) -> None: @upper_half async def _new_session(self, address: Union[str, Tuple[str, int]], - ssl: Optional[SSLContext] = None) -> None: + ssl: Optional[SSLContext] = None, + accept: bool = False) -> None: """ Establish a new connection and initialize the session. @@ -317,9 +336,10 @@ async def _new_session(self, to be set back to `IDLE`. :param address: - Address to connect to; + Address to connect to/listen on; UNIX socket path or TCP address/port. :param ssl: SSL context to use, if any. + :param accept: Accept a connection instead of connecting when `True`. :raise ConnectError: When a connection or session cannot be established. @@ -333,7 +353,7 @@ async def _new_session(self, try: phase = "connection" - await self._establish_connection(address, ssl) + await self._establish_connection(address, ssl, accept) phase = "session" await self._establish_session() @@ -367,6 +387,7 @@ async def _establish_connection( self, address: Union[str, Tuple[str, int]], ssl: Optional[SSLContext] = None, + accept: bool = False ) -> None: """ Establish a new connection. @@ -375,6 +396,7 @@ async def _establish_connection( Address to connect to/listen on; UNIX socket path or TCP address/port. :param ssl: SSL context to use, if any. + :param accept: Accept a connection instead of connecting when `True`. """ assert self.runstate == Runstate.IDLE self._set_state(Runstate.CONNECTING) @@ -384,7 +406,66 @@ async def _establish_connection( # otherwise yield. await asyncio.sleep(0) - await self._do_connect(address, ssl) + if accept: + await self._do_accept(address, ssl) + else: + await self._do_connect(address, ssl) + + @upper_half + async def _do_accept(self, address: Union[str, Tuple[str, int]], + ssl: Optional[SSLContext] = None) -> None: + """ + Acting as the transport server, accept a single connection. + + :param address: + Address to listen on; UNIX socket path or TCP address/port. + :param ssl: SSL context to use, if any. + + :raise OSError: For stream-related errors. + """ + self.logger.debug("Awaiting connection on %s ...", address) + connected = asyncio.Event() + server: Optional[asyncio.AbstractServer] = None + + async def _client_connected_cb(reader: asyncio.StreamReader, + writer: asyncio.StreamWriter) -> None: + """Used to accept a single incoming connection, see below.""" + nonlocal server + nonlocal connected + + # A connection has been accepted; stop listening for new ones. + assert server is not None + server.close() + await server.wait_closed() + server = None + + # Register this client as being connected + self._reader, self._writer = (reader, writer) + + # Signal back: We've accepted a client! + connected.set() + + if isinstance(address, tuple): + coro = asyncio.start_server( + _client_connected_cb, + host=address[0], + port=address[1], + ssl=ssl, + backlog=1, + ) + else: + coro = asyncio.start_unix_server( + _client_connected_cb, + path=address, + ssl=ssl, + backlog=1, + ) + + server = await coro # Starts listening + await connected.wait() # Waits for the callback to fire (and finish) + assert server is None + + self.logger.debug("Connection accepted.") @upper_half async def _do_connect(self, address: Union[str, Tuple[str, int]], From patchwork Mon Sep 27 19:24:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46FD6C433F5 for ; Mon, 27 Sep 2021 19:41:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D826760E54 for ; Mon, 27 Sep 2021 19:41:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D826760E54 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:42528 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwVi-0002Jg-3F for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:41:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57812) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHt-0003Sd-JX for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:38297) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwHr-0007P9-St for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:27:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5bXw48pGXenFR8Znjwp1tOJ/BfFdT4xpOZbVwehQB38=; b=Kj31D7iEAuAnrqEyw+rmPWEnQjx8zmc5b9eFj7ecDDel9ANIlbOLLbXmummLEir8C6oaj2 xmbBcUxN1aXNoBq6u2Nigbcj4KwLwwHFfBkLoAZp+Bd2zpVakypXYdPpeB43tVUo9YZ6i9 uZM1VNBhqGsdqmi8CwiuXgv6zi2WqMU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-358-tp7kp0mtPWuBiHh0KB9mow-1; Mon, 27 Sep 2021 15:27:37 -0400 X-MC-Unique: tp7kp0mtPWuBiHh0KB9mow-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A3C5E1006AA3; Mon, 27 Sep 2021 19:27:36 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 715F060871; Mon, 27 Sep 2021 19:27:21 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 10/32] python/aqmp: add configurable read buffer limit Date: Mon, 27 Sep 2021 15:24:51 -0400 Message-Id: <20210927192513.744199-11-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , G S Niteesh Babu , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" QMP can transmit some pretty big messages, and the default limit of 64KB isn't sufficient. Make sure that we can configure it. Reported-by: G S Niteesh Babu Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-11-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 62c26ede5a4..2ef19e96932 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -189,6 +189,9 @@ class AsyncProtocol(Generic[T]): #: Logger object for debugging messages from this connection. logger = logging.getLogger(__name__) + # Maximum allowable size of read buffer + _limit = (64 * 1024) + # ------------------------- # Section: Public interface # ------------------------- @@ -452,6 +455,7 @@ async def _client_connected_cb(reader: asyncio.StreamReader, port=address[1], ssl=ssl, backlog=1, + limit=self._limit, ) else: coro = asyncio.start_unix_server( @@ -459,6 +463,7 @@ async def _client_connected_cb(reader: asyncio.StreamReader, path=address, ssl=ssl, backlog=1, + limit=self._limit, ) server = await coro # Starts listening @@ -482,9 +487,18 @@ async def _do_connect(self, address: Union[str, Tuple[str, int]], self.logger.debug("Connecting to %s ...", address) if isinstance(address, tuple): - connect = asyncio.open_connection(address[0], address[1], ssl=ssl) + connect = asyncio.open_connection( + address[0], + address[1], + ssl=ssl, + limit=self._limit, + ) else: - connect = asyncio.open_unix_connection(path=address, ssl=ssl) + connect = asyncio.open_unix_connection( + path=address, + ssl=ssl, + limit=self._limit, + ) self._reader, self._writer = await connect self.logger.debug("Connected.") From patchwork Mon Sep 27 19:24:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AB55C433EF for ; Mon, 27 Sep 2021 19:42:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9501F60F58 for ; Mon, 27 Sep 2021 19:42:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9501F60F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:46298 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwWe-0004nc-On for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:42:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57922) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIH-0004dI-52 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:44841) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIF-0007hV-C3 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770881; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vCFPN4woAGZkEh8h0rkrAcdSMP8m+13G+1mPAhIik60=; b=hYAkd+V8kpxkKyi8IKC8rrxn5S+LnO2JPkIiy86hz7u81x35K2p8+gdfeOPLjwFpoGHez6 XON8AcmrBGCx9LcaqJyVNVaTfsI+ZpHMxVq9zdc71xdMpL3jvkw1/QvzLLZYNKbGmNgcay A620XGqZgMwAFgUbZXlDYbIB9KdcwJs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-551-M3uKzz15O2W1BjICd1rpag-1; Mon, 27 Sep 2021 15:28:00 -0400 X-MC-Unique: M3uKzz15O2W1BjICd1rpag-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 28F221084683; Mon, 27 Sep 2021 19:27:59 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id E9EC360871; Mon, 27 Sep 2021 19:27:36 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 11/32] python/aqmp: add _cb_inbound and _cb_outbound logging hooks Date: Mon, 27 Sep 2021 15:24:52 -0400 Message-Id: <20210927192513.744199-12-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add hooks designed to log/filter incoming/outgoing messages. The primary intent for these is to be able to support iotests which may want to log messages with specific filters for reproducible output. Another use is for plugging into Urwid frameworks; all messages in/out can be automatically added to a rendering list for the purposes of a qmp-shell like tool. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-12-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 50 +++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 2ef19e96932..80c2004737e 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -177,6 +177,11 @@ class AsyncProtocol(Generic[T]): can be written after the super() call. - `_on_message`: Actions to be performed when a message is received. + - `_cb_outbound`: + Logging/Filtering hook for all outbound messages. + - `_cb_inbound`: + Logging/Filtering hook for all inbound messages. + This hook runs *before* `_on_message()`. :param name: Name used for logging messages, if any. By default, messages @@ -752,6 +757,43 @@ async def _bh_recv_message(self) -> None: # Section: Message I/O # -------------------- + @upper_half + @bottom_half + def _cb_outbound(self, msg: T) -> T: + """ + Callback: outbound message hook. + + This is intended for subclasses to be able to add arbitrary + hooks to filter or manipulate outgoing messages. The base + implementation does nothing but log the message without any + manipulation of the message. + + :param msg: raw outbound message + :return: final outbound message + """ + self.logger.debug("--> %s", str(msg)) + return msg + + @upper_half + @bottom_half + def _cb_inbound(self, msg: T) -> T: + """ + Callback: inbound message hook. + + This is intended for subclasses to be able to add arbitrary + hooks to filter or manipulate incoming messages. The base + implementation does nothing but log the message without any + manipulation of the message. + + This method does not "handle" incoming messages; it is a filter. + The actual "endpoint" for incoming messages is `_on_message()`. + + :param msg: raw inbound message + :return: processed inbound message + """ + self.logger.debug("<-- %s", str(msg)) + return msg + @upper_half @bottom_half async def _do_recv(self) -> T: @@ -780,8 +822,8 @@ async def _recv(self) -> T: :return: A single (filtered, processed) protocol message. """ - # A forthcoming commit makes this method less trivial. - return await self._do_recv() + message = await self._do_recv() + return self._cb_inbound(message) @upper_half @bottom_half @@ -811,7 +853,7 @@ async def _send(self, msg: T) -> None: :raise OSError: For problems with the underlying stream. """ - # A forthcoming commit makes this method less trivial. + msg = self._cb_outbound(msg) self._do_send(msg) @bottom_half @@ -826,6 +868,6 @@ async def _on_message(self, msg: T) -> None: directly cause the loop to halt, so logic may be best-kept to a minimum if at all possible. - :param msg: The incoming message + :param msg: The incoming message, already logged/filtered. """ # Nothing to do in the abstract case. From patchwork Mon Sep 27 19:24:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8FFDC433EF for ; Mon, 27 Sep 2021 19:37:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 922D860F58 for ; Mon, 27 Sep 2021 19:37:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 922D860F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:57638 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwRo-0001x2-ND for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:37:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57948) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIK-0004qT-AS for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:40312) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwII-0007kK-Lq for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770886; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8HgubT+gcsKzy0EClhxzcr9TxdKVNnDWr7mCMUpDRDI=; b=WJ6AXoWA4C4+1Q37+9n2/2YGQLRCvH9ALIZS6XbLRq2F7oD/aHoje6J7Y9LdYpaBoNKgPa oN/9NVfEqONt7SVTydxm2s5SgBmvu9bc7Xlpgf9fiVXBn84kZoFSgm8xJiu3MyOcGhsAcy Qzpzy0XyBdSwYa5iken11NablsdQvaU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-384-j_FpR6xwPlmKnXLY7fWw1g-1; Mon, 27 Sep 2021 15:28:04 -0400 X-MC-Unique: j_FpR6xwPlmKnXLY7fWw1g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7447C835DE1; Mon, 27 Sep 2021 19:28:03 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7315260871; Mon, 27 Sep 2021 19:27:59 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 12/32] python/aqmp: add AsyncProtocol._readline() method Date: Mon, 27 Sep 2021 15:24:53 -0400 Message-Id: <20210927192513.744199-13-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is added as a courtesy: many protocols are line-based, including QMP. Putting it in AsyncProtocol lets us keep the QMP class implementation just a pinch more abstract. (And, if we decide to add a QTEST implementation later, it will need this, too. (Yes, I have a QTEST implementation.)) Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-13-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/protocol.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py index 80c2004737e..32e78749c11 100644 --- a/python/qemu/aqmp/protocol.py +++ b/python/qemu/aqmp/protocol.py @@ -794,6 +794,35 @@ def _cb_inbound(self, msg: T) -> T: self.logger.debug("<-- %s", str(msg)) return msg + @upper_half + @bottom_half + async def _readline(self) -> bytes: + """ + Wait for a newline from the incoming reader. + + This method is provided as a convenience for upper-layer + protocols, as many are line-based. + + This method *may* return a sequence of bytes without a trailing + newline if EOF occurs, but *some* bytes were received. In this + case, the next call will raise `EOFError`. It is assumed that + the layer 5 protocol will decide if there is anything meaningful + to be done with a partial message. + + :raise OSError: For stream-related errors. + :raise EOFError: + If the reader stream is at EOF and there are no bytes to return. + :return: bytes, including the newline. + """ + assert self._reader is not None + msg_bytes = await self._reader.readline() + + if not msg_bytes: + if self._reader.at_eof(): + raise EOFError + + return msg_bytes + @upper_half @bottom_half async def _do_recv(self) -> T: From patchwork Mon Sep 27 19:24:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CD19C433F5 for ; Mon, 27 Sep 2021 19:44:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ACE5360F41 for ; Mon, 27 Sep 2021 19:44:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org ACE5360F41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:51010 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwYQ-0007yS-SV for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:44:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58016) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIb-0005lk-1X for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20335) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIY-0007yF-Ro for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770902; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y0vQnRO3EhC0l6WAct3Xp3KFy9imeh9ELlbknELxXP0=; b=LyOBYTxOXSZQT+wzJhiAQpU1QCeV7I9DL2CrrVpTzkOOscJFEqJ24dSn7fUR9Et6pvdz8E tGiZ/+EgVG/aMNru9AvxThRAH7LK6Wi6lBzdQ317Cx223oxCGEQXt+jyyhIaX3hEIwukkG r1vJ0Rf05WDOGsgBWpO9nMAxz6PolS8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-223-tDJ2vsttM5Kmliz-WFQ5pQ-1; Mon, 27 Sep 2021 15:28:19 -0400 X-MC-Unique: tDJ2vsttM5Kmliz-WFQ5pQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CA9D31084681; Mon, 27 Sep 2021 19:28:18 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id A05AD60871; Mon, 27 Sep 2021 19:28:03 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 13/32] python/aqmp: add QMP Message format Date: Mon, 27 Sep 2021 15:24:54 -0400 Message-Id: <20210927192513.744199-14-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The Message class is here primarily to serve as a solid type to use for mypy static typing for unambiguous annotation and documentation. We can also stuff JSON serialization and deserialization into this class itself so it can be re-used even outside this infrastructure. Signed-off-by: John Snow Reviewed-by: Eric Blake Message-id: 20210915162955.333025-14-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 4 +- python/qemu/aqmp/message.py | 209 +++++++++++++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 python/qemu/aqmp/message.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 88ead4c0238..96fff1e5f3e 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,12 +22,14 @@ # the COPYING file in the top-level directory. from .error import AQMPError +from .message import Message from .protocol import ConnectError, Runstate, StateError # The order of these fields impact the Sphinx documentation order. __all__ = ( - # Classes + # Classes, most to least important + 'Message', 'Runstate', # Exceptions, most generic to most explicit diff --git a/python/qemu/aqmp/message.py b/python/qemu/aqmp/message.py new file mode 100644 index 00000000000..f76ccc90746 --- /dev/null +++ b/python/qemu/aqmp/message.py @@ -0,0 +1,209 @@ +""" +QMP Message Format + +This module provides the `Message` class, which represents a single QMP +message sent to or from the server. +""" + +import json +from json import JSONDecodeError +from typing import ( + Dict, + Iterator, + Mapping, + MutableMapping, + Optional, + Union, +) + +from .error import ProtocolError + + +class Message(MutableMapping[str, object]): + """ + Represents a single QMP protocol message. + + QMP uses JSON objects as its basic communicative unit; so this + Python object is a :py:obj:`~collections.abc.MutableMapping`. It may + be instantiated from either another mapping (like a `dict`), or from + raw `bytes` that still need to be deserialized. + + Once instantiated, it may be treated like any other MutableMapping:: + + >>> msg = Message(b'{"hello": "world"}') + >>> assert msg['hello'] == 'world' + >>> msg['id'] = 'foobar' + >>> print(msg) + { + "hello": "world", + "id": "foobar" + } + + It can be converted to `bytes`:: + + >>> msg = Message({"hello": "world"}) + >>> print(bytes(msg)) + b'{"hello":"world","id":"foobar"}' + + Or back into a garden-variety `dict`:: + + >>> dict(msg) + {'hello': 'world'} + + + :param value: Initial value, if any. + :param eager: + When `True`, attempt to serialize or deserialize the initial value + immediately, so that conversion exceptions are raised during + the call to ``__init__()``. + """ + # pylint: disable=too-many-ancestors + + def __init__(self, + value: Union[bytes, Mapping[str, object]] = b'{}', *, + eager: bool = True): + self._data: Optional[bytes] = None + self._obj: Optional[Dict[str, object]] = None + + if isinstance(value, bytes): + self._data = value + if eager: + self._obj = self._deserialize(self._data) + else: + self._obj = dict(value) + if eager: + self._data = self._serialize(self._obj) + + # Methods necessary to implement the MutableMapping interface, see: + # https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping + + # We get pop, popitem, clear, update, setdefault, __contains__, + # keys, items, values, get, __eq__ and __ne__ for free. + + def __getitem__(self, key: str) -> object: + return self._object[key] + + def __setitem__(self, key: str, value: object) -> None: + self._object[key] = value + self._data = None + + def __delitem__(self, key: str) -> None: + del self._object[key] + self._data = None + + def __iter__(self) -> Iterator[str]: + return iter(self._object) + + def __len__(self) -> int: + return len(self._object) + + # Dunder methods not related to MutableMapping: + + def __repr__(self) -> str: + if self._obj is not None: + return f"Message({self._object!r})" + return f"Message({bytes(self)!r})" + + def __str__(self) -> str: + """Pretty-printed representation of this QMP message.""" + return json.dumps(self._object, indent=2) + + def __bytes__(self) -> bytes: + """bytes representing this QMP message.""" + if self._data is None: + self._data = self._serialize(self._obj or {}) + return self._data + + # Conversion Methods + + @property + def _object(self) -> Dict[str, object]: + """ + A `dict` representing this QMP message. + + Generated on-demand, if required. This property is private + because it returns an object that could be used to invalidate + the internal state of the `Message` object. + """ + if self._obj is None: + self._obj = self._deserialize(self._data or b'{}') + return self._obj + + @classmethod + def _serialize(cls, value: object) -> bytes: + """ + Serialize a JSON object as `bytes`. + + :raise ValueError: When the object cannot be serialized. + :raise TypeError: When the object cannot be serialized. + + :return: `bytes` ready to be sent over the wire. + """ + return json.dumps(value, separators=(',', ':')).encode('utf-8') + + @classmethod + def _deserialize(cls, data: bytes) -> Dict[str, object]: + """ + Deserialize JSON `bytes` into a native Python `dict`. + + :raise DeserializationError: + If JSON deserialization fails for any reason. + :raise UnexpectedTypeError: + If the data does not represent a JSON object. + + :return: A `dict` representing this QMP message. + """ + try: + obj = json.loads(data) + except JSONDecodeError as err: + emsg = "Failed to deserialize QMP message." + raise DeserializationError(emsg, data) from err + if not isinstance(obj, dict): + raise UnexpectedTypeError( + "QMP message is not a JSON object.", + obj + ) + return obj + + +class DeserializationError(ProtocolError): + """ + A QMP message was not understood as JSON. + + When this Exception is raised, ``__cause__`` will be set to the + `json.JSONDecodeError` Exception, which can be interrogated for + further details. + + :param error_message: Human-readable string describing the error. + :param raw: The raw `bytes` that prompted the failure. + """ + def __init__(self, error_message: str, raw: bytes): + super().__init__(error_message) + #: The raw `bytes` that were not understood as JSON. + self.raw: bytes = raw + + def __str__(self) -> str: + return "\n".join([ + super().__str__(), + f" raw bytes were: {str(self.raw)}", + ]) + + +class UnexpectedTypeError(ProtocolError): + """ + A QMP message was JSON, but not a JSON object. + + :param error_message: Human-readable string describing the error. + :param value: The deserialized JSON value that wasn't an object. + """ + def __init__(self, error_message: str, value: object): + super().__init__(error_message) + #: The JSON value that was expected to be an object. + self.value: object = value + + def __str__(self) -> str: + strval = json.dumps(self.value, indent=2) + return "\n".join([ + super().__str__(), + f" json value was: {strval}", + ]) From patchwork Mon Sep 27 19:24:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58041C433F5 for ; Mon, 27 Sep 2021 19:39:51 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9C5E60F58 for ; Mon, 27 Sep 2021 19:39:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E9C5E60F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:37752 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwTd-0007TO-2q for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:39:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58048) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIg-0005vl-7H for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44894) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIe-00083L-7a for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770907; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xq8uAZ2bVac51Rl+53v3BWKwYM+8dh/mE6ekr8YLEGA=; b=AVmvWf33T0lGiQrFUW8V9kCDpxaM+YCLfaJOMr3ttzLQQs9kEEG6Lo4/g4w8yrafC5Os95 JD0Ix5fKx4ZX4ddcRpjZNc5fQR8IAGm6tzmM3/0yied3TUvbtanCIUuIbAPnBoYqsYhYMu FE9VDbkQ83NY3phhNBVcO+U5Fhr4oQA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-482-aXAKj7gtMAKuMVGWAGT5QQ-1; Mon, 27 Sep 2021 15:28:24 -0400 X-MC-Unique: aXAKj7gtMAKuMVGWAGT5QQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D04A8835DEE; Mon, 27 Sep 2021 19:28:23 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0549D60871; Mon, 27 Sep 2021 19:28:18 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 14/32] python/aqmp: add well-known QMP object models Date: Mon, 27 Sep 2021 15:24:55 -0400 Message-Id: <20210927192513.744199-15-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The QMP spec doesn't define very many objects that are iron-clad in their format, but there are a few. This module makes it trivial to validate them without relying on an external third-party library. Signed-off-by: John Snow Message-id: 20210915162955.333025-15-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/models.py | 133 +++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 python/qemu/aqmp/models.py diff --git a/python/qemu/aqmp/models.py b/python/qemu/aqmp/models.py new file mode 100644 index 00000000000..24c94123ac0 --- /dev/null +++ b/python/qemu/aqmp/models.py @@ -0,0 +1,133 @@ +""" +QMP Data Models + +This module provides simplistic data classes that represent the few +structures that the QMP spec mandates; they are used to verify incoming +data to make sure it conforms to spec. +""" +# pylint: disable=too-few-public-methods + +from collections import abc +from typing import ( + Any, + Mapping, + Optional, + Sequence, +) + + +class Model: + """ + Abstract data model, representing some QMP object of some kind. + + :param raw: The raw object to be validated. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + self._raw = raw + + def _check_key(self, key: str) -> None: + if key not in self._raw: + raise KeyError(f"'{self._name}' object requires '{key}' member") + + def _check_value(self, key: str, type_: type, typestr: str) -> None: + assert key in self._raw + if not isinstance(self._raw[key], type_): + raise TypeError( + f"'{self._name}' member '{key}' must be a {typestr}" + ) + + def _check_member(self, key: str, type_: type, typestr: str) -> None: + self._check_key(key) + self._check_value(key, type_, typestr) + + @property + def _name(self) -> str: + return type(self).__name__ + + def __repr__(self) -> str: + return f"{self._name}({self._raw!r})" + + +class Greeting(Model): + """ + Defined in qmp-spec.txt, section 2.2, "Server Greeting". + + :param raw: The raw Greeting object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'QMP' member + self.QMP: QMPGreeting # pylint: disable=invalid-name + + self._check_member('QMP', abc.Mapping, "JSON object") + self.QMP = QMPGreeting(self._raw['QMP']) + + +class QMPGreeting(Model): + """ + Defined in qmp-spec.txt, section 2.2, "Server Greeting". + + :param raw: The raw QMPGreeting object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'version' member + self.version: Mapping[str, object] + #: 'capabilities' member + self.capabilities: Sequence[object] + + self._check_member('version', abc.Mapping, "JSON object") + self.version = self._raw['version'] + + self._check_member('capabilities', abc.Sequence, "JSON array") + self.capabilities = self._raw['capabilities'] + + +class ErrorResponse(Model): + """ + Defined in qmp-spec.txt, section 2.4.2, "error". + + :param raw: The raw ErrorResponse object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'error' member + self.error: ErrorInfo + #: 'id' member + self.id: Optional[object] = None # pylint: disable=invalid-name + + self._check_member('error', abc.Mapping, "JSON object") + self.error = ErrorInfo(self._raw['error']) + + if 'id' in raw: + self.id = raw['id'] + + +class ErrorInfo(Model): + """ + Defined in qmp-spec.txt, section 2.4.2, "error". + + :param raw: The raw ErrorInfo object. + :raise KeyError: If any required fields are absent. + :raise TypeError: If any required fields have the wrong type. + """ + def __init__(self, raw: Mapping[str, Any]): + super().__init__(raw) + #: 'class' member, with an underscore to avoid conflicts in Python. + self.class_: str + #: 'desc' member + self.desc: str + + self._check_member('class', str, "string") + self.class_ = self._raw['class'] + + self._check_member('desc', str, "string") + self.desc = self._raw['desc'] From patchwork Mon Sep 27 19:24:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFDF0C433F5 for ; Mon, 27 Sep 2021 19:43:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 44D3A60F58 for ; Mon, 27 Sep 2021 19:43:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 44D3A60F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:46354 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwWh-0004px-Be for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:42:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58084) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIj-0005zH-7g for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:35563) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIg-00083p-4y for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770909; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YLwYyupFY6a1Wg0kOiMiwpc5DiHAwJ0oGoTqpY2Ps8I=; b=CaKWBKoi4B7XC2rnljkrWYVwLkjLuNVF+Bpgn9ZzuVJYPHB9kCKblZal1KvH8xXhDH0Vvc Sofx025Ay+bmxRfymyKja+8U7PCXW3klJr3wixVwKhtsYEGx7Q22sTtT7FBo4VhJn3Skv1 CHTMeLuoUHzqVXwzQXzPct9OiI+6ckw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-305-MXocsPHdNpiQ8UpCqkmKYQ-1; Mon, 27 Sep 2021 15:28:26 -0400 X-MC-Unique: MXocsPHdNpiQ8UpCqkmKYQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C34F2835DE3; Mon, 27 Sep 2021 19:28:25 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0896260871; Mon, 27 Sep 2021 19:28:23 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 15/32] python/aqmp: add QMP event support Date: Mon, 27 Sep 2021 15:24:56 -0400 Message-Id: <20210927192513.744199-16-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This class was designed as a "mix-in" primarily so that the feature could be given its own treatment in its own python module. It gets quite a bit too long otherwise. Signed-off-by: John Snow Message-id: 20210915162955.333025-16-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 2 + python/qemu/aqmp/events.py | 706 +++++++++++++++++++++++++++++++++++ 2 files changed, 708 insertions(+) create mode 100644 python/qemu/aqmp/events.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 96fff1e5f3e..829166a2e2e 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -22,6 +22,7 @@ # the COPYING file in the top-level directory. from .error import AQMPError +from .events import EventListener from .message import Message from .protocol import ConnectError, Runstate, StateError @@ -30,6 +31,7 @@ __all__ = ( # Classes, most to least important 'Message', + 'EventListener', 'Runstate', # Exceptions, most generic to most explicit diff --git a/python/qemu/aqmp/events.py b/python/qemu/aqmp/events.py new file mode 100644 index 00000000000..fb81d216102 --- /dev/null +++ b/python/qemu/aqmp/events.py @@ -0,0 +1,706 @@ +""" +AQMP Events and EventListeners + +Asynchronous QMP uses `EventListener` objects to listen for events. An +`EventListener` is a FIFO event queue that can be pre-filtered to listen +for only specific events. Each `EventListener` instance receives its own +copy of events that it hears, so events may be consumed without fear or +worry for depriving other listeners of events they need to hear. + + +EventListener Tutorial +---------------------- + +In all of the following examples, we assume that we have a `QMPClient` +instantiated named ``qmp`` that is already connected. + + +`listener()` context blocks with one name +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most basic usage is by using the `listener()` context manager to +construct them: + +.. code:: python + + with qmp.listener('STOP') as listener: + await qmp.execute('stop') + await listener.get() + +The listener is active only for the duration of the ‘with’ block. This +instance listens only for ‘STOP’ events. + + +`listener()` context blocks with two or more names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Multiple events can be selected for by providing any ``Iterable[str]``: + +.. code:: python + + with qmp.listener(('STOP', 'RESUME')) as listener: + await qmp.execute('stop') + event = await listener.get() + assert event['event'] == 'STOP' + + await qmp.execute('cont') + event = await listener.get() + assert event['event'] == 'RESUME' + + +`listener()` context blocks with no names +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By omitting names entirely, you can listen to ALL events. + +.. code:: python + + with qmp.listener() as listener: + await qmp.execute('stop') + event = await listener.get() + assert event['event'] == 'STOP' + +This isn’t a very good use case for this feature: In a non-trivial +running system, we may not know what event will arrive next. Grabbing +the top of a FIFO queue returning multiple kinds of events may be prone +to error. + + +Using async iterators to retrieve events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you’d like to simply watch what events happen to arrive, you can use +the listener as an async iterator: + +.. code:: python + + with qmp.listener() as listener: + async for event in listener: + print(f"Event arrived: {event['event']}") + +This is analogous to the following code: + +.. code:: python + + with qmp.listener() as listener: + while True: + event = listener.get() + print(f"Event arrived: {event['event']}") + +This event stream will never end, so these blocks will never terminate. + + +Using asyncio.Task to concurrently retrieve events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since a listener’s event stream will never terminate, it is not likely +useful to use that form in a script. For longer-running clients, we can +create event handlers by using `asyncio.Task` to create concurrent +coroutines: + +.. code:: python + + async def print_events(listener): + try: + async for event in listener: + print(f"Event arrived: {event['event']}") + except asyncio.CancelledError: + return + + with qmp.listener() as listener: + task = asyncio.Task(print_events(listener)) + await qmp.execute('stop') + await qmp.execute('cont') + task.cancel() + await task + +However, there is no guarantee that these events will be received by the +time we leave this context block. Once the context block is exited, the +listener will cease to hear any new events, and becomes inert. + +Be mindful of the timing: the above example will *probably*– but does +not *guarantee*– that both STOP/RESUMED events will be printed. The +example below outlines how to use listeners outside of a context block. + + +Using `register_listener()` and `remove_listener()` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To create a listener with a longer lifetime, beyond the scope of a +single block, create a listener and then call `register_listener()`: + +.. code:: python + + class MyClient: + def __init__(self, qmp): + self.qmp = qmp + self.listener = EventListener() + + async def print_events(self): + try: + async for event in self.listener: + print(f"Event arrived: {event['event']}") + except asyncio.CancelledError: + return + + async def run(self): + self.task = asyncio.Task(self.print_events) + self.qmp.register_listener(self.listener) + await qmp.execute('stop') + await qmp.execute('cont') + + async def stop(self): + self.task.cancel() + await self.task + self.qmp.remove_listener(self.listener) + +The listener can be deactivated by using `remove_listener()`. When it is +removed, any possible pending events are cleared and it can be +re-registered at a later time. + + +Using the built-in all events listener +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The `QMPClient` object creates its own default listener named +:py:obj:`~Events.events` that can be used for the same purpose without +having to create your own: + +.. code:: python + + async def print_events(listener): + try: + async for event in listener: + print(f"Event arrived: {event['event']}") + except asyncio.CancelledError: + return + + task = asyncio.Task(print_events(qmp.events)) + + await qmp.execute('stop') + await qmp.execute('cont') + + task.cancel() + await task + + +Using both .get() and async iterators +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The async iterator and `get()` methods pull events from the same FIFO +queue. If you mix the usage of both, be aware: Events are emitted +precisely once per listener. + +If multiple contexts try to pull events from the same listener instance, +events are still emitted only precisely once. + +This restriction can be lifted by creating additional listeners. + + +Creating multiple listeners +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Additional `EventListener` objects can be created at-will. Each one +receives its own copy of events, with separate FIFO event queues. + +.. code:: python + + my_listener = EventListener() + qmp.register_listener(my_listener) + + await qmp.execute('stop') + copy1 = await my_listener.get() + copy2 = await qmp.events.get() + + assert copy1 == copy2 + +In this example, we await an event from both a user-created +`EventListener` and the built-in events listener. Both receive the same +event. + + +Clearing listeners +~~~~~~~~~~~~~~~~~~ + +`EventListener` objects can be cleared, clearing all events seen thus far: + +.. code:: python + + await qmp.execute('stop') + qmp.events.clear() + await qmp.execute('cont') + event = await qmp.events.get() + assert event['event'] == 'RESUME' + +`EventListener` objects are FIFO queues. If events are not consumed, +they will remain in the queue until they are witnessed or discarded via +`clear()`. FIFO queues will be drained automatically upon leaving a +context block, or when calling `remove_listener()`. + + +Accessing listener history +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`EventListener` objects record their history. Even after being cleared, +you can obtain a record of all events seen so far: + +.. code:: python + + await qmp.execute('stop') + await qmp.execute('cont') + qmp.events.clear() + + assert len(qmp.events.history) == 2 + assert qmp.events.history[0]['event'] == 'STOP' + assert qmp.events.history[1]['event'] == 'RESUME' + +The history is updated immediately and does not require the event to be +witnessed first. + + +Using event filters +~~~~~~~~~~~~~~~~~~~ + +`EventListener` objects can be given complex filtering criteria if names +are not sufficient: + +.. code:: python + + def job1_filter(event) -> bool: + event_data = event.get('data', {}) + event_job_id = event_data.get('id') + return event_job_id == "job1" + + with qmp.listener('JOB_STATUS_CHANGE', job1_filter) as listener: + await qmp.execute('blockdev-backup', arguments={'job-id': 'job1', ...}) + async for event in listener: + if event['data']['status'] == 'concluded': + break + +These filters might be most useful when parameterized. `EventListener` +objects expect a function that takes only a single argument (the raw +event, as a `Message`) and returns a bool; True if the event should be +accepted into the stream. You can create a function that adapts this +signature to accept configuration parameters: + +.. code:: python + + def job_filter(job_id: str) -> EventFilter: + def filter(event: Message) -> bool: + return event['data']['id'] == job_id + return filter + + with qmp.listener('JOB_STATUS_CHANGE', job_filter('job2')) as listener: + await qmp.execute('blockdev-backup', arguments={'job-id': 'job2', ...}) + async for event in listener: + if event['data']['status'] == 'concluded': + break + + +Activating an existing listener with `listen()` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Listeners with complex, long configurations can also be created manually +and activated temporarily by using `listen()` instead of `listener()`: + +.. code:: python + + listener = EventListener(('BLOCK_JOB_COMPLETED', 'BLOCK_JOB_CANCELLED', + 'BLOCK_JOB_ERROR', 'BLOCK_JOB_READY', + 'BLOCK_JOB_PENDING', 'JOB_STATUS_CHANGE')) + + with qmp.listen(listener): + await qmp.execute('blockdev-backup', arguments={'job-id': 'job3', ...}) + async for event in listener: + print(event) + if event['event'] == 'BLOCK_JOB_COMPLETED': + break + +Any events that are not witnessed by the time the block is left will be +cleared from the queue; entering the block is an implicit +`register_listener()` and leaving the block is an implicit +`remove_listener()`. + + +Activating multiple existing listeners with `listen()` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +While `listener()` is only capable of creating a single listener, +`listen()` is capable of activating multiple listeners simultaneously: + +.. code:: python + + def job_filter(job_id: str) -> EventFilter: + def filter(event: Message) -> bool: + return event['data']['id'] == job_id + return filter + + jobA = EventListener('JOB_STATUS_CHANGE', job_filter('jobA')) + jobB = EventListener('JOB_STATUS_CHANGE', job_filter('jobB')) + + with qmp.listen(jobA, jobB): + qmp.execute('blockdev-create', arguments={'job-id': 'jobA', ...}) + qmp.execute('blockdev-create', arguments={'job-id': 'jobB', ...}) + + async for event in jobA.get(): + if event['data']['status'] == 'concluded': + break + async for event in jobB.get(): + if event['data']['status'] == 'concluded': + break + + +Extending the `EventListener` class +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the case that a more specialized `EventListener` is desired to +provide either more functionality or more compact syntax for specialized +cases, it can be extended. + +One of the key methods to extend or override is +:py:meth:`~EventListener.accept()`. The default implementation checks an +incoming message for: + +1. A qualifying name, if any :py:obj:`~EventListener.names` were + specified at initialization time +2. That :py:obj:`~EventListener.event_filter()` returns True. + +This can be modified however you see fit to change the criteria for +inclusion in the stream. + +For convenience, a ``JobListener`` class could be created that simply +bakes in configuration so it does not need to be repeated: + +.. code:: python + + class JobListener(EventListener): + def __init__(self, job_id: str): + super().__init__(('BLOCK_JOB_COMPLETED', 'BLOCK_JOB_CANCELLED', + 'BLOCK_JOB_ERROR', 'BLOCK_JOB_READY', + 'BLOCK_JOB_PENDING', 'JOB_STATUS_CHANGE')) + self.job_id = job_id + + def accept(self, event) -> bool: + if not super().accept(event): + return False + if event['event'] in ('BLOCK_JOB_PENDING', 'JOB_STATUS_CHANGE'): + return event['data']['id'] == job_id + return event['data']['device'] == job_id + +From here on out, you can conjure up a custom-purpose listener that +listens only for job-related events for a specific job-id easily: + +.. code:: python + + listener = JobListener('job4') + with qmp.listener(listener): + await qmp.execute('blockdev-backup', arguments={'job-id': 'job4', ...}) + async for event in listener: + print(event) + if event['event'] == 'BLOCK_JOB_COMPLETED': + break + + +Experimental Interfaces & Design Issues +--------------------------------------- + +These interfaces are not ones I am sure I will keep or otherwise modify +heavily. + +qmp.listener()’s type signature +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`listener()` does not return anything, because it was assumed the caller +already had a handle to the listener. However, for +``qmp.listener(EventListener())`` forms, the caller will not have saved +a handle to the listener. + +Because this function can accept *many* listeners, I found it hard to +accurately type in a way where it could be used in both “one” or “many” +forms conveniently and in a statically type-safe manner. + +Ultimately, I removed the return altogether, but perhaps with more time +I can work out a way to re-add it. + + +API Reference +------------- + +""" + +import asyncio +from contextlib import contextmanager +import logging +from typing import ( + AsyncIterator, + Callable, + Iterable, + Iterator, + List, + Optional, + Set, + Tuple, + Union, +) + +from .error import AQMPError +from .message import Message + + +EventNames = Union[str, Iterable[str], None] +EventFilter = Callable[[Message], bool] + + +class ListenerError(AQMPError): + """ + Generic error class for `EventListener`-related problems. + """ + + +class EventListener: + """ + Selectively listens for events with runtime configurable filtering. + + This class is designed to be directly usable for the most common cases, + but it can be extended to provide more rigorous control. + + :param names: + One or more names of events to listen for. + When not provided, listen for ALL events. + :param event_filter: + An optional event filtering function. + When names are also provided, this acts as a secondary filter. + + When ``names`` and ``event_filter`` are both provided, the names + will be filtered first, and then the filter function will be called + second. The event filter function can assume that the format of the + event is a known format. + """ + def __init__( + self, + names: EventNames = None, + event_filter: Optional[EventFilter] = None, + ): + # Queue of 'heard' events yet to be witnessed by a caller. + self._queue: 'asyncio.Queue[Message]' = asyncio.Queue() + + # Intended as a historical record, NOT a processing queue or backlog. + self._history: List[Message] = [] + + #: Primary event filter, based on one or more event names. + self.names: Set[str] = set() + if isinstance(names, str): + self.names.add(names) + elif names is not None: + self.names.update(names) + + #: Optional, secondary event filter. + self.event_filter: Optional[EventFilter] = event_filter + + @property + def history(self) -> Tuple[Message, ...]: + """ + A read-only history of all events seen so far. + + This represents *every* event, including those not yet witnessed + via `get()` or ``async for``. It persists between `clear()` + calls and is immutable. + """ + return tuple(self._history) + + def accept(self, event: Message) -> bool: + """ + Determine if this listener accepts this event. + + This method determines which events will appear in the stream. + The default implementation simply checks the event against the + list of names and the event_filter to decide if this + `EventListener` accepts a given event. It can be + overridden/extended to provide custom listener behavior. + + User code is not expected to need to invoke this method. + + :param event: The event under consideration. + :return: `True`, if this listener accepts this event. + """ + name_ok = (not self.names) or (event['event'] in self.names) + return name_ok and ( + (not self.event_filter) or self.event_filter(event) + ) + + async def put(self, event: Message) -> None: + """ + Conditionally put a new event into the FIFO queue. + + This method is not designed to be invoked from user code, and it + should not need to be overridden. It is a public interface so + that `QMPClient` has an interface by which it can inform + registered listeners of new events. + + The event will be put into the queue if + :py:meth:`~EventListener.accept()` returns `True`. + + :param event: The new event to put into the FIFO queue. + """ + if not self.accept(event): + return + + self._history.append(event) + await self._queue.put(event) + + async def get(self) -> Message: + """ + Wait for the very next event in this stream. + + If one is already available, return that one. + """ + return await self._queue.get() + + def clear(self) -> None: + """ + Clear this listener of all pending events. + + Called when an `EventListener` is being unregistered, this clears the + pending FIFO queue synchronously. It can be also be used to + manually clear any pending events, if desired. + + .. warning:: + Take care when discarding events. Cleared events will be + silently tossed on the floor. All events that were ever + accepted by this listener are visible in `history()`. + """ + while True: + try: + self._queue.get_nowait() + except asyncio.QueueEmpty: + break + + def __aiter__(self) -> AsyncIterator[Message]: + return self + + async def __anext__(self) -> Message: + """ + Enables the `EventListener` to function as an async iterator. + + It may be used like this: + + .. code:: python + + async for event in listener: + print(event) + + These iterators will never terminate of their own accord; you + must provide break conditions or otherwise prepare to run them + in an `asyncio.Task` that can be cancelled. + """ + return await self.get() + + +class Events: + """ + Events is a mix-in class that adds event functionality to the QMP class. + + It's designed specifically as a mix-in for `QMPClient`, and it + relies upon the class it is being mixed into having a 'logger' + property. + """ + def __init__(self) -> None: + self._listeners: List[EventListener] = [] + + #: Default, all-events `EventListener`. + self.events: EventListener = EventListener() + self.register_listener(self.events) + + # Parent class needs to have a logger + self.logger: logging.Logger + + async def _event_dispatch(self, msg: Message) -> None: + """ + Given a new event, propagate it to all of the active listeners. + + :param msg: The event to propagate. + """ + for listener in self._listeners: + await listener.put(msg) + + def register_listener(self, listener: EventListener) -> None: + """ + Register and activate an `EventListener`. + + :param listener: The listener to activate. + :raise ListenerError: If the given listener is already registered. + """ + if listener in self._listeners: + raise ListenerError("Attempted to re-register existing listener") + self.logger.debug("Registering %s.", str(listener)) + self._listeners.append(listener) + + def remove_listener(self, listener: EventListener) -> None: + """ + Unregister and deactivate an `EventListener`. + + The removed listener will have its pending events cleared via + `clear()`. The listener can be re-registered later when + desired. + + :param listener: The listener to deactivate. + :raise ListenerError: If the given listener is not registered. + """ + if listener == self.events: + raise ListenerError("Cannot remove the default listener.") + self.logger.debug("Removing %s.", str(listener)) + listener.clear() + self._listeners.remove(listener) + + @contextmanager + def listen(self, *listeners: EventListener) -> Iterator[None]: + r""" + Context manager: Temporarily listen with an `EventListener`. + + Accepts one or more `EventListener` objects and registers them, + activating them for the duration of the context block. + + `EventListener` objects will have any pending events in their + FIFO queue cleared upon exiting the context block, when they are + deactivated. + + :param \*listeners: One or more EventListeners to activate. + :raise ListenerError: If the given listener(s) are already active. + """ + _added = [] + + try: + for listener in listeners: + self.register_listener(listener) + _added.append(listener) + + yield + + finally: + for listener in _added: + self.remove_listener(listener) + + @contextmanager + def listener( + self, + names: EventNames = (), + event_filter: Optional[EventFilter] = None + ) -> Iterator[EventListener]: + """ + Context manager: Temporarily listen with a new `EventListener`. + + Creates an `EventListener` object and registers it, activating + it for the duration of the context block. + + :param names: + One or more names of events to listen for. + When not provided, listen for ALL events. + :param event_filter: + An optional event filtering function. + When names are also provided, this acts as a secondary filter. + + :return: The newly created and active `EventListener`. + """ + listener = EventListener(names, event_filter) + with self.listen(listener): + yield listener From patchwork Mon Sep 27 19:24:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEBB0C433F5 for ; Mon, 27 Sep 2021 19:32:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB8B360FD7 for ; Mon, 27 Sep 2021 19:32:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AB8B360FD7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:45574 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwMh-0001wK-Lu for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:32:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58072) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIi-0005z0-FA for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22500) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIg-00085R-Ug for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770910; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kDijz/LwifqcDFLzUEAiLFrJezSMu5m1rFLTTBscbD0=; b=XTC+GBIUnClhVLRBCIkLE5OQ9WhvcPxxNCkvEwqUNvzvleS79AX+PJF67Dk4NMWWLy0LXy wNuVal4VpZ9UH5vgDahHWVSZ4dYCnKkhYZA8mxaFRCD4i5i3Unld9uUBnvQUbItziMo0Cs p7o9VpqDdYnpZfjJhwLacf6KTL4exI0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-82-0Pv22NnlNj-kny2L7mOQrw-1; Mon, 27 Sep 2021 15:28:28 -0400 X-MC-Unique: 0Pv22NnlNj-kny2L7mOQrw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B5C161006AA4; Mon, 27 Sep 2021 19:28:27 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1395760871; Mon, 27 Sep 2021 19:28:25 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 16/32] python/pylint: disable too-many-function-args Date: Mon, 27 Sep 2021 15:24:57 -0400 Message-Id: <20210927192513.744199-17-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" too-many-function-args seems prone to failure when considering things like Method Resolution Order, which mypy gets correct. When dealing with multiple inheritance, pylint doesn't seem to understand which method will actually get called, while mypy does. Remove the less powerful, redundant check. Signed-off-by: John Snow Message-id: 20210915162955.333025-17-jsnow@redhat.com Signed-off-by: John Snow --- python/setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/python/setup.cfg b/python/setup.cfg index 9b67afd33e2..70957ab7e4d 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -89,6 +89,7 @@ ignore_missing_imports = True # no Warning level messages displayed, use "--disable=all --enable=classes # --disable=W". disable=consider-using-f-string, + too-many-function-args, # mypy handles this with less false positives. [pylint.basic] # Good variable names which should always be accepted, separated by a comma. From patchwork Mon Sep 27 19:24:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C98EC433FE for ; Mon, 27 Sep 2021 19:36:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9744D60FD7 for ; Mon, 27 Sep 2021 19:36:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9744D60FD7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:54072 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwQE-0007tH-Pn for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:36:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58102) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIm-00060w-1i for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:59684) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIk-00087Y-0N for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OJKA3ZuhCRHrDkdUwuwnHQea0qchPw+I2YBp1BnrWnc=; b=Ylnij/jV8uMFj08BVw5/Cp9zGJEXusVRfxWI2ZcCuzBEWFIc9lstNByVb1Gav5whfSEOw/ OOz2OMk/p4zp048jnpBb+sMhjPNgMLNpMoXYMNMP2ahzZtOgonlKR9kl3gzIPpIq6i0+hR c4lUytqOJuLC+SogyFFA0y4BFuzwteU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-103-S6BQvOI9N5yE1BQHJGae-Q-1; Mon, 27 Sep 2021 15:28:30 -0400 X-MC-Unique: S6BQvOI9N5yE1BQHJGae-Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8DB411084683; Mon, 27 Sep 2021 19:28:29 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id E11C760871; Mon, 27 Sep 2021 19:28:27 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 17/32] python/aqmp: add QMP protocol support Date: Mon, 27 Sep 2021 15:24:58 -0400 Message-Id: <20210927192513.744199-18-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The star of our show! Add most of the QMP protocol, sans support for actually executing commands. No problem, that happens in the next several commits. Signed-off-by: John Snow Message-id: 20210915162955.333025-18-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 2 + python/qemu/aqmp/qmp_client.py | 264 +++++++++++++++++++++++++++++++++ 2 files changed, 266 insertions(+) create mode 100644 python/qemu/aqmp/qmp_client.py diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 829166a2e2e..d975c752eaa 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -25,11 +25,13 @@ from .events import EventListener from .message import Message from .protocol import ConnectError, Runstate, StateError +from .qmp_client import QMPClient # The order of these fields impact the Sphinx documentation order. __all__ = ( # Classes, most to least important + 'QMPClient', 'Message', 'EventListener', 'Runstate', diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py new file mode 100644 index 00000000000..000ff59c7a7 --- /dev/null +++ b/python/qemu/aqmp/qmp_client.py @@ -0,0 +1,264 @@ +""" +QMP Protocol Implementation + +This module provides the `QMPClient` class, which can be used to connect +and send commands to a QMP server such as QEMU. The QMP class can be +used to either connect to a listening server, or used to listen and +accept an incoming connection from that server. +""" + +import logging +from typing import ( + Dict, + List, + Mapping, + Optional, +) + +from .error import ProtocolError +from .events import Events +from .message import Message +from .models import Greeting +from .protocol import AsyncProtocol +from .util import ( + bottom_half, + exception_summary, + pretty_traceback, + upper_half, +) + + +class _WrappedProtocolError(ProtocolError): + """ + Abstract exception class for Protocol errors that wrap an Exception. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + def __init__(self, error_message: str, exc: Exception): + super().__init__(error_message) + self.exc = exc + + def __str__(self) -> str: + return f"{self.error_message}: {self.exc!s}" + + +class GreetingError(_WrappedProtocolError): + """ + An exception occurred during the Greeting phase. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + + +class NegotiationError(_WrappedProtocolError): + """ + An exception occurred during the Negotiation phase. + + :param error_message: Human-readable string describing the error. + :param exc: The root-cause exception. + """ + + +class QMPClient(AsyncProtocol[Message], Events): + """ + Implements a QMP client connection. + + QMP can be used to establish a connection as either the transport + client or server, though this class always acts as the QMP client. + + :param name: Optional nickname for the connection, used for logging. + + Basic script-style usage looks like this:: + + qmp = QMPClient('my_virtual_machine_name') + await qmp.connect(('127.0.0.1', 1234)) + ... + res = await qmp.execute('block-query') + ... + await qmp.disconnect() + + Basic async client-style usage looks like this:: + + class Client: + def __init__(self, name: str): + self.qmp = QMPClient(name) + + async def watch_events(self): + try: + async for event in self.qmp.events: + print(f"Event: {event['event']}") + except asyncio.CancelledError: + return + + async def run(self, address='/tmp/qemu.socket'): + await self.qmp.connect(address) + asyncio.create_task(self.watch_events()) + await self.qmp.runstate_changed.wait() + await self.disconnect() + + See `aqmp.events` for more detail on event handling patterns. + """ + #: Logger object used for debugging messages. + logger = logging.getLogger(__name__) + + # Read buffer limit; large enough to accept query-qmp-schema + _limit = (256 * 1024) + + def __init__(self, name: Optional[str] = None) -> None: + super().__init__(name) + Events.__init__(self) + + #: Whether or not to await a greeting after establishing a connection. + self.await_greeting: bool = True + + #: Whether or not to perform capabilities negotiation upon connection. + #: Implies `await_greeting`. + self.negotiate: bool = True + + # Cached Greeting, if one was awaited. + self._greeting: Optional[Greeting] = None + + @upper_half + async def _establish_session(self) -> None: + """ + Initiate the QMP session. + + Wait for the QMP greeting and perform capabilities negotiation. + + :raise GreetingError: When the greeting is not understood. + :raise NegotiationError: If the negotiation fails. + :raise EOFError: When the server unexpectedly hangs up. + :raise OSError: For underlying stream errors. + """ + if self.await_greeting or self.negotiate: + self._greeting = await self._get_greeting() + + if self.negotiate: + await self._negotiate() + + # This will start the reader/writers: + await super()._establish_session() + + @upper_half + async def _get_greeting(self) -> Greeting: + """ + :raise GreetingError: When the greeting is not understood. + :raise EOFError: When the server unexpectedly hangs up. + :raise OSError: For underlying stream errors. + + :return: the Greeting object given by the server. + """ + self.logger.debug("Awaiting greeting ...") + + try: + msg = await self._recv() + return Greeting(msg) + except (ProtocolError, KeyError, TypeError) as err: + emsg = "Did not understand Greeting" + self.logger.error("%s: %s", emsg, exception_summary(err)) + self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) + raise GreetingError(emsg, err) from err + except BaseException as err: + # EOFError, OSError, or something unexpected. + emsg = "Failed to receive Greeting" + self.logger.error("%s: %s", emsg, exception_summary(err)) + self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) + raise + + @upper_half + async def _negotiate(self) -> None: + """ + Perform QMP capabilities negotiation. + + :raise NegotiationError: When negotiation fails. + :raise EOFError: When the server unexpectedly hangs up. + :raise OSError: For underlying stream errors. + """ + self.logger.debug("Negotiating capabilities ...") + + arguments: Dict[str, List[str]] = {'enable': []} + if self._greeting and 'oob' in self._greeting.QMP.capabilities: + arguments['enable'].append('oob') + msg = self.make_execute_msg('qmp_capabilities', arguments=arguments) + + # It's not safe to use execute() here, because the reader/writers + # aren't running. AsyncProtocol *requires* that a new session + # does not fail after the reader/writers are running! + try: + await self._send(msg) + reply = await self._recv() + assert 'return' in reply + assert 'error' not in reply + except (ProtocolError, AssertionError) as err: + emsg = "Negotiation failed" + self.logger.error("%s: %s", emsg, exception_summary(err)) + self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) + raise NegotiationError(emsg, err) from err + except BaseException as err: + # EOFError, OSError, or something unexpected. + emsg = "Negotiation failed" + self.logger.error("%s: %s", emsg, exception_summary(err)) + self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) + raise + + @bottom_half + async def _on_message(self, msg: Message) -> None: + """ + Add an incoming message to the appropriate queue/handler. + """ + # Incoming messages are not fully parsed/validated here; + # do only light peeking to know how to route the messages. + + if 'event' in msg: + await self._event_dispatch(msg) + return + + # Below, we assume everything left is an execute/exec-oob response. + # ... Which we'll implement in the next commit! + + @upper_half + @bottom_half + async def _do_recv(self) -> Message: + """ + :raise OSError: When a stream error is encountered. + :raise EOFError: When the stream is at EOF. + :raise ProtocolError: + When the Message is not understood. + See also `Message._deserialize`. + + :return: A single QMP `Message`. + """ + msg_bytes = await self._readline() + msg = Message(msg_bytes, eager=True) + return msg + + @upper_half + @bottom_half + def _do_send(self, msg: Message) -> None: + """ + :raise ValueError: JSON serialization failure + :raise TypeError: JSON serialization failure + :raise OSError: When a stream error is encountered. + """ + assert self._writer is not None + self._writer.write(bytes(msg)) + + @classmethod + def make_execute_msg(cls, cmd: str, + arguments: Optional[Mapping[str, object]] = None, + oob: bool = False) -> Message: + """ + Create an executable message to be sent later. + + :param cmd: QMP command name. + :param arguments: Arguments (if any). Must be JSON-serializable. + :param oob: If `True`, execute "out of band". + + :return: An executable QMP `Message`. + """ + msg = Message({'exec-oob' if oob else 'execute': cmd}) + if arguments is not None: + msg['arguments'] = arguments + return msg From patchwork Mon Sep 27 19:24:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFA54C433EF for ; Mon, 27 Sep 2021 19:46:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A16A360F41 for ; Mon, 27 Sep 2021 19:46:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A16A360F41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:54944 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwa5-0002DD-RQ for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:46:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58114) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIt-0006Ch-Qs for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:32489) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIs-0008EI-8F for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770921; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rUoPYIzG/K+ir8/fQg+fP0VAwCeYPwlf6dbwf+D4qY=; b=APmwlsDjPw2BcwGCECMmNDH0vRoY8kwc+VfV9G7B+huMWBbZaoG5xrMZxewxMgFo3HuJi8 mSqXf7sYfvZot4THFpcwc+G/Re9MlY7eCDWM5poy1uXjUb0iEXYuoaTx6vLV1WXphxK8qO Hg3Dg8o5E+lUhSqaQrPkI9SarMyY3kg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-17--pZwmIdZMbCMDdKuZaYmlw-1; Mon, 27 Sep 2021 15:28:39 -0400 X-MC-Unique: -pZwmIdZMbCMDdKuZaYmlw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7897F1006AA4; Mon, 27 Sep 2021 19:28:38 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id BB2C160871; Mon, 27 Sep 2021 19:28:29 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 18/32] python/pylint: disable no-member check Date: Mon, 27 Sep 2021 15:24:59 -0400 Message-Id: <20210927192513.744199-19-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" mypy handles this better -- but we only need the workaround because pylint under Python 3.6 does not understand that a MutableMapping really does have a .get() method attached. We could remove this again once 3.7 is our minimum. Signed-off-by: John Snow Message-id: 20210915162955.333025-19-jsnow@redhat.com Signed-off-by: John Snow --- python/setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/python/setup.cfg b/python/setup.cfg index 70957ab7e4d..e6407e05826 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -90,6 +90,7 @@ ignore_missing_imports = True # --disable=W". disable=consider-using-f-string, too-many-function-args, # mypy handles this with less false positives. + no-member, # mypy also handles this better. [pylint.basic] # Good variable names which should always be accepted, separated by a comma. From patchwork Mon Sep 27 19:25:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19677C433EF for ; Mon, 27 Sep 2021 19:50:51 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ACB8560F94 for ; Mon, 27 Sep 2021 19:50:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org ACB8560F94 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35030 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUweH-00083s-SG for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:50:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58126) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIz-0006LB-W5 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:26114) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwIx-0008Ia-Rn for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770927; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qkf5xhJPquVa5AMzqBKiyRqw1RCG9ek0TF794R0YezY=; b=i7wWfnGClwMI/APBSo5hP/RX71C/oHhlbgbdyVRqHLX+W/PD9tqlI1KQ0szJQrkyFOBIT3 LjJSeJJgoZ3F11qmGuRTQUoNvfOIxK9D/mbcK0Eb32SFbNvQ9KxZQfisI2WLJrLaLokiCG 5chUs8GtneO+Ud7KZECtuRv4e0LYwbg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-229--XI1x1uNPNK-uBmNOGmCEQ-1; Mon, 27 Sep 2021 15:28:45 -0400 X-MC-Unique: -XI1x1uNPNK-uBmNOGmCEQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9D69A1084681; Mon, 27 Sep 2021 19:28:44 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id A669B60871; Mon, 27 Sep 2021 19:28:38 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 19/32] python/aqmp: Add message routing to QMP protocol Date: Mon, 27 Sep 2021 15:25:00 -0400 Message-Id: <20210927192513.744199-20-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add the ability to handle and route messages in qmp_protocol.py. The interface for actually sending anything still isn't added until next commit. Signed-off-by: John Snow Message-id: 20210915162955.333025-20-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 122 ++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 2 deletions(-) diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py index 000ff59c7a7..fa0cc7c5ae5 100644 --- a/python/qemu/aqmp/qmp_client.py +++ b/python/qemu/aqmp/qmp_client.py @@ -7,15 +7,19 @@ accept an incoming connection from that server. """ +# The import workarounds here are fixed in the next commit. +import asyncio # pylint: disable=unused-import # noqa import logging from typing import ( Dict, List, Mapping, Optional, + Union, + cast, ) -from .error import ProtocolError +from .error import AQMPError, ProtocolError from .events import Events from .message import Message from .models import Greeting @@ -61,6 +65,53 @@ class NegotiationError(_WrappedProtocolError): """ +class ExecInterruptedError(AQMPError): + """ + Exception raised when an RPC is interrupted. + + This error is raised when an execute() statement could not be + completed. This can occur because the connection itself was + terminated before a reply was received. + + The true cause of the interruption will be available via `disconnect()`. + """ + + +class _MsgProtocolError(ProtocolError): + """ + Abstract error class for protocol errors that have a `Message` object. + + This Exception class is used for protocol errors where the `Message` + was mechanically understood, but was found to be inappropriate or + malformed. + + :param error_message: Human-readable string describing the error. + :param msg: The QMP `Message` that caused the error. + """ + def __init__(self, error_message: str, msg: Message): + super().__init__(error_message) + #: The received `Message` that caused the error. + self.msg: Message = msg + + def __str__(self) -> str: + return "\n".join([ + super().__str__(), + f" Message was: {str(self.msg)}\n", + ]) + + +class ServerParseError(_MsgProtocolError): + """ + The Server sent a `Message` indicating parsing failure. + + i.e. A reply has arrived from the server, but it is missing the "ID" + field, indicating a parsing error. + + :param error_message: Human-readable string describing the error. + :param msg: The QMP `Message` that caused the error. + """ + + class QMPClient(AsyncProtocol[Message], Events): """ Implements a QMP client connection. @@ -106,6 +157,9 @@ async def run(self, address='/tmp/qemu.socket'): # Read buffer limit; large enough to accept query-qmp-schema _limit = (256 * 1024) + # Type alias for pending execute() result items + _PendingT = Union[Message, ExecInterruptedError] + def __init__(self, name: Optional[str] = None) -> None: super().__init__(name) Events.__init__(self) @@ -120,6 +174,12 @@ def __init__(self, name: Optional[str] = None) -> None: # Cached Greeting, if one was awaited. self._greeting: Optional[Greeting] = None + # Incoming RPC reply messages. + self._pending: Dict[ + Union[str, None], + 'asyncio.Queue[QMPClient._PendingT]' + ] = {} + @upper_half async def _establish_session(self) -> None: """ @@ -132,6 +192,9 @@ async def _establish_session(self) -> None: :raise EOFError: When the server unexpectedly hangs up. :raise OSError: For underlying stream errors. """ + self._greeting = None + self._pending = {} + if self.await_greeting or self.negotiate: self._greeting = await self._get_greeting() @@ -203,10 +266,33 @@ async def _negotiate(self) -> None: self.logger.debug("%s:\n%s\n", emsg, pretty_traceback()) raise + @bottom_half + async def _bh_disconnect(self) -> None: + try: + await super()._bh_disconnect() + finally: + if self._pending: + self.logger.debug("Cancelling pending executions") + keys = self._pending.keys() + for key in keys: + self.logger.debug("Cancelling execution '%s'", key) + self._pending[key].put_nowait( + ExecInterruptedError("Disconnected") + ) + + self.logger.debug("QMP Disconnected.") + + @upper_half + def _cleanup(self) -> None: + super()._cleanup() + assert not self._pending + @bottom_half async def _on_message(self, msg: Message) -> None: """ Add an incoming message to the appropriate queue/handler. + + :raise ServerParseError: When Message indicates server parse failure. """ # Incoming messages are not fully parsed/validated here; # do only light peeking to know how to route the messages. @@ -216,7 +302,39 @@ async def _on_message(self, msg: Message) -> None: return # Below, we assume everything left is an execute/exec-oob response. - # ... Which we'll implement in the next commit! + + exec_id = cast(Optional[str], msg.get('id')) + + if exec_id in self._pending: + await self._pending[exec_id].put(msg) + return + + # We have a message we can't route back to a caller. + + is_error = 'error' in msg + has_id = 'id' in msg + + if is_error and not has_id: + # This is very likely a server parsing error. + # It doesn't inherently belong to any pending execution. + # Instead of performing clever recovery, just terminate. + # See "NOTE" in qmp-spec.txt, section 2.4.2 + raise ServerParseError( + ("Server sent an error response without an ID, " + "but there are no ID-less executions pending. " + "Assuming this is a server parser failure."), + msg + ) + + # qmp-spec.txt, section 2.4: + # 'Clients should drop all the responses + # that have an unknown "id" field.' + self.logger.log( + logging.ERROR if is_error else logging.WARNING, + "Unknown ID '%s', message dropped.", + exec_id, + ) + self.logger.debug("Unroutable message: %s", str(msg)) @upper_half @bottom_half From patchwork Mon Sep 27 19:25:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85BF6C433EF for ; Mon, 27 Sep 2021 19:48:30 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0987F60F94 for ; Mon, 27 Sep 2021 19:48:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0987F60F94 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:59372 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwc0-0005LA-WF for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:48:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58162) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJ6-0006UF-BO for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52073) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJ3-0008Lm-7d for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W6qRcWayYtbFZ1XErwlWGn15q7qsOLnZBhyJXS1hkNM=; b=Sk9wkOUF1fao8n61+0R9tjS6xWqZ3KBsLG6d+7ua1OHHN+moCAS5CYGOM0urkB0PAZ48yA DULbwOS/ClZRMzo7ZIAeuPjjVV5pFd97viclHJpnGpGwcbeE2HoftoYgpXM1ewjVyLEerJ yEYt3q4VcvnW1sOd65Xe+4r9LziRLNg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-588-s1f5zal1N_ewJxmCt3uDDQ-1; Mon, 27 Sep 2021 15:28:47 -0400 X-MC-Unique: s1f5zal1N_ewJxmCt3uDDQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5BAD719200C0; Mon, 27 Sep 2021 19:28:46 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id CB12460871; Mon, 27 Sep 2021 19:28:44 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 20/32] python/aqmp: add execute() interfaces Date: Mon, 27 Sep 2021 15:25:01 -0400 Message-Id: <20210927192513.744199-21-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add execute() and execute_msg(). _execute() is split into _issue() and _reply() halves so that hypothetical subclasses of QMP that want to support different execution paradigms can do so. I anticipate a synchronous interface may have need of separating the send/reply phases. However, I do not wish to expose that interface here and want to actively discourage it, so they remain private interfaces. Signed-off-by: John Snow Message-id: 20210915162955.333025-21-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 4 +- python/qemu/aqmp/qmp_client.py | 202 +++++++++++++++++++++++++++++++-- 2 files changed, 198 insertions(+), 8 deletions(-) diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index d975c752eaa..4b7df53e006 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -25,7 +25,7 @@ from .events import EventListener from .message import Message from .protocol import ConnectError, Runstate, StateError -from .qmp_client import QMPClient +from .qmp_client import ExecInterruptedError, ExecuteError, QMPClient # The order of these fields impact the Sphinx documentation order. @@ -40,4 +40,6 @@ 'AQMPError', 'StateError', 'ConnectError', + 'ExecuteError', + 'ExecInterruptedError', ) diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py index fa0cc7c5ae5..879348feaaa 100644 --- a/python/qemu/aqmp/qmp_client.py +++ b/python/qemu/aqmp/qmp_client.py @@ -7,8 +7,7 @@ accept an incoming connection from that server. """ -# The import workarounds here are fixed in the next commit. -import asyncio # pylint: disable=unused-import # noqa +import asyncio import logging from typing import ( Dict, @@ -22,8 +21,8 @@ from .error import AQMPError, ProtocolError from .events import Events from .message import Message -from .models import Greeting -from .protocol import AsyncProtocol +from .models import ErrorResponse, Greeting +from .protocol import AsyncProtocol, Runstate, require from .util import ( bottom_half, exception_summary, @@ -65,11 +64,32 @@ class NegotiationError(_WrappedProtocolError): """ +class ExecuteError(AQMPError): + """ + Exception raised by `QMPClient.execute()` on RPC failure. + + :param error_response: The RPC error response object. + :param sent: The sent RPC message that caused the failure. + :param received: The raw RPC error reply received. + """ + def __init__(self, error_response: ErrorResponse, + sent: Message, received: Message): + super().__init__(error_response.error.desc) + #: The sent `Message` that caused the failure + self.sent: Message = sent + #: The received `Message` that indicated failure + self.received: Message = received + #: The parsed error response + self.error: ErrorResponse = error_response + #: The QMP error class + self.error_class: str = error_response.error.class_ + + class ExecInterruptedError(AQMPError): """ - Exception raised when an RPC is interrupted. + Exception raised by `execute()` (et al) when an RPC is interrupted. - This error is raised when an execute() statement could not be + This error is raised when an `execute()` statement could not be completed. This can occur because the connection itself was terminated before a reply was received. @@ -112,6 +132,27 @@ class ServerParseError(_MsgProtocolError): """ +class BadReplyError(_MsgProtocolError): + """ + An execution reply was successfully routed, but not understood. + + If a QMP message is received with an 'id' field to allow it to be + routed, but is otherwise malformed, this exception will be raised. + + A reply message is malformed if it is missing either the 'return' or + 'error' keys, or if the 'error' value has missing keys or members of + the wrong type. + + :param error_message: Human-readable string describing the error. + :param msg: The malformed reply that was received. + :param sent: The message that was sent that prompted the error. + """ + def __init__(self, error_message: str, msg: Message, sent: Message): + super().__init__(error_message, msg) + #: The sent `Message` that caused the failure + self.sent = sent + + class QMPClient(AsyncProtocol[Message], Events): """ Implements a QMP client connection. @@ -174,6 +215,9 @@ def __init__(self, name: Optional[str] = None) -> None: # Cached Greeting, if one was awaited. self._greeting: Optional[Greeting] = None + # Command ID counter + self._execute_id = 0 + # Incoming RPC reply messages. self._pending: Dict[ Union[str, None], @@ -363,12 +407,135 @@ def _do_send(self, msg: Message) -> None: assert self._writer is not None self._writer.write(bytes(msg)) + @upper_half + def _get_exec_id(self) -> str: + exec_id = f"__aqmp#{self._execute_id:05d}" + self._execute_id += 1 + return exec_id + + @upper_half + async def _issue(self, msg: Message) -> Union[None, str]: + """ + Issue a QMP `Message` and do not wait for a reply. + + :param msg: The QMP `Message` to send to the server. + + :return: The ID of the `Message` sent. + """ + msg_id: Optional[str] = None + if 'id' in msg: + assert isinstance(msg['id'], str) + msg_id = msg['id'] + + self._pending[msg_id] = asyncio.Queue(maxsize=1) + await self._outgoing.put(msg) + + return msg_id + + @upper_half + async def _reply(self, msg_id: Union[str, None]) -> Message: + """ + Await a reply to a previously issued QMP message. + + :param msg_id: The ID of the previously issued message. + + :return: The reply from the server. + :raise ExecInterruptedError: + When the reply could not be retrieved because the connection + was lost, or some other problem. + """ + queue = self._pending[msg_id] + result = await queue.get() + + try: + if isinstance(result, ExecInterruptedError): + raise result + return result + finally: + del self._pending[msg_id] + + @upper_half + async def _execute(self, msg: Message, assign_id: bool = True) -> Message: + """ + Send a QMP `Message` to the server and await a reply. + + This method *assumes* you are sending some kind of an execute + statement that *will* receive a reply. + + An execution ID will be assigned if assign_id is `True`. It can be + disabled, but this requires that an ID is manually assigned + instead. For manually assigned IDs, you must not use the string + '__aqmp#' anywhere in the ID. + + :param msg: The QMP `Message` to execute. + :param assign_id: If True, assign a new execution ID. + + :return: Execution reply from the server. + :raise ExecInterruptedError: + When the reply could not be retrieved because the connection + was lost, or some other problem. + """ + if assign_id: + msg['id'] = self._get_exec_id() + elif 'id' in msg: + assert isinstance(msg['id'], str) + assert '__aqmp#' not in msg['id'] + + exec_id = await self._issue(msg) + return await self._reply(exec_id) + + @upper_half + @require(Runstate.RUNNING) + async def execute_msg(self, msg: Message) -> object: + """ + Execute a QMP command and return its value. + + :param msg: The QMP `Message` to execute. + + :return: + The command execution return value from the server. The type of + object returned depends on the command that was issued, + though most in QEMU return a `dict`. + :raise ValueError: + If the QMP `Message` does not have either the 'execute' or + 'exec-oob' fields set. + :raise ExecuteError: When the server returns an error response. + :raise ExecInterruptedError: if the connection was terminated early. + """ + if not ('execute' in msg or 'exec-oob' in msg): + raise ValueError("Requires 'execute' or 'exec-oob' message") + + # Copy the Message so that the ID assigned by _execute() is + # local to this method; allowing the ID to be seen in raised + # Exceptions but without modifying the caller's held copy. + msg = Message(msg) + reply = await self._execute(msg) + + if 'error' in reply: + try: + error_response = ErrorResponse(reply) + except (KeyError, TypeError) as err: + # Error response was malformed. + raise BadReplyError( + "QMP error reply is malformed", reply, msg, + ) from err + + raise ExecuteError(error_response, msg, reply) + + if 'return' not in reply: + raise BadReplyError( + "QMP reply is missing a 'error' or 'return' member", + reply, msg, + ) + + return reply['return'] + @classmethod def make_execute_msg(cls, cmd: str, arguments: Optional[Mapping[str, object]] = None, oob: bool = False) -> Message: """ - Create an executable message to be sent later. + Create an executable message to be sent by `execute_msg` later. :param cmd: QMP command name. :param arguments: Arguments (if any). Must be JSON-serializable. @@ -380,3 +547,24 @@ def make_execute_msg(cls, cmd: str, if arguments is not None: msg['arguments'] = arguments return msg + + @upper_half + async def execute(self, cmd: str, + arguments: Optional[Mapping[str, object]] = None, + oob: bool = False) -> object: + """ + Execute a QMP command and return its value. + + :param cmd: QMP command name. + :param arguments: Arguments (if any). Must be JSON-serializable. + :param oob: If `True`, execute "out of band". + + :return: + The command execution return value from the server. The type of + object returned depends on the command that was issued, + though most in QEMU return a `dict`. + :raise ExecuteError: When the server returns an error response. + :raise ExecInterruptedError: if the connection was terminated early. + """ + msg = self.make_execute_msg(cmd, arguments, oob=oob) + return await self.execute_msg(msg) From patchwork Mon Sep 27 19:25:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BAE1C433EF for ; Mon, 27 Sep 2021 19:55:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0257660F91 for ; Mon, 27 Sep 2021 19:55:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0257660F91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:45814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwiS-00077E-72 for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:55:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58142) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJ3-0006Ql-HK for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:37889) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJ1-0008L9-7X for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770930; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WT3vLnSnqr9P/xvfw3vqStlB+9kau9jv/JfY+fYXk1I=; b=cjEgshF1gKLCxidWllkLaf2XvX7/b4FoTlXvK29GF16lfB6yhBSxDnPGpMzlTlAuxDjkIB Fxic5gNy6cQ9TlFGKe1o2u5W3er7mAWk9li76TLSnQ4R/R99sqIgK/DxHXTkPiKaqMlqMu /0aqE2ArwwDDwPyPbqRA3gvQxzLuobI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-586-9Ox25qgnOEGEG_iq4nDqQw-1; Mon, 27 Sep 2021 15:28:49 -0400 X-MC-Unique: 9Ox25qgnOEGEG_iq4nDqQw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1FA811006AA3; Mon, 27 Sep 2021 19:28:48 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id A37A360871; Mon, 27 Sep 2021 19:28:46 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 21/32] python/aqmp: add _raw() execution interface Date: Mon, 27 Sep 2021 15:25:02 -0400 Message-Id: <20210927192513.744199-22-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is added in anticipation of wanting it for a synchronous wrapper for the iotest interface. Normally, execute() and execute_msg() both raise QMP errors in the form of Python exceptions. Many iotests expect the entire reply as-is. To reduce churn there, add a private execution interface that will ease transition churn. However, I do not wish to encourage its use, so it will remain a private interface. Signed-off-by: John Snow Message-id: 20210915162955.333025-22-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/qmp_client.py | 51 ++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py index 879348feaaa..82e9dab124c 100644 --- a/python/qemu/aqmp/qmp_client.py +++ b/python/qemu/aqmp/qmp_client.py @@ -484,6 +484,57 @@ async def _execute(self, msg: Message, assign_id: bool = True) -> Message: exec_id = await self._issue(msg) return await self._reply(exec_id) + @upper_half + @require(Runstate.RUNNING) + async def _raw( + self, + msg: Union[Message, Mapping[str, object], bytes], + assign_id: bool = True, + ) -> Message: + """ + Issue a raw `Message` to the QMP server and await a reply. + + :param msg: + A Message to send to the server. It may be a `Message`, any + Mapping (including Dict), or raw bytes. + :param assign_id: + Assign an arbitrary execution ID to this message. If + `False`, the existing id must either be absent (and no other + such pending execution may omit an ID) or a string. If it is + a string, it must not start with '__aqmp#' and no other such + pending execution may currently be using that ID. + + :return: Execution reply from the server. + + :raise ExecInterruptedError: + When the reply could not be retrieved because the connection + was lost, or some other problem. + :raise TypeError: + When assign_id is `False`, an ID is given, and it is not a string. + :raise ValueError: + When assign_id is `False`, but the ID is not usable; + Either because it starts with '__aqmp#' or it is already in-use. + """ + # 1. convert generic Mapping or bytes to a QMP Message + # 2. copy Message objects so that we assign an ID only to the copy. + msg = Message(msg) + + exec_id = msg.get('id') + if not assign_id and 'id' in msg: + if not isinstance(exec_id, str): + raise TypeError(f"ID ('{exec_id}') must be a string.") + if exec_id.startswith('__aqmp#'): + raise ValueError( + f"ID ('{exec_id}') must not start with '__aqmp#'." + ) + + if not assign_id and exec_id in self._pending: + raise ValueError( + f"ID '{exec_id}' is in-use and cannot be used." + ) + + return await self._execute(msg, assign_id=assign_id) + @upper_half @require(Runstate.RUNNING) async def execute_msg(self, msg: Message) -> object: From patchwork Mon Sep 27 19:25:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95388C433EF for ; Mon, 27 Sep 2021 19:53:08 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6180260184 for ; Mon, 27 Sep 2021 19:53:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6180260184 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:39486 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwgV-0002p3-H2 for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:53:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58164) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJ6-0006UG-GO for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:57 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:48267) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJ4-0008Nc-32 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:28:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770933; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nLc6Y2eQJVeshW7PrOw7x9O65co80u6A/9pzHHuT3NE=; b=aOr1PLcgcLjbOI1GKat7pkQH9X32v9075L3/hvBUROC/BYOoihwdzglA9Nc2yTeF4TQsg7 JRvlqwkpB6jz/JBhSUy0Jo7gE9JyIV4mIPH0sMdlgJSgsYnJVkfRJ8WVePSYmgL56XuV5G 8TYkCBsJbb3Jq9MIWf4nlY+bIgJeVzY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-287-0edRXwmCNYe6FlhV2IDoOw-1; Mon, 27 Sep 2021 15:28:51 -0400 X-MC-Unique: 0edRXwmCNYe6FlhV2IDoOw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 28AE319200C0; Mon, 27 Sep 2021 19:28:50 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 62E5B60871; Mon, 27 Sep 2021 19:28:48 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 22/32] python/aqmp: add asyncio_run compatibility wrapper Date: Mon, 27 Sep 2021 15:25:03 -0400 Message-Id: <20210927192513.744199-23-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" As a convenience. It isn't used by the library itself, but it is used by the test suite. It will also come in handy for users of the library still on Python 3.6. Signed-off-by: John Snow Message-id: 20210915162955.333025-23-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/util.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/python/qemu/aqmp/util.py b/python/qemu/aqmp/util.py index 52a15321889..eaa5fc7d5f9 100644 --- a/python/qemu/aqmp/util.py +++ b/python/qemu/aqmp/util.py @@ -147,6 +147,25 @@ async def wait_closed(writer: asyncio.StreamWriter) -> None: await asyncio.sleep(0) +def asyncio_run(coro: Coroutine[Any, Any, T], *, debug: bool = False) -> T: + """ + Python 3.6-compatible `asyncio.run` wrapper. + + :param coro: A coroutine to execute now. + :return: The return value from the coroutine. + """ + if sys.version_info >= (3, 7): + return asyncio.run(coro, debug=debug) + + # Python 3.6 + loop = asyncio.get_event_loop() + loop.set_debug(debug) + ret = loop.run_until_complete(coro) + loop.close() + + return ret + + # ---------------------------- # Section: Logging & Debugging # ---------------------------- From patchwork Mon Sep 27 19:25:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3E70C433F5 for ; Mon, 27 Sep 2021 19:59:02 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8D8DC60E8B for ; Mon, 27 Sep 2021 19:59:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8D8DC60E8B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52668 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwmD-0003sN-Lj for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:59:01 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58216) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJH-0006jF-6J for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27350) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJF-00005i-Lp for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o7I90tWbhIB6RinLdxGw1//Z/hHBZaCrc6zocPnEgK8=; b=QyUkYBTB+iM9fiszGZRutEpmEabwN/npSCYJRdNtds7A92P0UKT74mkiDJvM0XkcIPR1Zg iuisSL+WXhovaKKRVweGh2mjSeJrUis54iMiHIcK/34k/f0xuA0WLSUEOXNpXWMOE/22OS cxei+FwR3++AEcOcLLXC1JBSdj2dtGc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-284-TgSlIDg7MJmLy4OONhF1Iw-1; Mon, 27 Sep 2021 15:29:03 -0400 X-MC-Unique: TgSlIDg7MJmLy4OONhF1Iw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 561A9835DE1; Mon, 27 Sep 2021 19:29:02 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 566BE60871; Mon, 27 Sep 2021 19:28:50 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 23/32] python/aqmp: add scary message Date: Mon, 27 Sep 2021 15:25:04 -0400 Message-Id: <20210927192513.744199-24-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add a warning whenever AQMP is used to steer people gently away from using it for the time-being. Signed-off-by: John Snow Message-id: 20210915162955.333025-24-jsnow@redhat.com Signed-off-by: John Snow --- python/qemu/aqmp/__init__.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py index 4b7df53e006..ab1782999cf 100644 --- a/python/qemu/aqmp/__init__.py +++ b/python/qemu/aqmp/__init__.py @@ -21,6 +21,8 @@ # This work is licensed under the terms of the GNU GPL, version 2. See # the COPYING file in the top-level directory. +import warnings + from .error import AQMPError from .events import EventListener from .message import Message @@ -28,6 +30,18 @@ from .qmp_client import ExecInterruptedError, ExecuteError, QMPClient +_WMSG = """ + +The Asynchronous QMP library is currently in development and its API +should be considered highly fluid and subject to change. It should +not be used by any other scripts checked into the QEMU tree. + +Proceed with caution! +""" + +warnings.warn(_WMSG, FutureWarning) + + # The order of these fields impact the Sphinx documentation order. __all__ = ( # Classes, most to least important From patchwork Mon Sep 27 19:25:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1687C433F5 for ; Mon, 27 Sep 2021 19:38:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 828B860F58 for ; Mon, 27 Sep 2021 19:38:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 828B860F58 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34482 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwSj-0005Hi-OL for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:38:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58256) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJL-0006yB-Vo for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:34988) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJK-0000AM-2K for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770949; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YsHPEFDQowmgcGgRX9YirwiXeDC8kwKET7qNafqw4AE=; b=IFSmaqOejUBHKd5eW1O442/d+/StN51sY/pOxcq1bUZOomW1D9qz0U1ZoE9uBT3rHAyWIl WBaRVEvvyb1PfCktEXJ1F4C5k1s5ZxRNF4nGholihLyr6MZMWt6rIuxag0MaynQyDyZPxB wqUJr4zTCJMXhtkWzqzaK2/32/wb1Pg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-534-VQVftcA3P3mhz7pMvRT8Lg-1; Mon, 27 Sep 2021 15:29:04 -0400 X-MC-Unique: VQVftcA3P3mhz7pMvRT8Lg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EE51819200C0; Mon, 27 Sep 2021 19:29:03 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8253C60877; Mon, 27 Sep 2021 19:29:02 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 24/32] python: bump avocado to v90.0 Date: Mon, 27 Sep 2021 15:25:05 -0400 Message-Id: <20210927192513.744199-25-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Avocado v90 includes improved support for running async unit tests. The workaround that existed prior to v90 causes the unit tests to fail afterwards, however, so upgrade our minimum version pin to the very latest and greatest. Signed-off-by: John Snow Message-id: 20210915162955.333025-25-jsnow@redhat.com Signed-off-by: John Snow --- python/Pipfile.lock | 8 ++++---- python/setup.cfg | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/python/Pipfile.lock b/python/Pipfile.lock index 8ab41a3f606..457f5c3fe87 100644 --- a/python/Pipfile.lock +++ b/python/Pipfile.lock @@ -1,7 +1,7 @@ { "_meta": { "hash": { - "sha256": "eff562a688ebc6f3ffe67494dbb804b883e2159ad81c4d55d96da9f7aec13e91" + "sha256": "784b327272db32403d5a488507853b5afba850ba26a5948e5b6a90c1baef2d9c" }, "pipfile-spec": 6, "requires": { @@ -39,11 +39,11 @@ }, "avocado-framework": { "hashes": [ - "sha256:3fca7226d7d164f124af8a741e7fa658ff4345a0738ddc32907631fd688b38ed", - "sha256:48ac254c0ae2ef0c0ceeb38e3d3df0388718eda8f48b3ab55b30b252839f42b1" + "sha256:244cb569f8eb4e50a22ac82e1a2b2bba2458999f4281efbe2651bd415d59c65b", + "sha256:6f15998b67ecd0e7dde790c4de4dd249d6df52dfe6d5cc4e2dd6596df51c3583" ], "index": "pypi", - "version": "==87.0" + "version": "==90.0" }, "distlib": { "hashes": [ diff --git a/python/setup.cfg b/python/setup.cfg index e6407e05826..8481fa7c923 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -37,7 +37,7 @@ packages = # version, use e.g. "pipenv install --dev pylint==3.0.0". # Subsequently, edit 'Pipfile' to remove e.g. 'pylint = "==3.0.0'. devel = - avocado-framework >= 87.0 + avocado-framework >= 90.0 flake8 >= 3.6.0 fusepy >= 2.0.4 isort >= 5.1.2 From patchwork Mon Sep 27 19:25:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E4CAC433F5 for ; Mon, 27 Sep 2021 19:46:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 27CA660F41 for ; Mon, 27 Sep 2021 19:46:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 27CA660F41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:54826 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwa4-000283-5N for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:46:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58282) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJN-000729-74 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:28841) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJK-0000AW-Fv for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770949; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z5EgV74/+US7NxAI1iBPcSZhl6xcNFAbPHDsW9ud6MI=; b=CvdzgrsJFVwneODc7TuRfWtQAyaB9OYdo1Zh3tYGb1HOiWR4pdgJ9+h41Tf3Kw7xXY9EeX 9ob9yWhvOD4NwxsHdm4aRWerUeYXuore/zZulASSHGF2JJwqt3wBHvDqNrRd3kM/QxyIf1 +AdI2OKDUT2gB5cFStbQ+U4oLRew1F0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-48-lYqDckSmPAqmJiEMZUHm0A-1; Mon, 27 Sep 2021 15:29:07 -0400 X-MC-Unique: lYqDckSmPAqmJiEMZUHm0A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0A8CD1006AA6; Mon, 27 Sep 2021 19:29:06 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 29F8060871; Mon, 27 Sep 2021 19:29:04 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 25/32] python/aqmp: add AsyncProtocol unit tests Date: Mon, 27 Sep 2021 15:25:06 -0400 Message-Id: <20210927192513.744199-26-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This tests most of protocol.py -- From a hacked up Coverage.py run, it's at about 86%. There's a few error cases that aren't very well tested yet, they're hard to induce artificially so far. I'm working on it. Signed-off-by: John Snow Message-id: 20210915162955.333025-26-jsnow@redhat.com Signed-off-by: John Snow --- python/tests/protocol.py | 535 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 535 insertions(+) create mode 100644 python/tests/protocol.py diff --git a/python/tests/protocol.py b/python/tests/protocol.py new file mode 100644 index 00000000000..f0682d29ce5 --- /dev/null +++ b/python/tests/protocol.py @@ -0,0 +1,535 @@ +import asyncio +from contextlib import contextmanager +import os +import socket +from tempfile import TemporaryDirectory + +import avocado + +from qemu.aqmp import ConnectError, Runstate +from qemu.aqmp.protocol import AsyncProtocol, StateError +from qemu.aqmp.util import asyncio_run, create_task + + +class NullProtocol(AsyncProtocol[None]): + """ + NullProtocol is a test mockup of an AsyncProtocol implementation. + + It adds a fake_session instance variable that enables a code path + that bypasses the actual connection logic, but still allows the + reader/writers to start. + + Because the message type is defined as None, an asyncio.Event named + 'trigger_input' is created that prohibits the reader from + incessantly being able to yield None; this event can be poked to + simulate an incoming message. + + For testing symmetry with do_recv, an interface is added to "send" a + Null message. + + For testing purposes, a "simulate_disconnection" method is also + added which allows us to trigger a bottom half disconnect without + injecting any real errors into the reader/writer loops; in essence + it performs exactly half of what disconnect() normally does. + """ + def __init__(self, name=None): + self.fake_session = False + self.trigger_input: asyncio.Event + super().__init__(name) + + async def _establish_session(self): + self.trigger_input = asyncio.Event() + await super()._establish_session() + + async def _do_accept(self, address, ssl=None): + if not self.fake_session: + await super()._do_accept(address, ssl) + + async def _do_connect(self, address, ssl=None): + if not self.fake_session: + await super()._do_connect(address, ssl) + + async def _do_recv(self) -> None: + await self.trigger_input.wait() + self.trigger_input.clear() + + def _do_send(self, msg: None) -> None: + pass + + async def send_msg(self) -> None: + await self._outgoing.put(None) + + async def simulate_disconnect(self) -> None: + """ + Simulates a bottom-half disconnect. + + This method schedules a disconnection but does not wait for it + to complete. This is used to put the loop into the DISCONNECTING + state without fully quiescing it back to IDLE. This is normally + something you cannot coax AsyncProtocol to do on purpose, but it + will be similar to what happens with an unhandled Exception in + the reader/writer. + + Under normal circumstances, the library design requires you to + await on disconnect(), which awaits the disconnect task and + returns bottom half errors as a pre-condition to allowing the + loop to return back to IDLE. + """ + self._schedule_disconnect() + + +def run_as_task(coro, allow_cancellation=False): + """ + Run a given coroutine as a task. + + Optionally, wrap it in a try..except block that allows this + coroutine to be canceled gracefully. + """ + async def _runner(): + try: + await coro + except asyncio.CancelledError: + if allow_cancellation: + return + raise + return create_task(_runner()) + + +@contextmanager +def jammed_socket(): + """ + Opens up a random unused TCP port on localhost, then jams it. + """ + socks = [] + + try: + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + sock.bind(('127.0.0.1', 0)) + sock.listen(1) + address = sock.getsockname() + + socks.append(sock) + + # I don't *fully* understand why, but it takes *two* un-accepted + # connections to start jamming the socket. + for _ in range(2): + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.connect(address) + socks.append(sock) + + yield address + + finally: + for sock in socks: + sock.close() + + +class Smoke(avocado.Test): + + def setUp(self): + self.proto = NullProtocol() + + def test__repr__(self): + self.assertEqual( + repr(self.proto), + "" + ) + + def testRunstate(self): + self.assertEqual( + self.proto.runstate, + Runstate.IDLE + ) + + def testDefaultName(self): + self.assertEqual( + self.proto.name, + None + ) + + def testLogger(self): + self.assertEqual( + self.proto.logger.name, + 'qemu.aqmp.protocol' + ) + + def testName(self): + self.proto = NullProtocol('Steve') + + self.assertEqual( + self.proto.name, + 'Steve' + ) + + self.assertEqual( + self.proto.logger.name, + 'qemu.aqmp.protocol.Steve' + ) + + self.assertEqual( + repr(self.proto), + "" + ) + + +class TestBase(avocado.Test): + + def setUp(self): + self.proto = NullProtocol(type(self).__name__) + self.assertEqual(self.proto.runstate, Runstate.IDLE) + self.runstate_watcher = None + + def tearDown(self): + self.assertEqual(self.proto.runstate, Runstate.IDLE) + + async def _asyncSetUp(self): + pass + + async def _asyncTearDown(self): + if self.runstate_watcher: + await self.runstate_watcher + + @staticmethod + def async_test(async_test_method): + """ + Decorator; adds SetUp and TearDown to async tests. + """ + async def _wrapper(self, *args, **kwargs): + loop = asyncio.get_event_loop() + loop.set_debug(True) + + await self._asyncSetUp() + await async_test_method(self, *args, **kwargs) + await self._asyncTearDown() + + return _wrapper + + # Definitions + + # The states we expect a "bad" connect/accept attempt to transition through + BAD_CONNECTION_STATES = ( + Runstate.CONNECTING, + Runstate.DISCONNECTING, + Runstate.IDLE, + ) + + # The states we expect a "good" session to transition through + GOOD_CONNECTION_STATES = ( + Runstate.CONNECTING, + Runstate.RUNNING, + Runstate.DISCONNECTING, + Runstate.IDLE, + ) + + # Helpers + + async def _watch_runstates(self, *states): + """ + This launches a task alongside (most) tests below to confirm that + the sequence of runstate changes that occur is exactly as + anticipated. + """ + async def _watcher(): + for state in states: + new_state = await self.proto.runstate_changed() + self.assertEqual( + new_state, + state, + msg=f"Expected state '{state.name}'", + ) + + self.runstate_watcher = create_task(_watcher()) + # Kick the loop and force the task to block on the event. + await asyncio.sleep(0) + + +class State(TestBase): + + @TestBase.async_test + async def testSuperfluousDisconnect(self): + """ + Test calling disconnect() while already disconnected. + """ + await self._watch_runstates( + Runstate.DISCONNECTING, + Runstate.IDLE, + ) + await self.proto.disconnect() + + +class Connect(TestBase): + """ + Tests primarily related to calling Connect(). + """ + async def _bad_connection(self, family: str): + assert family in ('INET', 'UNIX') + + if family == 'INET': + await self.proto.connect(('127.0.0.1', 0)) + elif family == 'UNIX': + await self.proto.connect('/dev/null') + + async def _hanging_connection(self): + with jammed_socket() as addr: + await self.proto.connect(addr) + + async def _bad_connection_test(self, family: str): + await self._watch_runstates(*self.BAD_CONNECTION_STATES) + + with self.assertRaises(ConnectError) as context: + await self._bad_connection(family) + + self.assertIsInstance(context.exception.exc, OSError) + self.assertEqual( + context.exception.error_message, + "Failed to establish connection" + ) + + @TestBase.async_test + async def testBadINET(self): + """ + Test an immediately rejected call to an IP target. + """ + await self._bad_connection_test('INET') + + @TestBase.async_test + async def testBadUNIX(self): + """ + Test an immediately rejected call to a UNIX socket target. + """ + await self._bad_connection_test('UNIX') + + @TestBase.async_test + async def testCancellation(self): + """ + Test what happens when a connection attempt is aborted. + """ + # Note that accept() cannot be cancelled outright, as it isn't a task. + # However, we can wrap it in a task and cancel *that*. + await self._watch_runstates(*self.BAD_CONNECTION_STATES) + task = run_as_task(self._hanging_connection(), allow_cancellation=True) + + state = await self.proto.runstate_changed() + self.assertEqual(state, Runstate.CONNECTING) + + # This is insider baseball, but the connection attempt has + # yielded *just* before the actual connection attempt, so kick + # the loop to make sure it's truly wedged. + await asyncio.sleep(0) + + task.cancel() + await task + + @TestBase.async_test + async def testTimeout(self): + """ + Test what happens when a connection attempt times out. + """ + await self._watch_runstates(*self.BAD_CONNECTION_STATES) + task = run_as_task(self._hanging_connection()) + + # More insider baseball: to improve the speed of this test while + # guaranteeing that the connection even gets a chance to start, + # verify that the connection hangs *first*, then await the + # result of the task with a nearly-zero timeout. + + state = await self.proto.runstate_changed() + self.assertEqual(state, Runstate.CONNECTING) + await asyncio.sleep(0) + + with self.assertRaises(asyncio.TimeoutError): + await asyncio.wait_for(task, timeout=0) + + @TestBase.async_test + async def testRequire(self): + """ + Test what happens when a connection attempt is made while CONNECTING. + """ + await self._watch_runstates(*self.BAD_CONNECTION_STATES) + task = run_as_task(self._hanging_connection(), allow_cancellation=True) + + state = await self.proto.runstate_changed() + self.assertEqual(state, Runstate.CONNECTING) + + with self.assertRaises(StateError) as context: + await self._bad_connection('UNIX') + + self.assertEqual( + context.exception.error_message, + "NullProtocol is currently connecting." + ) + self.assertEqual(context.exception.state, Runstate.CONNECTING) + self.assertEqual(context.exception.required, Runstate.IDLE) + + task.cancel() + await task + + @TestBase.async_test + async def testImplicitRunstateInit(self): + """ + Test what happens if we do not wait on the runstate event until + AFTER a connection is made, i.e., connect()/accept() themselves + initialize the runstate event. All of the above tests force the + initialization by waiting on the runstate *first*. + """ + task = run_as_task(self._hanging_connection(), allow_cancellation=True) + + # Kick the loop to coerce the state change + await asyncio.sleep(0) + assert self.proto.runstate == Runstate.CONNECTING + + # We already missed the transition to CONNECTING + await self._watch_runstates(Runstate.DISCONNECTING, Runstate.IDLE) + + task.cancel() + await task + + +class Accept(Connect): + """ + All of the same tests as Connect, but using the accept() interface. + """ + async def _bad_connection(self, family: str): + assert family in ('INET', 'UNIX') + + if family == 'INET': + await self.proto.accept(('example.com', 1)) + elif family == 'UNIX': + await self.proto.accept('/dev/null') + + async def _hanging_connection(self): + with TemporaryDirectory(suffix='.aqmp') as tmpdir: + sock = os.path.join(tmpdir, type(self.proto).__name__ + ".sock") + await self.proto.accept(sock) + + +class FakeSession(TestBase): + + def setUp(self): + super().setUp() + self.proto.fake_session = True + + async def _asyncSetUp(self): + await super()._asyncSetUp() + await self._watch_runstates(*self.GOOD_CONNECTION_STATES) + + async def _asyncTearDown(self): + await self.proto.disconnect() + await super()._asyncTearDown() + + #### + + @TestBase.async_test + async def testFakeConnect(self): + + """Test the full state lifecycle (via connect) with a no-op session.""" + await self.proto.connect('/not/a/real/path') + self.assertEqual(self.proto.runstate, Runstate.RUNNING) + + @TestBase.async_test + async def testFakeAccept(self): + """Test the full state lifecycle (via accept) with a no-op session.""" + await self.proto.accept('/not/a/real/path') + self.assertEqual(self.proto.runstate, Runstate.RUNNING) + + @TestBase.async_test + async def testFakeRecv(self): + """Test receiving a fake/null message.""" + await self.proto.accept('/not/a/real/path') + + logname = self.proto.logger.name + with self.assertLogs(logname, level='DEBUG') as context: + self.proto.trigger_input.set() + self.proto.trigger_input.clear() + await asyncio.sleep(0) # Kick reader. + + self.assertEqual( + context.output, + [f"DEBUG:{logname}:<-- None"], + ) + + @TestBase.async_test + async def testFakeSend(self): + """Test sending a fake/null message.""" + await self.proto.accept('/not/a/real/path') + + logname = self.proto.logger.name + with self.assertLogs(logname, level='DEBUG') as context: + # Cheat: Send a Null message to nobody. + await self.proto.send_msg() + # Kick writer; awaiting on a queue.put isn't sufficient to yield. + await asyncio.sleep(0) + + self.assertEqual( + context.output, + [f"DEBUG:{logname}:--> None"], + ) + + async def _prod_session_api( + self, + current_state: Runstate, + error_message: str, + accept: bool = True + ): + with self.assertRaises(StateError) as context: + if accept: + await self.proto.accept('/not/a/real/path') + else: + await self.proto.connect('/not/a/real/path') + + self.assertEqual(context.exception.error_message, error_message) + self.assertEqual(context.exception.state, current_state) + self.assertEqual(context.exception.required, Runstate.IDLE) + + @TestBase.async_test + async def testAcceptRequireRunning(self): + """Test that accept() cannot be called when Runstate=RUNNING""" + await self.proto.accept('/not/a/real/path') + + await self._prod_session_api( + Runstate.RUNNING, + "NullProtocol is already connected and running.", + accept=True, + ) + + @TestBase.async_test + async def testConnectRequireRunning(self): + """Test that connect() cannot be called when Runstate=RUNNING""" + await self.proto.accept('/not/a/real/path') + + await self._prod_session_api( + Runstate.RUNNING, + "NullProtocol is already connected and running.", + accept=False, + ) + + @TestBase.async_test + async def testAcceptRequireDisconnecting(self): + """Test that accept() cannot be called when Runstate=DISCONNECTING""" + await self.proto.accept('/not/a/real/path') + + # Cheat: force a disconnect. + await self.proto.simulate_disconnect() + + await self._prod_session_api( + Runstate.DISCONNECTING, + ("NullProtocol is disconnecting." + " Call disconnect() to return to IDLE state."), + accept=True, + ) + + @TestBase.async_test + async def testConnectRequireDisconnecting(self): + """Test that connect() cannot be called when Runstate=DISCONNECTING""" + await self.proto.accept('/not/a/real/path') + + # Cheat: force a disconnect. + await self.proto.simulate_disconnect() + + await self._prod_session_api( + Runstate.DISCONNECTING, + ("NullProtocol is disconnecting." + " Call disconnect() to return to IDLE state."), + accept=False, + ) From patchwork Mon Sep 27 19:25:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECC17C433F5 for ; Mon, 27 Sep 2021 20:07:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8240260FED for ; Mon, 27 Sep 2021 20:07:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8240260FED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:41022 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwum-0007Mp-ET for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 16:07:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58284) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJN-00072c-D9 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:21662) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJK-0000Ae-Tx for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770950; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3MiVMx32MtyXm7sd64YRGsQoeddO99I4bKnm+a2JB04=; b=St+awBAYeAOjqAMoAQyUYncmribINVFOHMWTg+V8cRdD2KvhujyaCgAUukvusRsO+ZKyGf UsPsJSP5rnJVRnQH0a9lR8T52GbkYgFyLc7bvuYVsD6WXoJYXGQm/iHROL3sdQGZdPAmDT rK9p8Wvu8tSdsCS7HS8o0mP5ZJQ47A4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-142-PIYQp_uUOGSTdpLaYPm4wQ-1; Mon, 27 Sep 2021 15:29:08 -0400 X-MC-Unique: PIYQp_uUOGSTdpLaYPm4wQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BB251362F8; Mon, 27 Sep 2021 19:29:07 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3CA5360871; Mon, 27 Sep 2021 19:29:06 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 26/32] python/aqmp: add LineProtocol tests Date: Mon, 27 Sep 2021 15:25:07 -0400 Message-Id: <20210927192513.744199-27-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Tests a real connect, a real accept, and really sending and receiving a message over a UNIX socket. Brings coverage of protocol.py up to ~93%. Signed-off-by: John Snow Message-id: 20210915162955.333025-27-jsnow@redhat.com Signed-off-by: John Snow --- python/tests/protocol.py | 48 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/python/tests/protocol.py b/python/tests/protocol.py index f0682d29ce5..5cd7938be35 100644 --- a/python/tests/protocol.py +++ b/python/tests/protocol.py @@ -78,6 +78,25 @@ async def simulate_disconnect(self) -> None: self._schedule_disconnect() +class LineProtocol(AsyncProtocol[str]): + def __init__(self, name=None): + super().__init__(name) + self.rx_history = [] + + async def _do_recv(self) -> str: + raw = await self._readline() + msg = raw.decode() + self.rx_history.append(msg) + return msg + + def _do_send(self, msg: str) -> None: + assert self._writer is not None + self._writer.write(msg.encode() + b'\n') + + async def send_msg(self, msg: str) -> None: + await self._outgoing.put(msg) + + def run_as_task(coro, allow_cancellation=False): """ Run a given coroutine as a task. @@ -533,3 +552,32 @@ async def testConnectRequireDisconnecting(self): " Call disconnect() to return to IDLE state."), accept=False, ) + + +class SimpleSession(TestBase): + + def setUp(self): + super().setUp() + self.server = LineProtocol(type(self).__name__ + '-server') + + async def _asyncSetUp(self): + await super()._asyncSetUp() + await self._watch_runstates(*self.GOOD_CONNECTION_STATES) + + async def _asyncTearDown(self): + await self.proto.disconnect() + try: + await self.server.disconnect() + except EOFError: + pass + await super()._asyncTearDown() + + @TestBase.async_test + async def testSmoke(self): + with TemporaryDirectory(suffix='.aqmp') as tmpdir: + sock = os.path.join(tmpdir, type(self.proto).__name__ + ".sock") + server_task = create_task(self.server.accept(sock)) + + # give the server a chance to start listening [...] + await asyncio.sleep(0) + await self.proto.connect(sock) From patchwork Mon Sep 27 19:25:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E55C433F5 for ; Mon, 27 Sep 2021 19:50:32 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 29FBD60F94 for ; Mon, 27 Sep 2021 19:50:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 29FBD60F94 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34762 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwdz-0007t1-0v for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:50:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58298) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJP-0007BW-Px for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:60120) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJN-0000Dd-Sb for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770953; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ifIsu/ICCbj/chY4OpnCe5zr18vtEXwudS721wHZnXk=; b=UQmUSWl6wEqHwPsDCXmtv4fcwhdIEhyseU2qJMxArxgu0KOqoJP72Ta2jsxqnGOBZGrH5F ZjJSb+MkJub10jKky+Q0gSgXaAy+xALbH0aTP/RLtVcQmC3q/b4uy+flTxOAlCk1zP89Hs bzO5vsBBfk72WYahA4qKMJq6Wgpt5+Y= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-513-cff4gzXfNUOnl7Q2T9Fz_Q-1; Mon, 27 Sep 2021 15:29:11 -0400 X-MC-Unique: cff4gzXfNUOnl7Q2T9Fz_Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E2B881084681; Mon, 27 Sep 2021 19:29:09 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5CAA160871; Mon, 27 Sep 2021 19:29:07 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 27/32] python/aqmp: Add Coverage.py support Date: Mon, 27 Sep 2021 15:25:08 -0400 Message-Id: <20210927192513.744199-28-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" I'm not exposing this via the Makefile help, it's not likely to be useful to passersby. Switch the avocado runner to the 'legacy' runner for now, as the new runner seems to obscure coverage reports, again. Usage is to enter your venv of choice and then: `make check-coverage && xdg-open htmlcov/index.html`. Signed-off-by: John Snow Message-id: 20210915162955.333025-28-jsnow@redhat.com Signed-off-by: John Snow --- python/.gitignore | 5 +++++ python/Makefile | 9 +++++++++ python/avocado.cfg | 3 +++ python/setup.cfg | 10 ++++++++++ 4 files changed, 27 insertions(+) diff --git a/python/.gitignore b/python/.gitignore index c8b0e67fe6c..904f324bb11 100644 --- a/python/.gitignore +++ b/python/.gitignore @@ -15,3 +15,8 @@ qemu.egg-info/ .venv/ .tox/ .dev-venv/ + +# Coverage.py reports +.coverage +.coverage.* +htmlcov/ diff --git a/python/Makefile b/python/Makefile index fe27a3e12ee..33343113625 100644 --- a/python/Makefile +++ b/python/Makefile @@ -92,6 +92,13 @@ check: check-tox: @tox $(QEMU_TOX_EXTRA_ARGS) +.PHONY: check-coverage +check-coverage: + @coverage run -m avocado --config avocado.cfg run tests/*.py + @coverage combine + @coverage html + @coverage report + .PHONY: clean clean: python3 setup.py clean --all @@ -100,3 +107,5 @@ clean: .PHONY: distclean distclean: clean rm -rf qemu.egg-info/ .venv/ .tox/ $(QEMU_VENV_DIR) dist/ + rm -f .coverage .coverage.* + rm -rf htmlcov/ diff --git a/python/avocado.cfg b/python/avocado.cfg index 10dc6fb6054..c7722e7ecd3 100644 --- a/python/avocado.cfg +++ b/python/avocado.cfg @@ -1,3 +1,6 @@ +[run] +test_runner = runner + [simpletests] # Don't show stdout/stderr in the test *summary* status.failure_fields = ['status'] diff --git a/python/setup.cfg b/python/setup.cfg index 8481fa7c923..4fd0c68a5b7 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -140,3 +140,13 @@ deps = .[fuse] # Workaround to trigger tox venv rebuild commands = make check + +# Coverage.py [https://coverage.readthedocs.io/en/latest/] is a tool for +# measuring code coverage of Python programs. It monitors your program, +# noting which parts of the code have been executed, then analyzes the +# source to identify code that could have been executed but was not. + +[coverage:run] +concurrency = multiprocessing +source = qemu/ +parallel = true From patchwork Mon Sep 27 19:25:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08710C433EF for ; Mon, 27 Sep 2021 19:55:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A099360F91 for ; Mon, 27 Sep 2021 19:55:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A099360F91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:46868 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwj9-0007pR-MJ for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:55:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58332) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJW-0007Vp-7c for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:35876) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJT-0000If-F1 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770958; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4mlMzaJfi5MEIHOdYJ37R8z1gqYD18vgI/setqcrYoI=; b=goCOqK7R05mhlaaQEMCfviN1TFJBNI7glLLw6IpLl73XRykdL8avt80ewwxD7S4nMzOhCB pWC659F3sbTGXTSoCOwYkfkcROQDZssjXYKHTVEwWb+/+TaBSYnIKcWdX4NumOA3VmYTVI NFZIRHNhm6/UxowDdTXGPWLMLUVkIOo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-35-CaqXCWyhOu-iU0AP6NDaGQ-1; Mon, 27 Sep 2021 15:29:17 -0400 X-MC-Unique: CaqXCWyhOu-iU0AP6NDaGQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2F3CD1084681; Mon, 27 Sep 2021 19:29:16 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1B77D60871; Mon, 27 Sep 2021 19:29:10 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 28/32] python: Add dependencies for AQMP TUI Date: Mon, 27 Sep 2021 15:25:09 -0400 Message-Id: <20210927192513.744199-29-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , G S Niteesh Babu , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: G S Niteesh Babu Added dependencies for the upcoming AQMP TUI under the optional 'tui' group. The same dependencies have also been added under the devel group since no work around has been found for optional groups to imply other optional groups. Signed-off-by: G S Niteesh Babu Message-Id: <20210823220746.28295-2-niteesh.gs@gmail.com> Signed-off-by: John Snow --- python/Pipfile.lock | 12 ++++++++++++ python/setup.cfg | 8 ++++++++ 2 files changed, 20 insertions(+) diff --git a/python/Pipfile.lock b/python/Pipfile.lock index 457f5c3fe87..da7a4ee164c 100644 --- a/python/Pipfile.lock +++ b/python/Pipfile.lock @@ -289,6 +289,18 @@ "markers": "python_version < '3.8'", "version": "==3.10.0.0" }, + "urwid": { + "hashes": [ + "sha256:588bee9c1cb208d0906a9f73c613d2bd32c3ed3702012f51efe318a3f2127eae" + ], + "version": "==2.1.2" + }, + "urwid-readline": { + "hashes": [ + "sha256:018020cbc864bb5ed87be17dc26b069eae2755cb29f3a9c569aac3bded1efaf4" + ], + "version": "==0.13" + }, "virtualenv": { "hashes": [ "sha256:14fdf849f80dbb29a4eb6caa9875d476ee2a5cf76a5f5415fa2f1606010ab467", diff --git a/python/setup.cfg b/python/setup.cfg index 4fd0c68a5b7..efcda23c48d 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -44,11 +44,18 @@ devel = mypy >= 0.770 pylint >= 2.8.0 tox >= 3.18.0 + urwid >= 2.1.2 + urwid-readline >= 0.13 # Provides qom-fuse functionality fuse = fusepy >= 2.0.4 +# AQMP TUI dependencies +tui = + urwid >= 2.1.2 + urwid-readline >= 0.13 + [options.entry_points] console_scripts = qom = qemu.qmp.qom:main @@ -138,6 +145,7 @@ allowlist_externals = make deps = .[devel] .[fuse] # Workaround to trigger tox venv rebuild + .[tui] # Workaround to trigger tox venv rebuild commands = make check From patchwork Mon Sep 27 19:25:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B55BEC4332F for ; Mon, 27 Sep 2021 19:54:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 215B860FD7 for ; Mon, 27 Sep 2021 19:54:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 215B860FD7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:43132 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwhw-0005Gj-78 for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:54:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58354) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJf-00084G-SI for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:40845) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJc-0000R2-PB for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770968; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oF+RGKsZhfNWIBidPD5HBCWE2F/IddZxMCgrWOqOrkY=; b=Zxi7cvi+1WlNOuW7D0yADmGdmIbd0Z6zavkMvKRe/cb/Gb14/IU861UCtdFS1MsV34x7f9 h+3vKiLWUaDwsq9Uup8l+S+MAXPfQOw68YtlOQlTmVWuBCwi3FbSMlEG3r8VJWZY2Bkywu ToHcWVs8HqKrXF3tY+4S8XI/52UYSmM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-453-58Oc66asNbahPOPk9Prwbg-1; Mon, 27 Sep 2021 15:29:25 -0400 X-MC-Unique: 58Oc66asNbahPOPk9Prwbg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B9F2A19200C8; Mon, 27 Sep 2021 19:29:23 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5F0BB60871; Mon, 27 Sep 2021 19:29:16 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 29/32] python/aqmp-tui: Add AQMP TUI Date: Mon, 27 Sep 2021 15:25:10 -0400 Message-Id: <20210927192513.744199-30-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , G S Niteesh Babu , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: G S Niteesh Babu Added AQMP TUI. Implements the follwing basic features: 1) Command transmission/reception. 2) Shows events asynchronously. 3) Shows server status in the bottom status bar. 4) Automatic retries on disconnects and error conditions. Also added type annotations and necessary pylint/mypy configurations. Signed-off-by: G S Niteesh Babu Message-Id: <20210823220746.28295-3-niteesh.gs@gmail.com> Signed-off-by: John Snow --- python/qemu/aqmp/aqmp_tui.py | 620 +++++++++++++++++++++++++++++++++++ python/setup.cfg | 13 +- 2 files changed, 632 insertions(+), 1 deletion(-) create mode 100644 python/qemu/aqmp/aqmp_tui.py diff --git a/python/qemu/aqmp/aqmp_tui.py b/python/qemu/aqmp/aqmp_tui.py new file mode 100644 index 00000000000..ac533541d2f --- /dev/null +++ b/python/qemu/aqmp/aqmp_tui.py @@ -0,0 +1,620 @@ +# Copyright (c) 2021 +# +# Authors: +# Niteesh Babu G S +# +# This work is licensed under the terms of the GNU GPL, version 2 or +# later. See the COPYING file in the top-level directory. +""" +AQMP TUI + +AQMP TUI is an asynchronous interface built on top the of the AQMP library. +It is the successor of QMP-shell and is bought-in as a replacement for it. + +Example Usage: aqmp-tui +Full Usage: aqmp-tui --help +""" + +import argparse +import asyncio +import json +import logging +from logging import Handler, LogRecord +import signal +from typing import ( + List, + Optional, + Tuple, + Type, + Union, + cast, +) + +import urwid +import urwid_readline + +from ..qmp import QEMUMonitorProtocol, QMPBadPortError +from .error import ProtocolError +from .message import DeserializationError, Message, UnexpectedTypeError +from .protocol import ConnectError, Runstate +from .qmp_client import ExecInterruptedError, QMPClient +from .util import create_task, pretty_traceback + + +# The name of the signal that is used to update the history list +UPDATE_MSG: str = 'UPDATE_MSG' + + +def format_json(msg: str) -> str: + """ + Formats valid/invalid multi-line JSON message into a single-line message. + + Formatting is first tried using the standard json module. If that fails + due to an decoding error then a simple string manipulation is done to + achieve a single line JSON string. + + Converting into single line is more asthetically pleasing when looking + along with error messages. + + Eg: + Input: + [ 1, + true, + 3 ] + The above input is not a valid QMP message and produces the following error + "QMP message is not a JSON object." + When displaying this in TUI in multiline mode we get + + [ 1, + true, + 3 ]: QMP message is not a JSON object. + + whereas in singleline mode we get the following + + [1, true, 3]: QMP message is not a JSON object. + + The single line mode is more asthetically pleasing. + + :param msg: + The message to formatted into single line. + + :return: Formatted singleline message. + """ + try: + msg = json.loads(msg) + return str(json.dumps(msg)) + except json.decoder.JSONDecodeError: + msg = msg.replace('\n', '') + words = msg.split(' ') + words = list(filter(None, words)) + return ' '.join(words) + + +def has_handler_type(logger: logging.Logger, + handler_type: Type[Handler]) -> bool: + """ + The Logger class has no interface to check if a certain type of handler is + installed or not. So we provide an interface to do so. + + :param logger: + Logger object + :param handler_type: + The type of the handler to be checked. + + :return: returns True if handler of type `handler_type`. + """ + for handler in logger.handlers: + if isinstance(handler, handler_type): + return True + return False + + +class App(QMPClient): + """ + Implements the AQMP TUI. + + Initializes the widgets and starts the urwid event loop. + + :param address: + Address of the server to connect to. + :param num_retries: + The number of times to retry before stopping to reconnect. + :param retry_delay: + The delay(sec) before each retry + """ + def __init__(self, address: Union[str, Tuple[str, int]], num_retries: int, + retry_delay: Optional[int]) -> None: + urwid.register_signal(type(self), UPDATE_MSG) + self.window = Window(self) + self.address = address + self.aloop: Optional[asyncio.AbstractEventLoop] = None + self.num_retries = num_retries + self.retry_delay = retry_delay if retry_delay else 2 + self.retry: bool = False + self.exiting: bool = False + super().__init__() + + def add_to_history(self, msg: str, level: Optional[str] = None) -> None: + """ + Appends the msg to the history list. + + :param msg: + The raw message to be appended in string type. + """ + urwid.emit_signal(self, UPDATE_MSG, msg, level) + + def _cb_outbound(self, msg: Message) -> Message: + """ + Callback: outbound message hook. + + Appends the outgoing messages to the history box. + + :param msg: raw outbound message. + :return: final outbound message. + """ + str_msg = str(msg) + + if not has_handler_type(logging.getLogger(), TUILogHandler): + logging.debug('Request: %s', str_msg) + self.add_to_history('<-- ' + str_msg) + return msg + + def _cb_inbound(self, msg: Message) -> Message: + """ + Callback: outbound message hook. + + Appends the incoming messages to the history box. + + :param msg: raw inbound message. + :return: final inbound message. + """ + str_msg = str(msg) + + if not has_handler_type(logging.getLogger(), TUILogHandler): + logging.debug('Request: %s', str_msg) + self.add_to_history('--> ' + str_msg) + return msg + + async def _send_to_server(self, msg: Message) -> None: + """ + This coroutine sends the message to the server. + The message has to be pre-validated. + + :param msg: + Pre-validated message to be to sent to the server. + + :raise Exception: When an unhandled exception is caught. + """ + try: + await self._raw(msg, assign_id='id' not in msg) + except ExecInterruptedError as err: + logging.info('Error server disconnected before reply %s', str(err)) + self.add_to_history('Server disconnected before reply', 'ERROR') + except Exception as err: + logging.error('Exception from _send_to_server: %s', str(err)) + raise err + + def cb_send_to_server(self, raw_msg: str) -> None: + """ + Validates and sends the message to the server. + The raw string message is first converted into a Message object + and is then sent to the server. + + :param raw_msg: + The raw string message to be sent to the server. + + :raise Exception: When an unhandled exception is caught. + """ + try: + msg = Message(bytes(raw_msg, encoding='utf-8')) + create_task(self._send_to_server(msg)) + except (DeserializationError, UnexpectedTypeError) as err: + raw_msg = format_json(raw_msg) + logging.info('Invalid message: %s', err.error_message) + self.add_to_history(f'{raw_msg}: {err.error_message}', 'ERROR') + + def unhandled_input(self, key: str) -> None: + """ + Handle's keys which haven't been handled by the child widgets. + + :param key: + Unhandled key + """ + if key == 'esc': + self.kill_app() + + def kill_app(self) -> None: + """ + Initiates killing of app. A bridge between asynchronous and synchronous + code. + """ + create_task(self._kill_app()) + + async def _kill_app(self) -> None: + """ + This coroutine initiates the actual disconnect process and calls + urwid.ExitMainLoop() to kill the TUI. + + :raise Exception: When an unhandled exception is caught. + """ + self.exiting = True + await self.disconnect() + logging.debug('Disconnect finished. Exiting app') + raise urwid.ExitMainLoop() + + async def disconnect(self) -> None: + """ + Overrides the disconnect method to handle the errors locally. + """ + try: + await super().disconnect() + except (OSError, EOFError) as err: + logging.info('disconnect: %s', str(err)) + self.retry = True + except ProtocolError as err: + logging.info('disconnect: %s', str(err)) + except Exception as err: + logging.error('disconnect: Unhandled exception %s', str(err)) + raise err + + def _set_status(self, msg: str) -> None: + """ + Sets the message as the status. + + :param msg: + The message to be displayed in the status bar. + """ + self.window.footer.set_text(msg) + + def _get_formatted_address(self) -> str: + """ + Returns a formatted version of the server's address. + + :return: formatted address + """ + if isinstance(self.address, tuple): + host, port = self.address + addr = f'{host}:{port}' + else: + addr = f'{self.address}' + return addr + + async def _initiate_connection(self) -> Optional[ConnectError]: + """ + Tries connecting to a server a number of times with a delay between + each try. If all retries failed then return the error faced during + the last retry. + + :return: Error faced during last retry. + """ + current_retries = 0 + err = None + + # initial try + await self.connect_server() + while self.retry and current_retries < self.num_retries: + logging.info('Connection Failed, retrying in %d', self.retry_delay) + status = f'[Retry #{current_retries} ({self.retry_delay}s)]' + self._set_status(status) + + await asyncio.sleep(self.retry_delay) + + err = await self.connect_server() + current_retries += 1 + # If all retries failed report the last error + if err: + logging.info('All retries failed: %s', err) + return err + return None + + async def manage_connection(self) -> None: + """ + Manage the connection based on the current run state. + + A reconnect is issued when the current state is IDLE and the number + of retries is not exhausted. + A disconnect is issued when the current state is DISCONNECTING. + """ + while not self.exiting: + if self.runstate == Runstate.IDLE: + err = await self._initiate_connection() + # If retry is still true then, we have exhausted all our tries. + if err: + self._set_status(f'[Error: {err.error_message}]') + else: + addr = self._get_formatted_address() + self._set_status(f'[Connected {addr}]') + elif self.runstate == Runstate.DISCONNECTING: + self._set_status('[Disconnected]') + await self.disconnect() + # check if a retry is needed + if self.runstate == Runstate.IDLE: + continue + await self.runstate_changed() + + async def connect_server(self) -> Optional[ConnectError]: + """ + Initiates a connection to the server at address `self.address` + and in case of a failure, sets the status to the respective error. + """ + try: + await self.connect(self.address) + self.retry = False + except ConnectError as err: + logging.info('connect_server: ConnectError %s', str(err)) + self.retry = True + return err + return None + + def run(self, debug: bool = False) -> None: + """ + Starts the long running co-routines and the urwid event loop. + + :param debug: + Enables/Disables asyncio event loop debugging + """ + self.aloop = asyncio.get_event_loop() + self.aloop.set_debug(debug) + + # Gracefully handle SIGTERM and SIGINT signals + cancel_signals = [signal.SIGTERM, signal.SIGINT] + for sig in cancel_signals: + self.aloop.add_signal_handler(sig, self.kill_app) + + event_loop = urwid.AsyncioEventLoop(loop=self.aloop) + main_loop = urwid.MainLoop(urwid.AttrMap(self.window, 'background'), + unhandled_input=self.unhandled_input, + handle_mouse=True, + event_loop=event_loop) + + create_task(self.manage_connection(), self.aloop) + try: + main_loop.run() + except Exception as err: + logging.error('%s\n%s\n', str(err), pretty_traceback()) + raise err + + +class StatusBar(urwid.Text): + """ + A simple statusbar modelled using the Text widget. The status can be + set using the set_text function. All text set is aligned to right. + + :param text: Initial text to be displayed. Default is empty str. + """ + def __init__(self, text: str = ''): + super().__init__(text, align='right') + + +class Editor(urwid_readline.ReadlineEdit): + """ + A simple editor modelled using the urwid_readline.ReadlineEdit widget. + Mimcs GNU readline shortcuts and provides history support. + + The readline shortcuts can be found below: + https://github.com/rr-/urwid_readline#features + + Along with the readline features, this editor also has support for + history. Pressing the 'up'/'down' switches between the prev/next messages + available in the history. + + Currently there is no support to save the history to a file. The history of + previous commands is lost on exit. + + :param parent: Reference to the TUI object. + """ + def __init__(self, parent: App) -> None: + super().__init__(caption='> ', multiline=True) + self.parent = parent + self.history: List[str] = [] + self.last_index: int = -1 + self.show_history: bool = False + + def keypress(self, size: Tuple[int, int], key: str) -> Optional[str]: + """ + Handles the keypress on this widget. + + :param size: + The current size of the widget. + :param key: + The key to be handled. + + :return: Unhandled key if any. + """ + msg = self.get_edit_text() + if key == 'up' and not msg: + # Show the history when 'up arrow' is pressed with no input text. + # NOTE: The show_history logic is necessary because in 'multiline' + # mode (which we use) 'up arrow' is used to move between lines. + if not self.history: + return None + self.show_history = True + last_msg = self.history[self.last_index] + self.set_edit_text(last_msg) + self.edit_pos = len(last_msg) + elif key == 'up' and self.show_history: + self.last_index = max(self.last_index - 1, -len(self.history)) + self.set_edit_text(self.history[self.last_index]) + self.edit_pos = len(self.history[self.last_index]) + elif key == 'down' and self.show_history: + if self.last_index == -1: + self.set_edit_text('') + self.show_history = False + else: + self.last_index += 1 + self.set_edit_text(self.history[self.last_index]) + self.edit_pos = len(self.history[self.last_index]) + elif key == 'meta enter': + # When using multiline, enter inserts a new line into the editor + # send the input to the server on alt + enter + self.parent.cb_send_to_server(msg) + self.history.append(msg) + self.set_edit_text('') + self.last_index = -1 + self.show_history = False + else: + self.show_history = False + self.last_index = -1 + return cast(Optional[str], super().keypress(size, key)) + return None + + +class EditorWidget(urwid.Filler): + """ + Wrapper around the editor widget. + + The Editor is a flow widget and has to wrapped inside a box widget. + This class wraps the Editor inside filler widget. + + :param parent: Reference to the TUI object. + """ + def __init__(self, parent: App) -> None: + super().__init__(Editor(parent), valign='top') + + +class HistoryBox(urwid.ListBox): + """ + This widget is modelled using the ListBox widget, contains the list of + all messages both QMP messages and log messsages to be shown in the TUI. + + The messages are urwid.Text widgets. On every append of a message, the + focus is shifted to the last appended message. + + :param parent: Reference to the TUI object. + """ + def __init__(self, parent: App) -> None: + self.parent = parent + self.history = urwid.SimpleFocusListWalker([]) + super().__init__(self.history) + + def add_to_history(self, history: str) -> None: + """ + Appends a message to the list and set the focus to the last appended + message. + + :param history: + The history item(message/event) to be appended to the list. + """ + self.history.append(urwid.Text(history)) + self.history.set_focus(len(self.history) - 1) + + def mouse_event(self, size: Tuple[int, int], _event: str, button: float, + _x: int, _y: int, focus: bool) -> None: + # Unfortunately there are no urwid constants that represent the mouse + # events. + if button == 4: # Scroll up event + super().keypress(size, 'up') + elif button == 5: # Scroll down event + super().keypress(size, 'down') + + +class HistoryWindow(urwid.Frame): + """ + This window composes the HistoryBox and EditorWidget in a horizontal split. + By default the first focus is given to the history box. + + :param parent: Reference to the TUI object. + """ + def __init__(self, parent: App) -> None: + self.parent = parent + self.editor_widget = EditorWidget(parent) + self.editor = urwid.LineBox(self.editor_widget) + self.history = HistoryBox(parent) + self.body = urwid.Pile([('weight', 80, self.history), + ('weight', 20, self.editor)]) + super().__init__(self.body) + urwid.connect_signal(self.parent, UPDATE_MSG, self.cb_add_to_history) + + def cb_add_to_history(self, msg: str, level: Optional[str] = None) -> None: + """ + Appends a message to the history box + + :param msg: + The message to be appended to the history box. + """ + if level: + msg = f'[{level}]: {msg}' + self.history.add_to_history(msg) + + +class Window(urwid.Frame): + """ + This window is the top most widget of the TUI and will contain other + windows. Each child of this widget is responsible for displaying a specific + functionality. + + :param parent: Reference to the TUI object. + """ + def __init__(self, parent: App) -> None: + self.parent = parent + footer = StatusBar() + body = HistoryWindow(parent) + super().__init__(body, footer=footer) + + +class TUILogHandler(Handler): + """ + This handler routes all the log messages to the TUI screen. + It is installed to the root logger to so that the log message from all + libraries begin used is routed to the screen. + + :param tui: Reference to the TUI object. + """ + def __init__(self, tui: App) -> None: + super().__init__() + self.tui = tui + + def emit(self, record: LogRecord) -> None: + """ + Emits a record to the TUI screen. + + Appends the log message to the TUI screen + """ + level = record.levelname + msg = record.getMessage() + self.tui.add_to_history(msg, level) + + +def main() -> None: + """ + Driver of the whole script, parses arguments, initialize the TUI and + the logger. + """ + parser = argparse.ArgumentParser(description='AQMP TUI') + parser.add_argument('qmp_server', help='Address of the QMP server. ' + 'Format ') + parser.add_argument('--num-retries', type=int, default=10, + help='Number of times to reconnect before giving up.') + parser.add_argument('--retry-delay', type=int, + help='Time(s) to wait before next retry. ' + 'Default action is to wait 2s between each retry.') + parser.add_argument('--log-file', help='The Log file name') + parser.add_argument('--log-level', default='WARNING', + help='Log level ') + parser.add_argument('--asyncio-debug', action='store_true', + help='Enable debug mode for asyncio loop. ' + 'Generates lot of output, makes TUI unusable when ' + 'logs are logged in the TUI. ' + 'Use only when logging to a file.') + args = parser.parse_args() + + try: + address = QEMUMonitorProtocol.parse_address(args.qmp_server) + except QMPBadPortError as err: + parser.error(str(err)) + + app = App(address, args.num_retries, args.retry_delay) + + root_logger = logging.getLogger() + root_logger.setLevel(logging.getLevelName(args.log_level)) + + if args.log_file: + root_logger.addHandler(logging.FileHandler(args.log_file)) + else: + root_logger.addHandler(TUILogHandler(app)) + + app.run(args.asyncio_debug) + + +if __name__ == '__main__': + main() diff --git a/python/setup.cfg b/python/setup.cfg index efcda23c48d..9bc25998c12 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -81,8 +81,19 @@ namespace_packages = True # fusepy has no type stubs: allow_subclassing_any = True +[mypy-qemu.aqmp.aqmp_tui] +# urwid and urwid_readline have no type stubs: +allow_subclassing_any = True + +# The following missing import directives are because these libraries do not +# provide type stubs. Allow them on an as-needed basis for mypy. [mypy-fuse] -# fusepy has no type stubs: +ignore_missing_imports = True + +[mypy-urwid] +ignore_missing_imports = True + +[mypy-urwid_readline] ignore_missing_imports = True [pylint.messages control] From patchwork Mon Sep 27 19:25:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96801C433F5 for ; Mon, 27 Sep 2021 19:42:03 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3747860EB6 for ; Mon, 27 Sep 2021 19:42:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3747860EB6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:42934 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwVm-0002aC-DN for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:42:02 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58366) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJh-00086n-Sx for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:34835) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJf-0000So-Oh for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R8Uq0WfdTpOBvKOJZTXpYgvE/iHJAMqPyEJHCYwOSoQ=; b=DXRQQV3quOEgoCEfJsD5ovg1567XVjrzsTN7iKBIaZ6n11hJW7kAXM6CuaTWtWbVH5HpyM ZyKJqzlIdXDv2GMi4RrFI9BjEzwKIYiQTSxQ41dMvjkO+LF961nL7K5VxsgKFmumvCZNok XeCsVJBJTpS9GdV8bygMoahnLnhOE5A= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-413-XPKOoR-kPla0S2EG8GrU8A-1; Mon, 27 Sep 2021 15:29:29 -0400 X-MC-Unique: XPKOoR-kPla0S2EG8GrU8A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 75BFA801E72; Mon, 27 Sep 2021 19:29:28 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id E437760871; Mon, 27 Sep 2021 19:29:23 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 30/32] python: Add entry point for aqmp-tui Date: Mon, 27 Sep 2021 15:25:11 -0400 Message-Id: <20210927192513.744199-31-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , G S Niteesh Babu , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: G S Niteesh Babu Add an entry point for aqmp-tui. This will allow it to be run from the command line using "aqmp-tui localhost:1234" More options available in the TUI can be found using "aqmp-tui -h" Signed-off-by: G S Niteesh Babu Message-Id: <20210823220746.28295-4-niteesh.gs@gmail.com> Signed-off-by: John Snow --- python/setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/python/setup.cfg b/python/setup.cfg index 9bc25998c12..eefa9613f18 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -66,6 +66,7 @@ console_scripts = qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point [fuse] qemu-ga-client = qemu.qmp.qemu_ga_client:main qmp-shell = qemu.qmp.qmp_shell:main + aqmp-tui = qemu.aqmp.aqmp_tui:main [tui] [flake8] extend-ignore = E722 # Prefer pylint's bare-except checks to flake8's From patchwork Mon Sep 27 19:25:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7CE4C433EF for ; Mon, 27 Sep 2021 19:44:49 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 718B360F41 for ; Mon, 27 Sep 2021 19:44:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 718B360F41 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:51216 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwYS-00086b-Jn for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:44:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58378) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJr-0008Hv-Tw for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:44 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:26290) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwJq-0000aZ-Ev for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770980; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s/i+TG0bbXfR88RS/VMfTTBCRXSGCCeVuCAVQLYJb44=; b=IggwY0ZAstlyKO1JWTsrKIfrAciDFG4nhbOwN0um2P2IAoMEv56p3+rGqfiR6ncHZooIna hdrYdruLwbbyamaxEi1BmOi8Nsxxtsy8EIu8olf+0exIWp0ownQDDwarZA4LzdqUXIUQxA TTIFdq1BC5U00+blfQDxMf/RrWd6tz8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-317-3vSMk7BQOWS2iHHLgwH-Og-1; Mon, 27 Sep 2021 15:29:39 -0400 X-MC-Unique: 3vSMk7BQOWS2iHHLgwH-Og-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 631A9835DE1; Mon, 27 Sep 2021 19:29:38 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FE5C60871; Mon, 27 Sep 2021 19:29:28 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 31/32] python: add optional pygments dependency Date: Mon, 27 Sep 2021 15:25:12 -0400 Message-Id: <20210927192513.744199-32-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , G S Niteesh Babu , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: G S Niteesh Babu Added pygments as optional dependency for AQMP TUI. This is required for the upcoming syntax highlighting feature in AQMP TUI. The dependency has also been added in the devel optional group. Added mypy 'ignore_missing_imports' for pygments since it does not have any type stubs. Signed-off-by: G S Niteesh Babu Message-Id: <20210823220746.28295-5-niteesh.gs@gmail.com> Signed-off-by: John Snow --- python/Pipfile.lock | 8 ++++++++ python/setup.cfg | 5 +++++ 2 files changed, 13 insertions(+) diff --git a/python/Pipfile.lock b/python/Pipfile.lock index da7a4ee164c..d2a7dbd88be 100644 --- a/python/Pipfile.lock +++ b/python/Pipfile.lock @@ -200,6 +200,14 @@ ], "version": "==2.0.0" }, + "pygments": { + "hashes": [ + "sha256:a18f47b506a429f6f4b9df81bb02beab9ca21d0a5fee38ed15aef65f0545519f", + "sha256:d66e804411278594d764fc69ec36ec13d9ae9147193a1740cd34d272ca383b8e" + ], + "markers": "python_version >= '3.5'", + "version": "==2.9.0" + }, "pylint": { "hashes": [ "sha256:082a6d461b54f90eea49ca90fff4ee8b6e45e8029e5dbd72f6107ef84f3779c0", diff --git a/python/setup.cfg b/python/setup.cfg index eefa9613f18..417e937839b 100644 --- a/python/setup.cfg +++ b/python/setup.cfg @@ -46,6 +46,7 @@ devel = tox >= 3.18.0 urwid >= 2.1.2 urwid-readline >= 0.13 + Pygments >= 2.9.0 # Provides qom-fuse functionality fuse = @@ -55,6 +56,7 @@ fuse = tui = urwid >= 2.1.2 urwid-readline >= 0.13 + Pygments >= 2.9.0 [options.entry_points] console_scripts = @@ -97,6 +99,9 @@ ignore_missing_imports = True [mypy-urwid_readline] ignore_missing_imports = True +[mypy-pygments] +ignore_missing_imports = True + [pylint.messages control] # Disable the message, report, category or checker with the given id(s). You # can either give multiple identifiers separated by comma (,) or put this From patchwork Mon Sep 27 19:25:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Snow X-Patchwork-Id: 12520685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61BA0C433EF for ; Mon, 27 Sep 2021 19:57:38 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E36B760E8B for ; Mon, 27 Sep 2021 19:57:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E36B760E8B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:49830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mUwkr-0001ZJ-3v for qemu-devel@archiver.kernel.org; Mon, 27 Sep 2021 15:57:37 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58398) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwK2-00004x-93 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:54 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:23105) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mUwK0-0000iC-N6 for qemu-devel@nongnu.org; Mon, 27 Sep 2021 15:29:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632770991; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JL+C+AZDaSlwYX3+M6yNwQ36iegdKO/Cb3wY3qCaz0g=; b=bMH/o6FrU+g4wTOsmxj/mweY4ZUY8tM30YKLH70zI417jaLFqVXZO+Y38JZTEGqWvPxVvv h6RfKHJtG8FybFgwc5AacMicLf6yWVYQ25PTIgsD+TdlBIxL9IrKQSozXduDKP1lkojr2q G1R98kd9Y1CRaKI8xo4XKBdmCDDLGXw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-551-Fkc1Ibo3OLO1DA5F29q9ig-1; Mon, 27 Sep 2021 15:29:50 -0400 X-MC-Unique: Fkc1Ibo3OLO1DA5F29q9ig-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EFF861005E53; Mon, 27 Sep 2021 19:29:48 +0000 (UTC) Received: from scv.redhat.com (unknown [10.22.9.55]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8EB6A60871; Mon, 27 Sep 2021 19:29:38 +0000 (UTC) From: John Snow To: qemu-devel@nongnu.org Subject: [PULL 32/32] python/aqmp-tui: Add syntax highlighting Date: Mon, 27 Sep 2021 15:25:13 -0400 Message-Id: <20210927192513.744199-33-jsnow@redhat.com> In-Reply-To: <20210927192513.744199-1-jsnow@redhat.com> References: <20210927192513.744199-1-jsnow@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Thomas Huth , Daniel Berrange , Eduardo Habkost , =?utf-8?q?Alex_Benn=C3=A9e?= , Markus Armbruster , Wainer dos Santos Moschetta , =?utf-8?q?Philippe_Mathi?= =?utf-8?q?eu-Daud=C3=A9?= , Willian Rampazzo , G S Niteesh Babu , Cleber Rosa , John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: G S Niteesh Babu Add syntax highlighting for the incoming and outgoing QMP messages. This is achieved using the pygments module which was added in a previous commit. The current implementation is a really simple one which doesn't allow for any configuration. In future this has to be improved to allow for easier theme config using an external config of some sort. Signed-off-by: G S Niteesh Babu Message-Id: <20210823220746.28295-6-niteesh.gs@gmail.com> Signed-off-by: John Snow --- python/qemu/aqmp/aqmp_tui.py | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/python/qemu/aqmp/aqmp_tui.py b/python/qemu/aqmp/aqmp_tui.py index ac533541d2f..a2929f771cf 100644 --- a/python/qemu/aqmp/aqmp_tui.py +++ b/python/qemu/aqmp/aqmp_tui.py @@ -30,6 +30,8 @@ cast, ) +from pygments import lexers +from pygments import token as Token import urwid import urwid_readline @@ -45,6 +47,22 @@ UPDATE_MSG: str = 'UPDATE_MSG' +palette = [ + (Token.Punctuation, '', '', '', 'h15,bold', 'g7'), + (Token.Text, '', '', '', '', 'g7'), + (Token.Name.Tag, '', '', '', 'bold,#f88', 'g7'), + (Token.Literal.Number.Integer, '', '', '', '#fa0', 'g7'), + (Token.Literal.String.Double, '', '', '', '#6f6', 'g7'), + (Token.Keyword.Constant, '', '', '', '#6af', 'g7'), + ('DEBUG', '', '', '', '#ddf', 'g7'), + ('INFO', '', '', '', 'g100', 'g7'), + ('WARNING', '', '', '', '#ff6', 'g7'), + ('ERROR', '', '', '', '#a00', 'g7'), + ('CRITICAL', '', '', '', '#a00', 'g7'), + ('background', '', 'black', '', '', 'g7'), +] + + def format_json(msg: str) -> str: """ Formats valid/invalid multi-line JSON message into a single-line message. @@ -353,6 +371,9 @@ def run(self, debug: bool = False) -> None: :param debug: Enables/Disables asyncio event loop debugging """ + screen = urwid.raw_display.Screen() + screen.set_terminal_properties(256) + self.aloop = asyncio.get_event_loop() self.aloop.set_debug(debug) @@ -364,6 +385,8 @@ def run(self, debug: bool = False) -> None: event_loop = urwid.AsyncioEventLoop(loop=self.aloop) main_loop = urwid.MainLoop(urwid.AttrMap(self.window, 'background'), unhandled_input=self.unhandled_input, + screen=screen, + palette=palette, handle_mouse=True, event_loop=event_loop) @@ -487,7 +510,8 @@ def __init__(self, parent: App) -> None: self.history = urwid.SimpleFocusListWalker([]) super().__init__(self.history) - def add_to_history(self, history: str) -> None: + def add_to_history(self, + history: Union[str, List[Tuple[str, str]]]) -> None: """ Appends a message to the list and set the focus to the last appended message. @@ -531,10 +555,18 @@ def cb_add_to_history(self, msg: str, level: Optional[str] = None) -> None: :param msg: The message to be appended to the history box. + :param level: + The log level of the message, if it is a log message. """ + formatted = [] if level: msg = f'[{level}]: {msg}' - self.history.add_to_history(msg) + formatted.append((level, msg)) + else: + lexer = lexers.JsonLexer() # pylint: disable=no-member + for token in lexer.get_tokens(msg): + formatted.append(token) + self.history.add_to_history(formatted) class Window(urwid.Frame):