From patchwork Fri Feb 12 17:20:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 12085711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76061C433E0 for ; Fri, 12 Feb 2021 17:24:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1077364D79 for ; Fri, 12 Feb 2021 17:24:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1077364D79 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38818 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAcAi-0000Er-06 for qemu-devel@archiver.kernel.org; Fri, 12 Feb 2021 12:24:00 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44424) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAc7X-0005rP-W4 for qemu-devel@nongnu.org; Fri, 12 Feb 2021 12:20:44 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:55688) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1lAc7W-0001yN-7V for qemu-devel@nongnu.org; Fri, 12 Feb 2021 12:20:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1613150439; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GC0xwzn1JA4ElVC6izjtMT/Az7zQUDEK6CQ7rC/GCuw=; b=DSAoesgnZvEBPBiI/ZqQ7kFlUyxds+0godIxzfvOeLsGJHDAR+xf0z5XzNGfjMQX+z2cRQ z+9xQebQo4Y7k6Dx8UWwq9gJ4A6+nTqEBweoWHpf+IeKEPBLm9CUVR7dTlVfKgkrOIlPIa n7PeEhk0oBbiJHNwJgTSC6ONwTk5wYs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-516-sNSn4QLzMwmbKJbLlsi_kA-1; Fri, 12 Feb 2021 12:20:38 -0500 X-MC-Unique: sNSn4QLzMwmbKJbLlsi_kA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 412E41005501 for ; Fri, 12 Feb 2021 17:20:37 +0000 (UTC) Received: from merkur.fritz.box (ovpn-112-163.ams2.redhat.com [10.36.112.163]) by smtp.corp.redhat.com (Postfix) with ESMTP id DE4F51B5FB; Fri, 12 Feb 2021 17:20:32 +0000 (UTC) From: Kevin Wolf To: qemu-devel@nongnu.org Subject: [PATCH 1/2] monitor: Fix assertion failure on shutdown Date: Fri, 12 Feb 2021 18:20:27 +0100 Message-Id: <20210212172028.288825-2-kwolf@redhat.com> In-Reply-To: <20210212172028.288825-1-kwolf@redhat.com> References: <20210212172028.288825-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=63.128.21.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.569, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Commit 357bda95 already tried to fix the order in monitor_cleanup() by moving shutdown of the dispatcher coroutine further to the start. However, it didn't go far enough: iothread_stop() makes sure that all pending work (bottom halves) in the AioContext of the monitor iothread is completed. iothread_destroy() depends on this and fails an assertion if there is still a pending BH. While the dispatcher coroutine is running, it will try to resume the monitor after taking a request out of the queue, which involves a BH. The dispatcher is run until it terminates in the AIO_WAIT_WHILE() loop. However, adding new BHs between iothread_stop() and iothread_destroy() is forbidden. Fix this by stopping the dispatcher first before shutting down the other parts of the monitor. This means we can now receive requests that aren't handled any more when QEMU is shutting down, but this is unlikely to be a problem for QMP clients. Fixes: 357bda9590784ff75803d52de43150d4107ed98e Signed-off-by: Kevin Wolf --- monitor/monitor.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/monitor/monitor.c b/monitor/monitor.c index 1e4a6b3f20..e94f532cf5 100644 --- a/monitor/monitor.c +++ b/monitor/monitor.c @@ -618,16 +618,6 @@ void monitor_data_destroy(Monitor *mon) void monitor_cleanup(void) { - /* - * We need to explicitly stop the I/O thread (but not destroy it), - * clean up the monitor resources, then destroy the I/O thread since - * we need to unregister from chardev below in - * monitor_data_destroy(), and chardev is not thread-safe yet - */ - if (mon_iothread) { - iothread_stop(mon_iothread); - } - /* * The dispatcher needs to stop before destroying the monitor and * the I/O thread. @@ -637,6 +627,11 @@ void monitor_cleanup(void) * eventually terminates. qemu_aio_context is automatically * polled by calling AIO_WAIT_WHILE on it, but we must poll * iohandler_ctx manually. + * + * Letting the iothread continue while shutting down the dispatcher + * means that new requests may still be coming in. This is okay, + * we'll just leave them in the queue without sending a response + * and monitor_data_destroy() will free them. */ qmp_dispatcher_co_shutdown = true; if (!qatomic_xchg(&qmp_dispatcher_co_busy, true)) { @@ -647,6 +642,16 @@ void monitor_cleanup(void) (aio_poll(iohandler_get_aio_context(), false), qatomic_mb_read(&qmp_dispatcher_co_busy))); + /* + * We need to explicitly stop the I/O thread (but not destroy it), + * clean up the monitor resources, then destroy the I/O thread since + * we need to unregister from chardev below in + * monitor_data_destroy(), and chardev is not thread-safe yet + */ + if (mon_iothread) { + iothread_stop(mon_iothread); + } + /* Flush output buffers and destroy monitors */ qemu_mutex_lock(&monitor_lock); monitor_destroyed = true; From patchwork Fri Feb 12 17:20:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Wolf X-Patchwork-Id: 12085707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C86C0C433DB for ; Fri, 12 Feb 2021 17:23:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 46CBC64E56 for ; Fri, 12 Feb 2021 17:23:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 46CBC64E56 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37238 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lAc9w-00080z-A6 for qemu-devel@archiver.kernel.org; Fri, 12 Feb 2021 12:23:12 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:44478) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lAc7b-0005xB-CV for qemu-devel@nongnu.org; Fri, 12 Feb 2021 12:20:47 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20255) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1lAc7Y-00020s-Qv for qemu-devel@nongnu.org; Fri, 12 Feb 2021 12:20:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1613150443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S0kyGhpR8yB4K4YNl5zqpUon9HXiHudomNtiOKgzoaw=; b=eFY95le4TRaASZV2wRPZknujykLYeqgyQJ3vfAhf0TCHff0vRqkIGBhg+pB0EgloIiev79 GctT5Tlnlg3YnoedQi8wyfTUldXGKaDau8wMRYLV0vchWlo+4Mk0/XwYqihPel/5/Tafut phuV8PM42lcMJR08ha1B5+/ZtoYQcXw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-333-Kij5I4YxMD-wrJV7O-J8JA-1; Fri, 12 Feb 2021 12:20:42 -0500 X-MC-Unique: Kij5I4YxMD-wrJV7O-J8JA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 47DBD192CC40 for ; Fri, 12 Feb 2021 17:20:41 +0000 (UTC) Received: from merkur.fritz.box (ovpn-112-163.ams2.redhat.com [10.36.112.163]) by smtp.corp.redhat.com (Postfix) with ESMTP id 89C9A5DF21; Fri, 12 Feb 2021 17:20:37 +0000 (UTC) From: Kevin Wolf To: qemu-devel@nongnu.org Subject: [PATCH 2/2] monitor/qmp: Stop processing requests when shutdown is requested Date: Fri, 12 Feb 2021 18:20:28 +0100 Message-Id: <20210212172028.288825-3-kwolf@redhat.com> In-Reply-To: <20210212172028.288825-1-kwolf@redhat.com> References: <20210212172028.288825-1-kwolf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=kwolf@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=216.205.24.124; envelope-from=kwolf@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.569, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, armbru@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" monitor_qmp_dispatcher_co() used to check whether shutdown is requested only when it would have to wait for new requests. If there were still some queued requests, it would try to execute all of them before shutting down. This can be surprising when the queued QMP commands take long or hang because Ctrl-C may not actually exit QEMU as soon as possible. Change monitor_qmp_dispatcher_co() so that it additionally checks whether shutdown is request before it gets a new request from the queue. Signed-off-by: Kevin Wolf --- monitor/qmp.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/monitor/qmp.c b/monitor/qmp.c index 43880fa623..2326bd7f9b 100644 --- a/monitor/qmp.c +++ b/monitor/qmp.c @@ -227,6 +227,11 @@ void coroutine_fn monitor_qmp_dispatcher_co(void *data) */ qatomic_mb_set(&qmp_dispatcher_co_busy, false); + /* On shutdown, don't take any more requests from the queue */ + if (qmp_dispatcher_co_shutdown) { + return; + } + while (!(req_obj = monitor_qmp_requests_pop_any_with_lock())) { /* * No more requests to process. Wait to be reentered from