From patchwork Thu Apr 2 17:17:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11471153 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1686C81 for ; Thu, 2 Apr 2020 17:18:42 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EFEAA20757 for ; Thu, 2 Apr 2020 17:18:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EFEAA20757 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jK3TW-0004WL-0h; Thu, 02 Apr 2020 17:17:54 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jK3TU-0004WG-Gb for xen-devel@lists.xenproject.org; Thu, 02 Apr 2020 17:17:52 +0000 X-Inumbo-ID: e1d9d0f8-7505-11ea-b4f4-bc764e2007e4 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e1d9d0f8-7505-11ea-b4f4-bc764e2007e4; Thu, 02 Apr 2020 17:17:51 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1jK3TS-0005Fa-TZ; Thu, 02 Apr 2020 18:17:50 +0100 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH 1/2] ts-logs-capture: Cope better with unbootable host Date: Thu, 2 Apr 2020 18:17:46 +0100 Message-Id: <20200402171747.4662-1-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" If we cannot ssh to the host to fish out its own logs, do not try to do any of the other log captures which involve ssh'ing to the host. This includes {fetch,extract}_logs_guest, which tolerated this situation - so then it's an optimisation. But it also includes shutdown_guests, which was introduced in c5f8d41143ab "ts-logs-capture: Fish some logs out of guest filesystem" and is not tolerant enough. Since that commit, unbootable hosts have caused ts-logs-capture to wrongly declare jobs broken. Signed-off-by: Ian Jackson --- ts-logs-capture | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/ts-logs-capture b/ts-logs-capture index c67856cd..7940aece 100755 --- a/ts-logs-capture +++ b/ts-logs-capture @@ -190,7 +190,7 @@ sub fetch_logs_host () { 1; }) { logm("host reboot failed, abandoning log fetches: $@"); - return; + return 0; } try_fetch_logs($ho, $logs); } @@ -219,6 +219,8 @@ sub fetch_logs_host () { ) { try_cmd_output_save($cmd); } + + return 1; } sub fetch_xenctx_guest ($) { @@ -293,8 +295,9 @@ power_state($ho,1); find_guests(); fetch_xenctx_guest($_) foreach @guests; serial_fetch_logs($ho); -fetch_logs_host(); -fetch_logs_guest($_) foreach @guests; -shutdown_guests(); -extract_logs_guest($_) foreach @allguests; +if (fetch_logs_host()) { + fetch_logs_guest($_) foreach @guests; + shutdown_guests(); + extract_logs_guest($_) foreach @allguests; +} logm("logs captured to $stash"); From patchwork Thu Apr 2 17:17:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ian Jackson X-Patchwork-Id: 11471155 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B73492A for ; Thu, 2 Apr 2020 17:18:50 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E56E320757 for ; Thu, 2 Apr 2020 17:18:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E56E320757 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=eu.citrix.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jK3Ta-0004Wi-9F; Thu, 02 Apr 2020 17:17:58 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jK3TZ-0004Wa-EY for xen-devel@lists.xenproject.org; Thu, 02 Apr 2020 17:17:57 +0000 X-Inumbo-ID: e1f9e8f2-7505-11ea-83d8-bc764e2007e4 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e1f9e8f2-7505-11ea-83d8-bc764e2007e4; Thu, 02 Apr 2020 17:17:51 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1jK3TT-0005Fa-4i; Thu, 02 Apr 2020 18:17:51 +0100 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH 2/2] ts-logs-capture: Cope better with unbootable host (2) Date: Thu, 2 Apr 2020 18:17:47 +0100 Message-Id: <20200402171747.4662-2-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20200402171747.4662-1-ian.jackson@eu.citrix.com> References: <20200402171747.4662-1-ian.jackson@eu.citrix.com> X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" shutdown_guests might conceivably fail due to a flaky host. In that case we want not to declare the job broken so ts-logs-capture most not fail. But in that case we should skip fishing in-guest logs out of guest fs's. Signed-off-by: Ian Jackson --- ts-logs-capture | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/ts-logs-capture b/ts-logs-capture index 7940aece..0320a5a5 100755 --- a/ts-logs-capture +++ b/ts-logs-capture @@ -264,17 +264,23 @@ sub fetch_logs_guest ($) { } sub shutdown_guests () { - target_cmd_root($ho, <<'END', 180); - set -x - ( - ( exec 2>/dev/null; sleep 30 ; echo y ) & - ( xl shutdown -a -F -w ; echo y ) & - ) | ( - read x - xl list | awk '!/^Domain-0 |^Name / {print $2}' \ - | xargs -t -r -n1 xl destroy ||: - ) + if (!eval { + target_cmd_root($ho, <<'END', 180); + set -x + ( + ( exec 2>/dev/null; sleep 30 ; echo y ) & + ( xl shutdown -a -F -w ; echo y ) & + ) | ( + read x + xl list | awk '!/^Domain-0 |^Name / {print $2}' \ + | xargs -t -r -n1 xl destroy ||: + ) END + }) { + logm("cannot ensure no guests running, cannot fish their logs"); + return 0; + } + return 1; } sub extract_logs_guest ($) { @@ -297,7 +303,8 @@ fetch_xenctx_guest($_) foreach @guests; serial_fetch_logs($ho); if (fetch_logs_host()) { fetch_logs_guest($_) foreach @guests; - shutdown_guests(); - extract_logs_guest($_) foreach @allguests; + if (shutdown_guests()) { + extract_logs_guest($_) foreach @allguests; + } } logm("logs captured to $stash");